Feb 28 2013

Stitching anime screenshots in overdrive

Category: Anime,Overmix,Programs,SoftwareSpiller @ 00:44

I have been developing a new application named Overmix, which attempts to improve the quality of anime screenshot stitching. This article will shortly explain what stitching is, what issues affect the quality and how Overmix tries to fix those. At the end a short summery of the results for the current progress is given.

Background

One common animation technique is panning where the camera moves/pans over the image, showing only a part of it at a given time:

animation of pan shot

(Shot on YouTube: http://youtu.be/DsHjblyEG88?t=6m25s)

Very little movement actually happens during the shot, in fact only the mouth is moving (presumably to reduce animation costs). This makes it possible to combine the frames together to one large image, which is known as “stitching”.

Source quality

The issue is however that more often than not, the video quality isn’t that great. The video has been compressed and especially if the source is a TV-transmission or webcast, visual artifacts can be quite noticeable:

Example of noise artifacts

The two most significant artifacts with anime encodes is noise (shown above) and color banding/posterization (shown below).

Example of color banding

Reducing artifacts

A stitch is normally done by taking two frames, finding the offset between the two images and then soften the edges between the images to make the transition less apparent (which is usually done by applying a gradient on the alpha channel).

Since this is a time consuming process, as few frames as possible is used. The idea is to do the opposite, use as many frames as possible. The reason is that the artifacts are not static, for every frame they differ slightly. In result, every frame carries a slightly different set of information. The goal is then to derive the original information, based on this set of inconsistent information.

Just by using the average, we can get quite decent results:

Comparison between average and single

(Right is a single frame, left is the average of all unique frames.)

Results

Noise artifacts has shown to nearly disappear completely when simply averaging every frame with each other, even when the source has a significant amount of noise artifacts. Color banding is also reduced but with much more varying amounts.

Even with modern TV-encodes, stitches sees a significant improvement from using this technique and can visually be tell apart at normal magnification. Surprisingly, even when using good BD-encodes there is usually a slight improvement, but normally requires 2-4 times magnification to be noticeable.

It has shown that it often is not possible to make a perfect alignment when sticking to the pixel grid. This causes the images to be slightly more blurry than originally. It is an area which still requires work.

Using the average to derive the result is not always desirable, as the encode might contain information not related to the image. Such information could be subtitles, TV logos or simply errors in the source. See the following image as example, the most-right column of pixels was completely black and shows up as lines in the averaged image.

Stitched with Overmix

However the currently devised algorithms has a tendency to choke on the slight misalignment mentioned previously and cause unwanted artifacts. If this is solved best by fixing the misalignment or by improving the algorithm is up to discussion.

Binaries and source code

Overmix is licensed as GPLv3 and can be found here: Overmix on Github

Binaries for Windows 64-bit can also be found on Github here: Overmix releases

Tags: ,

5 Responses to “Stitching anime screenshots in overdrive”

  1. Dmy says:

    Sadly, that there is no 32bit version…

    • Spiller says:

      I do not currently have a setup capable that can compile a 32-bit version, and it would take me several hours (at least) to set it up and I honestly don’t want to.

      However I do not see any reason to why anyone would benefit from using a 32-bit OS today. If you can convince me that 32 bit is a necessarily for some people, I will put in on my todo list, but as I see it, people have had 10 years now to upgrade to a 64-bit system and failing to do so is their own fault. (For those who swear to Windows XP, note that Overmix relies on software which have dropped support of XP.)

  2. manongmorcon says:

    It works nicely well for non moving images but can’t seem to get it to work on anime images that have the view moving around.

    • Spiller says:

      Sorry for the late reply, I didn’t notice your comment and it has been in the spam queue all this time.

      The aligning method isn’t perfect and might require you to mess with the settings a little. I haven’t spent time improving it as I rarely end up with an image which will not align, but I have been considering to improve it.
      If you have difficulties, feel free to ask. If you share the images you are trying to stitch as well I can give some more specific advice. Preferably create an issue on Github: https://github.com/spillerrec/Overmix/issues but if you do not want to create an account on Github you can send me an mail: spillerrec@gmail.com. Or you can just comment here, but seeing as it took me about 1½ years to notice it I don’t recommend it.

      There is a bit of information on how to use the program on the wiki, but it is still fairly incomplete: https://github.com/spillerrec/Overmix/wiki
      If there is anything specific you would like to know, notify me and I will try to add it.

  3. manongmorcon says:

    You guys should also make Youtube examples of this

Leave a Reply to Spiller