Jan 25 2014

A year of Overmixing

Category: Overmix,Programs,SoftwareSpiller @ 01:54

It is now about a year ago since I started this project, and not in my wildest dreams I would have imagined how much work I have put into this by now. So here is a quick overview of my progress.

(Overmix is now located at github: https://github.com/spillerrec/Overmix, download here: https://github.com/spillerrec/Overmix/releases )

Automatic aligning

While zooming and rotating is not supported (yet at least), horizontal and vertical movement is rather reliably detected and can be done to sub-pixel precision. Sub-pixel precision is rather slow and memory intensive though, as it works by image upscaling. Still needs some work when aligning images with transparent regions though, and I also believe it can be made quite a bit faster.

De-noising

Noise is removed by averaging all frames and works very well, even on blue-ray rips. However since I have not been able to render while taking sub-pixel alignment into account, it blurs the image ever so slightly.

10-bit raw input and de-telecine

I have made a frame dumper to grab the raw YUV data from the video streams, to ensure no quality loss happen between video file and Overmix. This actually showed that VLC outputs in pretty low quality, and appears to have some trouble with Hi10p content and color spaces. Most noticeable however is that VLC uses nearest-neighbor for chroma-upsampling, which especially looks bad with reds. Having access directly to the chroma channels also opens up for the possibility of doing chroma-restoration using super resolution.

I have made several tools for my custom “dump” format, including the video dumper, Windows shell thumbnailer (vista/7/8), QT5 image plugin and a compressor to reduce the filesize. They can be found in their separate repository here: https://github.com/spillerrec/dump-tools

De-telecine have also been added, as it is necessary in order to work with the interlaced MPEG2 transport stream anime usually is broadcasted in.

Deconvolution

Using deconvolution I have found that I’m able to make the resulting images much sharper, at the expense of a little bit of noise. It appears that anime is significantly blurred, perhaps because of the way it have been rendered, or perhaps intentionally. More on this later…

TV-logo detection and removal

It works decently as shown in a recent post and can also remove other static content such as credits. I have tried to do it fully automatically, but it doesn’t work well enough compared to the manual process. Perhaps some Otsu thresholding combined with dilation will work?

I’m intending to try doing further work with this method to see if I can get it to work with moving items, and doing the reverse, separating the moving item. This is interesting as some panning scenes moves the background independently of the foreground (for faking depth of field).

Steam removal

Prof-of-concept showed that if the steam is moving, we can chose the parts from each frame which contains the least amount of steam. Thus it doesn’t remove it completely, but it can make a significant difference. The current implementation however deals with the colors incorrectly, which is fixable, but not something I care to do unless requested…

Animation

Some scenes contains a repeating animation, while doing vertical/horizontal movement. Especially H-titles seems have much of this, but can also be found as mouth movement and similar. While still in its early stages, I have successfully managed to separate the animation into its individual frames and stitch those, using a manual global threshold.

I doubt it would currently work with minor animations, such as changes with the mouth, noise would probably mess it up. So I’m considering investigating other way of calculating the difference, perhaps using the edge-detected image instead, or doing local differences.

10 Responses to “A year of Overmixing”

  1. AW says:

    Do you have binaries for download?

    • Spiller says:

      There haven’t been any interest so I haven’t really bothered much, but you can find one from April here: https://github.com/spillerrec/Overmix/releases It is only for Windows 64-bit though.
      Which DLLs to include can be a bit tricky to confirm without having a clean computer, so notice me if there is any trouble.

      Quick usage information: Drag-and-drop interface, drag image files into the main window. Use the “Vertical” and “Horizontal” check boxes to control which movement is possible. Click “Align&Draw” to complete it. All the other options can be left on its defaults in most cases, so don’t freak out if it looks complicated.

  2. asdf says:

    First of all, thank you for this excellent tool, it makes stitching a lot easier.

    I do have one question though which I hope you can help me with. I’ve noticed that when you take a sequence of frames that have differing levels of brightness/intensity and feed it into Overmix, you get these horizontal bars across the stitch. I’ve been experimenting around with the pre and post-processing filters and I can’t seem to get rid of them. Would you have a solution?

    Here’s an example: http://i.imgur.com/7fgvfzA.png

    • Spiller says:

      Sorry for the late response.

      Yes, you can fix this by using the “Alpha mask” options in the “Preprocess – Filters” tab.
      By clicking “Alpha mask”, you choose a grayscale image (with the same size as your input images) which will act like layer masks in Gimp. Black areas will be transparent, white areas will not. If you use a gradient, it should blend together much more nicely. (You can also remove TV logos and on-screen text such as credits using this.)

      All preprocessing options are applied as soon as the image is loaded, and do not affect already loaded images. (So you will need to reload all images if you forgot to change something.)
      Therefore, the image selected by using “Alpha mask” will be applied to new images only and similarly “Clear mask” will not affect already loaded images.

      While inconvenient, you need to prepare those gradient images outside Overmix. As long the resolution match, you can reuse them in many cases though.

      • asdf says:

        Thanks for the reply. I’m not really a photoshop wizard, so I’ll have a look around the internet and see if I can find some tutorials on how to make alpha masks.

      • asdf says:

        Actually, after a bit of further reading, wouldn’t it be easier to implement some kind of exposure correction filter when blending the images together? Most panorama imaging programs seem to utilize this in the case of real-life photos.

        • Spiller says:

          All you need to do is to create a image like this and use it in Overmix: http://i.imgur.com/c0d2kps.png

          Perhaps far into the future, Overmix will be able to do such corrections automatically. But as of now, there is a working solution that isn’t very time consuming and works well, so it is fairly low priority.

  3. Kaiio says:

    Hello, does this software support changes in the camera’s speed? I tried stitching an image with it by breaking it down to different sections, because I assumed the problem is the camera’s zig-zagging, but even when the direction doesn’t change there are after-images. It also seems inconsistent. These two images are made using the same settings: http://i.imgur.com/iSQzi1h.png http://i.imgur.com/Bwdkh6z.png

    • Spiller says:

      I does support changes in speed, however they do need to be in a consistent order. (It adds them one-by-one, and it needs to be able to find its position on the so-far aligned images.) If you receive inconsistent results, it sounds like you add them in different orders. If you are dragging it from Windows Explorer, keep in mind that when dragging multiple files, the file you drag on affects the order. Drag on the very first one to keep the ordering as seen in Windows Explorer.

      While I rarely have issues with the alignment, it is not perfect when there is movement in both directions:

      • If they start to far away from each other like in the second image, moving the movement slider to the left can help.
      • Sometimes I had issues, to later notice that a small bit of zooming or rotation was happening. It is not currently supported.
      • There is currently an issue if there is a lot of movement in both directions, as the transparent areas are threated as black areas by alignment algorithm.
        This makes it penalize positions where the image overlaps transparent areas, sometimes causing results like the second image. I don’t think that is what happened in your screenshots though. (Follow this issue on Github: https://github.com/spillerrec/Overmix/issues/40 )

      If you still have issues, tell me the series name and episode number, and I will try to look into it. Having cases where Overmix struggles is important to improve it further.

      • Kaiio says:

        It was as the first thing you suggested. Dragging by the first file fixed everything, thanks!

Leave a Reply to asdf