May 23 2015

Extracting an image overlay

Category: Anime,Overmix,Programs,SoftwareSpiller @ 05:15

I’m getting close to having a Super-Resolution implementation, but I took a small detour using what I have learned to a slightly easier problem, extracting overlayed images. Here is an example:
Background merged with overlay
The character is looking into a glass cage and the reflection is added by adding the face as a semi-transparent layer. We also have the background image, i.e. the image without the overlay:
Background layer
A simple way of trying to extract the overlayed image is to subtract the background image, and indeed we can somewhat improve the situation:
Extracted overlay using Gimp
Another approach is to try estimate the overlayed image, which when overlayed on top on the background image will produce the merged image.
This is done starting with a semi-random estimate and then iteratively improve your estimation. (The initial estimate is just the merged image in this case.)
For each iteration we take our estimation and overlays it on the background. If our estimation is off it will obviously differ from our merged image, so we find the difference between the two and use that difference to improve our estimate. After doing enough iterations, we end up with the following:
Extracted overlay using estimation
While not perfect, this is quite a bit better. I still haven’t added regularization which stabilizes the image and thus could improve it, but I’m not quite sure how it affects the image in this context.

Super-Resolution works in a similar fashion, just instead of overlaying an image, it downscales a high-resolution estimate and compares it against the low-resolution images. It just so happens that I never implemented downscaling…

5 Responses to “Extracting an image overlay”

  1. asdf says:

    It’s good to see there’s been lots of progress on your side. Any chance of a new binary that has all the improvements 0.2.0?

    • Spiller says:

      I uploaded a new build, you can find it on the github page like usual:
      There have been added quite a bit to the GUI, but I will probably not be updating the wiki until next release, as I’m still working on it. I will be releasing the next version as soon I have Super Resolution ready and a GUI to control it, which I hope will not take too long.

      • asdf says:

        Awesome! On the topic of super resolution, are you aware of waifu2x? “Single-Image Super-Resolution for anime/fan-art using Deep Convolutional Neural Networks.” They have a website and also a GitHub, so I thought you might be interested.

        • Spiller says:

          No, I haven’t heard about it before. It is pretty nice to see others also have interest in improving anime images, and I will likely be using it for JPEG noise removal (when there really isn’t a better source around).
          I’m interested in multi-image Super Resolution however. FYI, single-image SR makes a large resolution image by looking at how images in general loses details when downscaled. Multi-frame SR uses sub-pixel shifts between multiple images representing the same. The difference is that multi-image SR recovers lost detail, while single-image SR fills in “random” details.
          Here is an example:
          As you can see, my implementation still have issues with darker-than-black areas, but notice how it is more similar to the original than the result produced with waifu2x. (Especially noticeable with the kanji.) However the drawback is that it only works if you have several images, which is often not the case.

          There have been research papers on combining single- and multi-image SR, and I might look into that direction if I run out of things to do. But what I’m really curios about is if anime contains more detail than it appears to. I have noticed and even confirmed with one anime that there is quite a heavy blur applied to it, and it doesn’t appear to be caused by it being rendered in a lower resolution.

          • asdf says:

            In very rare cases (disregarding traditional cel drawn anime) you can find sub-pixel aliasing on lineart and edges for digitally produced anime. This is perhaps evidence that the actual anime was produced > FHD and then downscaled to 1080p during the BD authoring process. For example Kanati Collection being a recent one, and also The Garden of Words (this is a very noticeable example) and the K-ON! Movie.

            So in some instances, yes an anime can have > 1080p worth of details.

Leave a Reply