I’m getting close to having a Super-Resolution implementation, but I took a small detour using what I have learned to a slightly easier problem, extracting overlayed images. Here is an example:
The character is looking into a glass cage and the reflection is added by adding the face as a semi-transparent layer. We also have the background image, i.e. the image without the overlay:
A simple way of trying to extract the overlayed image is to subtract the background image, and indeed we can somewhat improve the situation:
Another approach is to try estimate the overlayed image, which when overlayed on top on the background image will produce the merged image.
This is done starting with a semi-random estimate and then iteratively improve your estimation. (The initial estimate is just the merged image in this case.)
For each iteration we take our estimation and overlays it on the background. If our estimation is off it will obviously differ from our merged image, so we find the difference between the two and use that difference to improve our estimate. After doing enough iterations, we end up with the following:
While not perfect, this is quite a bit better. I still haven’t added regularization which stabilizes the image and thus could improve it, but I’m not quite sure how it affects the image in this context.
Super-Resolution works in a similar fashion, just instead of overlaying an image, it downscales a high-resolution estimate and compares it against the low-resolution images. It just so happens that I never implemented downscaling…
June 4th, 2015 12:41
It’s good to see there’s been lots of progress on your side. Any chance of a new binary that has all the improvements 0.2.0?
June 6th, 2015 19:39
I uploaded a new build, you can find it on the github page like usual: https://github.com/spillerrec/Overmix/releases/tag/v0.3.0
There have been added quite a bit to the GUI, but I will probably not be updating the wiki until next release, as I’m still working on it. I will be releasing the next version as soon I have Super Resolution ready and a GUI to control it, which I hope will not take too long.
June 7th, 2015 14:07
Awesome! On the topic of super resolution, are you aware of waifu2x? “Single-Image Super-Resolution for anime/fan-art using Deep Convolutional Neural Networks.” They have a website and also a GitHub, so I thought you might be interested.
June 7th, 2015 19:05
No, I haven’t heard about it before. It is pretty nice to see others also have interest in improving anime images, and I will likely be using it for JPEG noise removal (when there really isn’t a better source around).
I’m interested in multi-image Super Resolution however. FYI, single-image SR makes a large resolution image by looking at how images in general loses details when downscaled. Multi-frame SR uses sub-pixel shifts between multiple images representing the same. The difference is that multi-image SR recovers lost detail, while single-image SR fills in “random” details.
Here is an example: http://i.imgur.com/Fn6vXko.png
As you can see, my implementation still have issues with darker-than-black areas, but notice how it is more similar to the original than the result produced with waifu2x. (Especially noticeable with the kanji.) However the drawback is that it only works if you have several images, which is often not the case.
There have been research papers on combining single- and multi-image SR, and I might look into that direction if I run out of things to do. But what I’m really curios about is if anime contains more detail than it appears to. I have noticed and even confirmed with one anime that there is quite a heavy blur applied to it, and it doesn’t appear to be caused by it being rendered in a lower resolution.
June 10th, 2015 04:59
In very rare cases (disregarding traditional cel drawn anime) you can find sub-pixel aliasing on lineart and edges for digitally produced anime. This is perhaps evidence that the actual anime was produced > FHD and then downscaled to 1080p during the BD authoring process. For example Kanati Collection being a recent one, and also The Garden of Words (this is a very noticeable example) and the K-ON! Movie.
So in some instances, yes an anime can have > 1080p worth of details.