Nov 03 2013

First Super Resolution results

Category: Overmix,Programs,SoftwareSpiller @ 23:22

Just five months later… Here are some early results using artificial data.

Using Wikimedia Commons “picture of the day” for October 31. 2013 by Diego Delso (CC BY-SA 3.0), I created LR (Low Resolution) images which were 4 times smaller in each direction. Each LR image had its own offset, so to have one LR image for all possible offsets, 16 images was created.

To detect the sub-pixel alignment afterwards, the images were upscaled to 4x their size and ordinary pixel-based alignment was used. The upscaled versions were only used for the alignment and thus discarded afterwards. The final image was then rendered at 4x resolution using cubic interpolation, but taking the sub-pixel alignment into account. Lastly the image was deconvolved in GIMP using the G’MIC plugin to remove blur. The results are shown below:

Left side shows the LR (shown upscaled using Nearest neighbor interpolation) and original image respectively. Right side shows the SR (Super Resolution) results, using different interpolation methods. Both are cubic, however the top is using Mitchell and the bottom is using Spline. In simple terms, Spline is more blurry than Mitchell but has less blocking artifacts. Mitchell is usually pretty good choice (as it is a compromise between several other cubic interpolation methods), however the blocking is pretty noticeable here. Using Spline here avoids that and since we attempt to remove blur afterwards it works pretty well. However do notice that Mitchell does recover slightly more detail in the windows to the right.

But while Mitchell often does appear to be slightly more sharp, it tends to mess up more often, which can clearly be seen on the “The power of” building to the left. The windows are strangely mixed up into each other, while they are perfectly aligned when using Spline.


Results are much better than the LR images, however it is more an magnification of 2x instead of the optimal 4x. And to make matters worse, this is generated optimal data without blur or noise.

However this is the simplest way of doing SR and I believe other methods do give better results. Next I want to try the Fourier-based approach which is also one of the early SR methods. It should give pretty good results, but it is not used much anymore because it does not work for rotated or skewed images.

Using artificial data has really shown me why I have had so little success with it so far. I’m mainly working with anime screenshots and the amount of detail which can be restored is probably not that much. My goal is actually more to avoid blurriness that happens when they are not aligned perfectly. Thus while it should have been obvious, lesson learned, do not test on data which you are not sure whether will give an result or not… What I did gain from this is that anime tends to be rather blurry and that image deconvolution can help a lot. When I understand this blurriness in detail I will probably write more about it though.

2 Responses to “First Super Resolution results”

  1. Bernhard says:

    is this what Olympus is announcing for it’s new E-M5 II ?
    “While the sensor resolution of 16 megapixels is unchanged, the E-M5 II can now shoot in a clever resolution-enhancing 40-megapixel mode. Simply put, this takes eight shots in quick succession, with the sensor position adjusted just half a pixel’s distance between each shot. All eight frames are then combined into a single final image with greatly increased resolution, the only catch being that this trick won’t work with moving subjects.”

    • Spiller says:

      It very much sounds like it, very exciting. I have been wondering why there haven’t been more commercial exploitation, after all super resolution (SR) research started 30 years ago.

      I’m a bit doubtful about half-pixel steps, as this will only allow you to do 4 images, not 8. Two images with offsets -0.5 and 0.5 are exactly one pixel apart, thus the second will have no benefit. But perhaps it is just a simpler explanation for marketing it.

      SR can be used for moving content, as there have been quite a bit work on doing SR video. So it might be to avoid post processing taking even more than the 2 seconds as mentioned, or perhaps it is just to avoid giving people the impression it will work great no matter what.

      This might also have your interest:
      It is about using inertia sensors to digitally remove motion blur caused by camera movement. Would be a pretty neat addition to more low-end cameras such as ones in smart-phones.

Leave a Reply