Jul 14 2015

Removing mosaic censor

Category: Anime,Overmix,Programs,SoftwareSpiller @ 12:01

Note: Due to the nature of censored content, I’m showcasing synthetic data in this post.

Japan has its crazy censorship laws, so most hentai anime uses mosaic censoring to hide the important bits. Of course with my work on Overmix I’m interested in camera pans that can be stitched, and those could look like the following:

What is interesting however is that, while it might not be too apparent at first, the mosaic censor changes from frame to frame. Instead of censoring the image which is panned, a specific area is marked as “dirty” and censoring is applied on each frame separately. So looking even closer, we can see that the mosaics are arranged in a grid which is relative to the viewpoint (i.e. fixed) and not the panned image (which is moving).

So depending on how the panned image is offset from the viewpoint, each tile in the mosaic will cover a different area in different frames. If we take all our frames and find the average for each pixels, we can see a vast improvement already:

It is however quite blurry… However super-resolution have thought us that if we have several images of the same motif but where each image caries slightly different information, we can exploit it to increase the resolution. So let’s try to do better.

Before we can start, we need to know how the mosaic censoring works to finish our degradation model. Contrary to what I thought, it is not done by averaging all pixels inside each tile. Instead it is done by picking a single pixel (which I assume is the center one) and use that for the color of the entire tile.
The approach is therefore really simple. We create a high-resolution (HR) grid, and for each frame we plot the tile color into the HR grid into the pixel located at the center of the tile. When we have done that for all the frames, we end up with the following:

There are a lot of black pixels, these are pixels which were not covered by any of our frames, pixels we know nothing about. There is no way of extracting these pixels from our source images, so we need to fake them by using interpolation. I used the “Inpaint – diffusion” filter in G’MIC and the result isn’t too bad:

Those “missing pixels” fully controls how well the decensoring will work. If the viewpoint only pans in a vertical direction we will only have vertical strips of known pixels, thus we cannot increase the resolution in the horizontal dimension. If we have movement in both dimensions it tends to work pretty well as you can see in the overview image below:

Full image available below: (click to view full resolution)

7 Responses to “Removing mosaic censor”

  1. Kaiio says:

    Hello, what do I need to read before I can understand and replicate the things you write about?

    • Spiller says:

      Are you interested in understanding the theoretical aspects, or do you just want to be able to use it in practice?
      I’m currently basing a lot of my work on Image restoration and Super Resolution, so I would recommend the following article as a starting point: http://www-sipl.technion.ac.il/new/Teaching/Projects/Winter2007/SR_Overview.pdf

      If you just want to be able to use some technique I have written, let me know and I will try include it in Overmix and try to explain how to use it on the Wiki.

  2. baul says:

    Could you make a tutorial on the wiki?

    Thanks

  3. Camilo says:

    This is a fun problem to toy around with. Have you considered how to detect the mosaic itself? I’ve been thinking about it and each time I imagine some “trick” to detect it based on e.g., a simple Sobel edge detection, I remember there are some complex, but common, cases where the mosaics aren’t “perfect”.
    For instance, sometimes you have more than one mosaic in the image:
    https://en.wikipedia.org/wiki/File:Pixelization_mosaic.jpg
    But this is not the biggest problem; for single images sometimes the edge of the mosaic is non-rectangular, and in particular, ends in curves. Sometimes, the mosaic’s edge is blurry. For moving pictures, this is not as much of a problem, I guess.

    If you are interested in single-images, maybe you would want to check this project out.
    https://github.com/deeppomf/DeepCreamPy

    • Spiller says:

      Yes, I was messing around a little with detecting mosaic with edge detection, but I was more interested in finding the size and offset than which parts of the image contains the mosaic. I didn’t finish it, but I was making a histogram and then finding the lines which consistently has a lot of edges. Thinking about it, it could probably be improved by looking at the direction of the gradient of the Sobel operator as well instead of just the magnitude. For detecting the mosaic “pixels” as well, I guess you would have to combine the vertical edge strength, the horizontal edge strength, and the standard variance inside each potential “pixel”, to somehow come up with a measure for the likelihood that this mosaic pixel is part of the mosaic. Using a simple machine learning model such as Linear Regression is probably the best thing to do here, as this will never be perfect so using a model which is based on probability will be better able to handle this fact.

      With my Overmix project I have accepted the fact it will never be perfect and I try to make it possible for the user to fix minor issues if possible. That is not always possible, but I personally don’t want something that is fully automatic but only works 95 % of the time.

      I have seen DeepCreamPy but I haven’t tried the 2.0 version yet. They are definitely two very different approaches, this one can only be used in a few cases but produces correct results, while DCP can be used for most images but makes up random results to what it thinks is fitting. This method rarely manages to find the result for every pixel, so it actually also needs some single-image solution like DCP, but the algorithm it uses appears to be a bad fit here. I’m thinking K-SVD might work well, see my issue on the topic if interested: https://github.com/spillerrec/Overmix/issues/135

Leave a Reply to Camilo