Predictive Image Restoration Technique Combines Frequency, Spatial Information

University of Tokyo researchers developed a frequency-based “inpainting” method that considers both frequency and spatial information to generate missing portions of images. The technique is a synthesis of two distinct restorative qualities, giving it significant advantages over currently used methods that use only spatial domain information during the learning process. That can cause the loss of interior reconstruction details, resulting in the estimation of only a low-frequency part of the original patch.

The Tokyo researchers showed that frequency-based image inpainting sufficed to convert inpainting to deconvolution in the frequency domain. The effect of the process enabled the accurate prediction of the local structure of missing image regions.

Input images with missing regions (a), DFT of first-stage reconstruction by the authors' deconvolution network (b), image inpainting results (after the second stage) of the proposed approach (c), and ground truth (GT) image (d). The last column shows the prediction of the missing region obtained from the new method and original pixel values for the same region in the GT image. Courtesy of Hiya Roy et al.

“The frequency-domain information contains rich representations that allow the network to perform the image understanding tasks in a better way than the conventional way of using only spatial-domain information,” said Hiya Roy, a researcher at the University of Tokyo. “Therefore in this work, we try to achieve better image inpainting performance by training the networks using both frequency and spatial domain information.”

Historically, image inpainting algorithms have fallen into two categories. Diffusion-based image inpainting algorithms replicate the appearance of the image into the missing region. Such algorithms fill in small holes, or gaps, effectively. As the size of the hole being filled increases, however, the quality of the results diminishes.

The other category is known as patch-based inpainting algorithms. By looking for the best-fitting patch in the image to fill missing portions, these algorithms are able to fill larger holes, though they struggle with complex, or distinctive, portions of an image.

“The originality of the research resides in the fact that the authors used the frequency domain representation, namely the spectrum of the images obtained by fast Fourier transform, at the first stage of inpainting with a deconvolution network,” said Jenny Benois-Pineau of the University of Bordeaux; she is a senior editor at the Journal of Electronic Imaging, which published the work. “This yields a rough inpainting result capturing the structural elements of the image. Then the refinement is fulfilled in the pixel domain by a GAN network. Their approach outperforms the state-of-the-art in all quality metrics: PSNR, SSIM, and L1.”

The work showed that deconvolution in the frequency domain is able to infer the missing regions of the image structure using context that the image provides. In its first stage, the researcher’s model learned the context using frequency domain information. It then reconstructed the high-frequency parts. In the second stage, it used spatial domain information to guide the color scheme of the image and then enhanced the details and structures obtained in the first stage.

“Experimental results showed that our method could achieve results better than state-of-the-art performances on challenging datasets by generating sharper details and perceptually realistic inpainting results,” the researchers noted in their paper. “Based on our empirical results, we believe that methods using both frequency and spatial information should gain dominance because of their superior performance.”

The team predicts that the work will spur the extended use of other types of frequency domain transformations to solve image restoration tasks such as image de-noising.

The research was published in the Journal of Electronic Imaging (www.doi.org/10.1117/1.jei.30.2.023016).