Abstract

StyleGAN has demonstrated outstanding contributions due to the high-quality generated images. As a result, it has emerged a question, how to embed images into the StyleGAN latent space? This work tries to answer this question by studying and reproducing Image2StyleGAN (I2S) and ImprovedImage2StyleGAN (II2S). These embedding algorithms compute a latent code for a given input image by optimizing the latent code that minimizes a loss function. This loss function is based on perceptual metrics, which capture the similarity between reference and generated images. Our reproduction partially replaces the algorithm for further experimentation of initializers and perceptual losses. Additionally, we perform some post-image processing operations such as inpainting, super-resolution, colorization, morphing, style transfer, and expression transfer. We verify the behavior of the studied components and stand out particular concerns.

Image2StyleGAN (I2S)

Image2StyleGAN by Rameen Abdal et al. is an optimization algorithm that aims to map a given image (I) into the latent space (w+ encoding) of a pre-trained StyleGAN . These mappings known as latent codes are helpful to perform posterior image processing applications.

In order to get the w+ embedding, the authors’ proposed an optimization framework based on gradient descent. The steps are the following:

1. Start with an initial latent code w* (initial guess).
2. Generate an image with the latent code w*
3. Compare the generated image I* with the reference image I, using a loss function.
4. Based on the loss function (error), update the latent code w* by Gradient Descent.
5. Repeat this process by a given number of iterations.

The optimization process will proceed as follows:

The loss function consists of two terms a VGG-16 perceptual loss function and means square error (MSE).

The MSE loss is a standard pixel-wise error between the generated and reference images.

The perceptual loss is based on a pre-trained neural network (VGG-16 trained with ImageNet), which helps to calculate a similarity between two images in terms of hidden features. The idea to calculate this similarity is to send both images through VGG-16 and capture its output feature maps (hidden features) from different hidden layers.

Improved Image2StyleGAN (II2S)

Improved Image2StyleGAN by Peihao Zhu et al. introduces an additionally normalized space P to analyze the diversity and the quality of the reconstructed latent codes. This space can help answer the question of where good latent codes are located in latent space.

Then, the normalized space is used together with a PCA step to propose a regularization method, which helps to trade off between the reconstruction and image editing capabilities of the algorithm.

Results

Using Image2StyleGAN, we studied three semantic image editing applications: morphing, expression transfer, and style transfer. Each test can be done by simple latent code manipulation of vectors w.

Morphing

Expression Transfer

Style Transfer

Using ImprovedImage2StyleGAN, we studied three editing operations: colorization, inpainting, super-resolution, and style transfer.

Colorization

Inpainting

Super-Resolution

Style Transfer

Related links

Image2StyleGAN is an exciting algorithm with a variety of uses beyond those described here. Bellow, I show some resources that you find valuable.

- StyleGAN 1: A Style-Based Generator Architecture for Generative Adversarial Networks.

- StyleGAN 2: Analyzing and Improving the Image Quality of StyleGAN.

- StyleGAN 2 — Reduced Data: Training Generative Adversarial Networks with Limited Data.

- LPIPS (state-of-the-art Perceptual Metric): The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.

- Image2StyleGAN++: How to Edit the Embedded Images?

Citation

Powered by Jon Barron and Michaël Gharbi.