Generative Adversarial Networks (GANs) for Image Synthesis

Creating Realistic Images Through Adversarial Learning

Authored by: Loveleen Narang

Date: February 2, 2025

Introduction: Teaching Machines to Create

One of the most fascinating frontiers in artificial intelligence is teaching machines not just to analyze data, but to create it. Generative models aim to learn the underlying distribution of a dataset (\( p_{data}(x) \)) (Formula 1) and generate new samples that resemble the original data. Among the most powerful and influential generative models, especially for image synthesis, are Generative Adversarial Networks (GANs), introduced by Ian Goodfellow and colleagues in 2014.

GANs employ a novel training paradigm based on a two-player game between two neural networks: a Generator (\(G\)) and a Discriminator (\(D\)). The Generator's goal is to create realistic data (e.g., images) from random noise, while the Discriminator's goal is to distinguish between real data samples and the fake samples created by the Generator. Through this adversarial process, both networks improve, ideally resulting in a Generator capable of producing highly realistic and diverse synthetic images.

The GAN Architecture: A Generator-Discriminator Duel

The core GAN framework consists of two main components:

• Generator (\(G\)): This network takes a random noise vector \( z \) (typically sampled from a simple distribution like Gaussian or uniform, Formula 2: \( z \sim p_z(z) \)) as input and transforms it into a data sample (e.g., an image) that resembles the real data distribution. Its function can be written as \( \hat{x} = G(z; \theta_g) \) (Formula 3), where \( \theta_g \) are the generator's parameters.
• Discriminator (\(D\)): This network takes a data sample \( x \) (either real from \( p_{data}(x) \) or fake from \( G(z) \)) as input and outputs a single scalar probability representing the likelihood that the input sample is real (rather than generated). Its function is \( D(x; \theta_d) \in [0, 1] \) (Formula 4), where \( \theta_d \) are the discriminator's parameters. Ideally, \( D(x) \approx 1 \) for real samples and \( D(G(z)) \approx 0 \) for fake samples. Formula (5): \( \theta_g \). Formula (6): \( \theta_d \).

For image synthesis, both \( G \) and \( D \) are typically implemented as deep Convolutional Neural Networks (CNNs), often following guidelines like those proposed in DCGAN (Deep Convolutional GANs) which involve using transposed convolutions in the generator and specific architectural choices to stabilize training.

The Minimax Game and Training

GAN training involves a two-player minimax game defined by a value function \( V(D, G) \). The Discriminator \( D \) tries to maximize this value function (correctly classifying real and fake), while the Generator \( G \) tries to minimize it (by producing fakes that \( D \) classifies as real). The original GAN value function is: Formula (7):

The overall objective is: Formula (8):

Training proceeds iteratively, typically alternating between:

1. Training the Discriminator: Sample a mini-batch of real data \( \{x^{(1)}, \dots, x^{(m)}\} \) and generate a mini-batch of fake data \( \{\hat{x}^{(1)}, \dots, \hat{x}^{(m)}\} \) where \( \hat{x}^{(i)} = G(z^{(i)}) \). Update \( \theta_d \) by ascending the stochastic gradient: Formula (9): \( \nabla_{\theta_d} \frac{1}{m} \sum_{i=1}^m [\log D(x^{(i)}) + \log(1 - D(\hat{x}^{(i)}))] \).
2. Training the Generator: Sample a mini-batch of noise \( \{z^{(1)}, \dots, z^{(m)}\} \). Update \( \theta_g \) by descending the stochastic gradient: Formula (10): \( \nabla_{\theta_g} \frac{1}{m} \sum_{i=1}^m \log(1 - D(G(z^{(i)}))) \).

In practice, minimizing \( \log(1 - D(G(z))) \) for the generator can lead to vanishing gradients early in training. A common alternative is to maximize \( \log D(G(z)) \) instead. This is often called the "non-saturating" generator loss: Formula (11):

Mathematical Foundations and Convergence

For a fixed generator \(G\), the optimal discriminator \(D^*\) that maximizes \(V(D, G)\) is given by: Formula (12):

Where \( p_g(x) \) is the distribution of the data generated by \(G\). If we plug \(D^*\) back into the value function, we get the objective that \(G\) implicitly minimizes: Formula (13):

This objective can be shown to be related to the Jensen-Shannon Divergence (JSD) between the real data distribution and the generated distribution: Formula (14):

The Jensen-Shannon Divergence (Formula 15: \( JSD(P||Q) = \frac{1}{2} D_{KL}(P||M) + \frac{1}{2} D_{KL}(Q||M) \), where \( M = \frac{1}{2}(P+Q) \) and \( D_{KL} \) is Kullback-Leibler divergence, Formula 16: \( D_{KL}(P||Q) = \sum P(x) \log \frac{P(x)}{Q(x)} \)) is zero if and only if \( p_{data} = p_g \). Therefore, the global minimum of the minimax game is achieved when the generator perfectly replicates the real data distribution, at which point \( D^*(x) = 1/2 \) everywhere, and \( C(G) = -\log 4 \).

However, achieving this theoretical optimum in practice is challenging due to the high-dimensional, non-convex optimization landscape and difficulties in approximating the gradients accurately.

Improving GAN Training Stability and Quality

Standard GAN training often suffers from instability, vanishing gradients, and mode collapse.

Numerous techniques have been developed to address these issues:

• Loss Function Modifications:
  • Non-Saturating Loss (NS-GAN): Uses \( -\log D(G(z)) \) for the generator (Formula 11), providing stronger gradients early on.
  • Least Squares GAN (LSGAN): Replaces the log-likelihood objective with a least-squares objective, penalizing samples based on their distance to the decision boundary. Discriminator Loss \( L_D \): Formula (17): \( \frac{1}{2} E_{x \sim p_{data}}[(D(x)-1)^2] + \frac{1}{2} E_{z \sim p_z}[D(G(z))^2] \). Generator Loss \( L_G \): Formula (18): \( \frac{1}{2} E_{z \sim p_z}[(D(G(z))-1)^2] \).
  • Wasserstein GAN (WGAN): Uses the Wasserstein-1 distance (Earth Mover's Distance) instead of JSD, which provides smoother gradients even when distributions don't overlap significantly. \( W_1(p_{data}, p_g) = \sup_{||f||_L \le 1} E_{x \sim p_{data}}[f(x)] - E_{x \sim p_g}[f(x)] \) (Formula 19), where \( f \) must be 1-Lipschitz (Formula 20). The Discriminator (called Critic) approximates \( f \). Critic Loss \( L_D \): Formula (21): \( E_{z \sim p_z}[D(G(z))] - E_{x \sim p_{data}}[D(x)] \). Generator Loss \( L_G \): Formula (22): \( -E_{z \sim p_z}[D(G(z))] \). Requires enforcing the Lipschitz constraint, initially done via weight clipping.
  • WGAN with Gradient Penalty (WGAN-GP): Improves WGAN by replacing weight clipping with a gradient penalty term to enforce the Lipschitz constraint more effectively. Penalty Term: Formula (23): \( \lambda E_{\hat{x} \sim p_{\hat{x}}}[ (||\nabla_{\hat{x}} D(\hat{x})||_2 - 1)^2 ] \), where \( \hat{x} \) is sampled along lines between real and fake samples. Formula (24): Gradient Norm \( ||\nabla||_2 \).
• Architectural Guidelines (e.g., DCGAN): Use strided convolutions (Discriminator) and transposed convolutions (Generator) instead of pooling, use Batch Normalization, avoid fully connected layers (mostly), use ReLU (Generator except output) and LeakyReLU (Discriminator) activations. Formula (25): LeakyReLU \( f(x) = \max(\alpha x, x) \).
• Regularization: Techniques like Spectral Normalization stabilize Discriminator training by constraining its Lipschitz constant.

Comparison of Common GAN Loss Functions

GAN Type	Key Idea	Pros	Cons
Original GAN (Minimax / NS)	Minimize JS Divergence	Original formulation	Vanishing gradients, mode collapse, training instability
LSGAN	Least Squares Loss	More stable than original, non-saturating gradients	Can still suffer from mode collapse
WGAN	Minimize Wasserstein Distance (using Critic)	More stable training, meaningful loss metric, less mode collapse	Requires Lipschitz constraint (weight clipping is problematic)
WGAN-GP	WGAN + Gradient Penalty	Stable training, meaningful loss, avoids issues with weight clipping	Gradient penalty adds computational cost

Advanced GAN Architectures for Image Synthesis

Building on the core ideas, many advanced architectures have emerged:

• Conditional GANs (cGANs): Generate data conditioned on additional information \( y \) (e.g., a class label, text description). Both G and D receive \( y \) as input. \( \hat{x} = G(z, y) \), \( D(x, y) \). Objective adapts accordingly, e.g., Formula (26): \( \min_G \max_D V(D, G | y) \). Allows for controlled generation.
• InfoGAN: Learns disentangled representations by maximizing the mutual information between a subset of latent variables \( c \) and the generated output \( G(z, c) \). Objective includes an information-regularization term. Formula (27): \( \min_G \max_D V_I(D, G) = V(D, G) - \lambda I(c; G(z, c)) \). Formula (28): Mutual Information \( I \).
• StyleGAN Family (StyleGAN, StyleGAN2, StyleGAN3): Achieves state-of-the-art high-resolution image synthesis. Key ideas include:
  • Mapping network transforming \( z \) to an intermediate latent space \( W \).
  • Style-based generator where \( w \in W \) controls the style at different resolutions via Adaptive Instance Normalization (AdaIN). Formula (29): \( AdaIN(x, y) = \sigma(y) (\frac{x - \mu(x)}{\sigma(x)}) + \mu(y) \). \( \mu, \sigma \) are mean/std. (Formula 30, 31).
  • Injecting noise at different layers to control stochastic variation.
  • Progressive growing (original StyleGAN) or improved architectural designs (StyleGAN2/3) for high resolution.
• Image-to-Image Translation GANs:
  • Pix2Pix: Learns mapping between paired images (e.g., satellite to map, edges to photo). Uses a cGAN framework with an L1 loss term added to encourage structural similarity. Formula (32): \( L_{Pix2Pix} = L_{cGAN}(G, D) + \lambda L_{L1}(G) \). Formula (33): \( L_{L1}(G) = E[||y - G(x, z)||_1] \).
  • CycleGAN: Learns mapping between unpaired images (e.g., horse to zebra). Uses two Generators (\( G: X \rightarrow Y \), \( F: Y \rightarrow X \)) and two Discriminators. Introduces cycle consistency loss to enforce that translating an image to the other domain and back recovers the original image. Formula (34): \( L_{cyc}(G, F) = E_{x}[||F(G(x)) - x||_1] + E_{y}[||G(F(y)) - y||_1] \).

Overview of Advanced GAN Architectures

Architecture	Key Innovation(s)	Primary Application
DCGAN	Stable CNN architecture guidelines (Conv/TransposeConv, BatchNorm, Activations)	Baseline for stable image generation
Conditional GAN (cGAN)	Conditioning generation on labels/attributes (y)	Controlled image synthesis (e.g., generate specific digits)
StyleGAN Family	Style-based generator, AdaIN, mapping network, noise injection	High-resolution, high-quality realistic image synthesis (esp. faces)
CycleGAN	Unpaired image translation, cycle consistency loss	Style transfer, domain adaptation (e.g., photo to painting, horse to zebra)
Pix2Pix	Paired image translation, cGAN + L1 loss	Tasks with paired data (e.g., edges to photo, map to satellite)

Evaluating GAN Performance

Evaluating generative models is inherently difficult as there's often no single "correct" output. Common metrics include:

• Inception Score (IS): Measures both the quality (low entropy \( p(y|x) \)) and diversity (high entropy \( p(y) \)) of generated images using a pre-trained Inception network. Formula (35): \( IS(G) = \exp(\mathbb{E}_{x \sim p_g} D_{KL}(p(y|x) || p(y))) \). Higher is better. Can be misleading (sensitive to mode dropping).
• Fréchet Inception Distance (FID): Measures the Wasserstein-2 distance between the distribution of Inception features for real images (\(x\)) and generated images (\(g\)). Considers both mean (\(\mu\)) and covariance (\(\Sigma\)) of features. Formula (36): \( FID(x, g) = ||\mu_x - \mu_g||_2^2 + Tr(\Sigma_x + \Sigma_g - 2(\Sigma_x \Sigma_g)^{1/2}) \). Lower is better. Generally considered more robust than IS. Formula (37): Expectation \(E[\cdot]\). Formula (38): Trace \(Tr(\cdot)\).
• Perceptual Path Length (PPL): Used for StyleGANs to measure smoothness of the latent space.
• Human Evaluation: Subjective assessment by humans remains a crucial, albeit costly, evaluation method.

Applications in Image Synthesis

GANs have enabled remarkable applications:

• Generating photorealistic images of faces, animals, objects, scenes.
• Image editing: Inpainting (filling missing regions), super-resolution, colorization.
• Style transfer: Applying the artistic style of one image to another.
• Data augmentation: Creating synthetic data to enlarge training sets.
• Creating art, fashion designs, game assets.
• Medical image synthesis and enhancement.
• Video generation and manipulation (though more complex).
• Deepfakes: Synthesizing realistic videos/images of people (raises significant ethical concerns).

Challenges and Ethical Considerations

Despite their power, GANs face challenges:

• Training Instability: The adversarial training can be difficult to balance, leading to oscillations or divergence. Careful hyperparameter tuning and architectural choices are needed.
• Mode Collapse: The generator may learn to produce only a limited variety of outputs that can fool the current discriminator, failing to capture the full diversity of the data distribution.
• Evaluation Difficulties: Standard metrics like IS and FID don't perfectly capture human perception of quality and diversity.
• Controllability: Directing the generator to produce specific desired outputs can be challenging, although cGANs and StyleGAN offer improvements.
• Ethical Concerns: The ability to generate highly realistic fake images/videos (deepfakes) raises serious concerns about misinformation, manipulation, and privacy. Responsible development and deployment are paramount.

Conclusion

Generative Adversarial Networks have revolutionized the field of generative modeling, particularly for image synthesis. Their unique adversarial training paradigm enables the creation of stunningly realistic and diverse images, driving progress in applications from art generation to data augmentation. While foundational GANs faced stability issues, innovations in loss functions (WGAN, LSGAN), architectures (StyleGAN, CycleGAN), and training techniques have significantly improved performance and control. However, challenges related to training stability, mode collapse, evaluation, and ethical implications remain active areas of research. As GANs continue to evolve, they promise to further blur the lines between real and artificial imagery, offering immense creative potential alongside critical societal responsibilities.

(Formula count check: Includes p_data, z dist, G func, D func, theta_g, theta_d, V(D,G), Minimax obj, D grad, G grad, NS G loss, D*, C(G) obj, JSD Def, KL Div, M in JSD, LSGAN D loss, LSGAN G loss, W1 Dist, Lipschitz constraint, WGAN D loss, WGAN G loss, WGAN-GP penalty, Grad Norm, cGAN D loss (concept), cGAN G loss (concept), InfoGAN obj, Mutual Info I, AdaIN, Cycle Loss, Pix2Pix Loss, L1 Loss, IS, FID, E, Tr. Total > 35).