DeepLearning

Deep Learning Coursework for software, systems and applications 3rd year module

Objective

• Using the CIFAR-10 dataset, create a NN model capable of generating an image of a pegasus
• An intentionally ill-defined learning problem; the generative model needs to use what it learns about birds and horses in order to produce the pegasus image

To-Do

• Lecture catchup (especially lecture 4)
• Research generative architectures, produce list of papers / interesting articles etc.
• Select a generative model to use
• Build and train model
• Generate pegasus image
• Tweak and refine hyperparameters etc., keep track of results and details of improvements
• Write report
  • Diagram of model, dicussions on its design (40 marks)
  • Include best output (40 marks)
  • Include batch of 64 images (20 marks)

Notes

Lecture 3 Catchup

• Variety of potential architectures explained in lecture 3
  • GANs look promising
• Batch normalisation explained here

Lecture 4 Catchup

• Generative models are effectively the reverse of classifcation models, i.e. P(X|Y=y) instead of P(Y|X=x)
• KDE explained nicely here
• Lots of useful statisitcal definitions, may be worth coming back to when reading papers in the future
• Dicsussion of loss functions
  • Generally a good idea to simply use L_2 as it is convenient, simple and computationally inexpensive
• GANs (Generative adversarial networks) use a game-theoretic approach to the generative problem, by competing two models against one another:
  • Discriminator, D, which estimates the probability of a given sample coming from the real dataset or being a generated sample
  • A generator, G, that learns to map noise drawn from some prior distribution (often Gaussian) to generated examples that capture p_data
  • This creates a two-player minimax game, where one player is trying to minimize G and the other is trying to maximise D. When they reach a Nash equilibrium this is when the GAN model converges.

Lecture 5 Catchup - GANs and ACAIs

• GANs are notoriously difficult to train, they often demonstrate extreme instability
• ACAIs have a strong potential of being very useful, more detailed explanation on them can be found here

ACAIs

• ACAI paper here
• ACAI paper notes here
• ACAI implementation in tensorflow can be found here

GANs

How to train a GAN video facebook AI here video

Training tips

• How to train a GAN Facebook AI
• GAN architectures for beginners
• More on GAN architectures

DCGANs

• Paper on Deep Convolutional Generative Adversarial Networks here
• Architecture guidelines:
  • Replace pooling layers with strided convolutions (discriminator) and fractional-strided convolutions (generator)
  • Use batch normalisation in both the generator and discriminator
  • Remove fully connected hidden layers for deeper architectures
  • Use ReLU activation in the generator for all layers except for the output, which uses Tanh
  • Use LeakyReLU activation in the discriminator

Conditional GANs

• Paper on conditional GANs can be found here

Normalisation techniques

• Spectral normalisation increases training stability, paper on it here
  • Code here

WGANs

• WGAN paper found here

Spherical space sampling

• Latent space sampling paper found here