Problems & Solutions

ConvNet outputs constant value after ~2 epochs

I augmented the CARPK dataset (aug16) and retrained FCRN-A network, expecting to improve the performance on car counting task. I noticed a weird behavior when the network was trained more than 2 epochs (2 epochs ~ 2*16 initial epochs, because the dataset has been increased 16 times): it outputs a constant value.

Some details:

• Optimizer: Adam(lr=1e-3);
• Activation function: ReLU;
• Initializer: He_Normal;

It seems that the constant output (for each input) is caused by the "dying ReLU" problem.
Activation/feature maps from the 7th layer of the network contains only zeros. This problem is likely to occur when the learning rate is too high or there is a large negative bias.
Some ideas to solve this problem:

• Use LeakyReLU instead of ReLU;
• Change the initialization method of the weights;
• Try Cyclic Learning Rate.