Effect of rnn layers num

[xingyueye]

@BriansIDP Hi~ Firstly, appreciate your kindly sharing. I have some questions about this implementation, and hope for your replies.

1. I notice that the default "layerlist" is "0", and I wonder if numbers of rnn layers effect a lot? Have you tried other settings, such as add rnn before each attention block.
2. RNN layers are builded by "nn.LSTM(d_model, rnndim, 1)". I guess if we should additionally set the "batch_first" options to "True". For the first dim of tensor is "batch" in Transformer, but I have not found Reshape operations before rnn_forward.

[BriansIDP]

Hi. Thank you for asking.

1. layerlist is a hyper-parameter and I used 0 in the paper. I experimented a bit with other values and those gave me similar results. For convenience I set it to be 0. Potentially it could have multiple layers as well, but I did not have the resource to perform thorough experiments.
2. We do not need to set batch_first because the "core_out" variable throughout the calculation is of dimension [seqlen, bsz, other_dims]. Please check the forward() function of RelLearnableMultiHeadAttn() class to see detailed computation.
   Hope these help.

[xingyueye]

I see ... Thanks a lot.

[xingyueye]

@BriansIDP Sorry to bother you again. I have experienced RTLM arch on my task, But it is hard to train. Would you kindly share some training tricks you used in your work.

[BriansIDP]

Hi,
I did not use any specific trick. Following my training hyper-parameter on SWB you should get similar perplexity to those in my paper. Could you please provide the training log and hyper-parameters you used?

[xingyueye]

I've tried to add LSTM on pretrained MobileBERT and to finetune on SQuADv1. Thus the hyper-parameter and logs are not similar as yours. I'm not sure the reason why my experiments perform terribly, wheather the training process, or some wrong in my code, or that this arch is not suitable for this task,,,

[BriansIDP]

Have you tried pre-training the LSTM as well? or to pre-train LSTM together with BERT? I am not sure because it sounds like quite a different task. It feels like you have a quite well-trained model, but sending random input to it (if you have LSTM before Transformer blocks). It basically has to learn everything again with your small amount of data. Maybe you could try to freeze the parameters for the MobileBERT, and only train the LSTM part on the language modelling task for some epochs to see if the criterion improves, and then finetune everything on SQuAD.

[xingyueye]

Yeah, I'd like to freeze the BERT part and train LSTM firstly. Thanks a lot for your advice.