Details on training of published models

[bminixhofer]

Hi! I really enjoyed your paper and the code. Do you have some more details regarding the published Italian and Dutch models?

Specifically:

• for how many steps did you train only the embeddings?
• for how many steps did you train the full model?
• at which batch size / learning rate?

Sorry if this is already mentioned somewhere in the Repo or the paper, I wasn't able to find it. I am trying to use some of your results in my work, so this information would help a lot :) Thanks!

[wietsedv]

Hi, thanks for your interest!

I deliberately did not report number of training steps because it highly depends on your data and language. In our opinion, the correct strategy is to make a selection of validation data that is not part of the training data and monitor the loss value on that data. Stop training if it stops decreasing. Learning rate is similar, but you can aim at the order of magnitude of 1e-5 to 1e-4.