Version of Transformers and Tokenizers

[CNGaoWenbo]

Hi,

In the evaluation of the caption task, when I upgrade the package Transformers along with the Tokenizers to a newer version (tokenizer 0.15 or >0.2), the caption results get worse. Does this mean the checkpoints only work in this specific version (tokenizer == 0.12.1)? If I want to use the latest version of Transformers, which requires the latest tokenizer, do you have any idea how I can solve this? thanks!

[RunsenXu]

Hi,

I think you need to use the same tokenizer version for both training and inference. If you used our pretrained checkpoint, it would be better to create a separate env for that.