Hello author, when I was loading the pre trained GSVA model, it showed that 5 new tokens were added, but only 4 tokens were added in the paper and code. What does this extra token represent

[zgyustc]

Hello author, when I was loading the pre trained GSVA model, it showed that 5 new tokens were added, but only 4 tokens were added in the paper and code. What does this extra token represent

[ShadowStrikess]

have you run this project in your system?

[Royxia0818]

Same problem. The extra tokens are added in model/prepare_model_tokenizer.py, line 14:

def add_task_tokens(tokenizer, args):
    # 1. pad_token set to unknown token
    tokenizer.pad_token = tokenizer.unk_token
    # 2. add a [SEG] and a [REJ] token
    tokenizer.add_tokens("[SEG]")
    args.seg_token_idx = tokenizer("[SEG]", add_special_tokens=False).input_ids[0]
    tokenizer.add_tokens("[REJ]")
    args.rej_token_idx = tokenizer("[REJ]", add_special_tokens=False).input_ids[0]
    # 3. add <im_start> and <im_end>, same as llava into tokenizer
    if args.use_mm_start_end:
        tokenizer.add_tokens([DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN], special_tokens=True)
    return tokenizer, args

Here, len(tokenizer)=32004, while the error goes like:

Traceback (most recent call last):
File "GSVA/main.py", line 377, in
main()
File "GSVA/main.py", line 157, in main
model.load_state_dict(state_dict, strict=False)
File ".conda/envs/lisa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
size mismatch for base_model.model.model.embed_tokens.weight: copying a param with shape torch.Size([32005, 4096]) from checkpoint, the shape in current model is torch.Size([32004, 4096]).
size mismatch for base_model.model.lm_head.weight: copying a param with shape torch.Size([32005, 4096]) from checkpoint, the shape in current model is torch.Size([32004, 4096]).