Reproducibility problem

[victoriazinkovich]

Dear Authors,

Thank you for your outstanding work on this project. I am interested in including your model in my upcoming paper and would greatly appreciate your assistance in clarifying a few details. Could you kindly specify which part of ReasonSeg (test or val) and which model (gsva-7b-ft-res or gsva-7b-ft-gres) the table 5 from A.2. was generated from?

I am a bit concerned because, after running your evaluation script (scripts/eval_demo.sh), I obtained the following metrics for gsva-7b-ft-res on the test set: cIoU = 40.1 and gIoU = 44.8.

scripts/eval_demo.sh:

#!/bin/bash
export TRANSFORMERS_OFFLINE=1
export DS_SKIP_CUDA_CHECK=1
export CUDA_VISIBLE_DEVICES=0

ds main.py \
  --val_dataset="ReasonSeg|val" \
  --segmentation_model_path="../SAM/pretrained_weights/sam_vit_h_4b8939.pth" \
  --mllm_model_path="../LLaVA/pretrained_weights/llava-7b-v1-1" \
  --vision-tower=openai/clip-vit-large-patch14 \
  --dataset_dir=../dataset \
  --weight=pretrained_weights/gsva-7b-ft-res.bin \
  --precision=fp32 \
  --lora_r=8 \
  --eval_only 

Thank you very much for your time and assistance.

Sincerely,
Viktoriia Zinkovich

[Ttmx11]

I met the same problem. Have you solved it?
Surprisingly, the reproduction metric results of the same GSVA model on the refCOCO, refCOCO+, and refCOCOg datasets are similar to those of the paper, so why would it perform unsatisfactorily on the ReasonSeg dataset?
Thank you very much for your time and assistance.

[victoriazinkovich]

I met the same problem. Have you solved it? Surprisingly, the reproduction metric results of the same GSVA model on the refCOCO, refCOCO+, and refCOCOg datasets are similar to those of the paper, so why would it perform unsatisfactorily on the ReasonSeg dataset? Thank you very much for your time and assistance.

Unfortunately, I haven't been able to determine the exact cause of this discrepancy. For now, I'm proceeding with the model as is.