Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulary Scene Graph Generation

Official Implementation of
"Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulary Scene Graph Generation"

Setup

For simplicity, you can directly run:

bash install.sh

which includes the following steps:

0. Install PyTorch 1.9.1 and other dependencies:

pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt

(Adjust CUDA version if necessary.)

1. Install GroundingDINO:

cd GroundingDINO && python3 setup.py install

---

Dataset

Prepare the dataset under data/ folder following the instruction.

---

Pretrain SGG

Generate SG Caption

bash scripts/gen_sg_triplets.sh

Generate Pseudo SGG Annotations

bash scripts/gen_pseudo_triplets.sh

Training

bash scripts/train.sh

Adjust CUDA_VISIBLE_DEVICES if needed. Effective batch size = batch size × number of GPUs.

Inference

bash scripts/DINO_eval.sh vg [config file] [data path] [output path] [checkpoint]

---

Checkpoints

The checkpoints are released at here.

Acknowledgement

We thank:

• Scene-Graph-Benchmark.pytorch
• GroundingDINO
• OvSGTR for their awesome open-source codes and models.

---

Citation

If you find our work helpful, please cite:

@inproceedings{chen2024expanding,
  title={Interaction-Centric Knowledge Infusion and Transfer for Open-Vocabulary Scene Graph Generation},
  author={Li, Lin and Zhang, Chuhan and Zhang, Dong and Sun, Chong and Li, Chen and Chen, Long},
  booktitle={NeurIPS},
  year={2025}
}