-
Notifications
You must be signed in to change notification settings - Fork 6
add VL-LN Bench doc #7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
DuangZhu
wants to merge
2
commits into
InternRobotics:main
Choose a base branch
from
DuangZhu:feat/vlln-bench-guide
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+174
−0
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,4 +13,5 @@ myst: | |
|
|
||
| quick_start/index | ||
| tutorials/index | ||
| projects/index | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| # Extended Benchmarks in InternNav | ||
|
|
||
| This guide details how to use specific dataset for training a VLA model for different navigation benchmark. | ||
|
|
||
| ## VL-LN Bench | ||
|
|
||
| VL-LN Bench is a large-scale benchmark for Interactive Instance Goal Navigation. VL-LN Bench provides: (1) an automatically dialog-augmented trajectory generation pipeline, (2) a comprehensive evaluation protocol for training and assessing dialog-capable navigation models, and (3) the dataset and base model used in our experiments. For full details, see our [paper](https://arxiv.org/abs/2512.22342) and the [project website](https://0309hws.github.io/VL-LN.github.io/). | ||
|
|
||
DuangZhu marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - [Data Collection Pipeline](https://github.com/InternRobotics/VL-LN) | ||
| - [Training and Evaluation Code](https://github.com/InternRobotics/InternNav) | ||
| - [Dataset](https://huggingface.co/datasets/InternRobotics/VL-LN-Bench) and [Base Model](https://huggingface.co/InternRobotics/VL-LN-Bench-basemodel) | ||
|
|
||
|
|
||
| ### 1. Download Data & Assets | ||
| VL-LN Bench is built on Matterport3D (MP3D) Scene Dataset, so you need to download both the MP3D scene dataset and the VL-LN Bench dataset. | ||
| - Scene Datasets | ||
|
|
||
| Download the [MP3D Scene Dataset](https://niessner.github.io/Matterport/) | ||
| - [VL-LN Data](https://huggingface.co/datasets/InternRobotics/VL-LN-Bench) | ||
| - [VL-LN Base Model](https://huggingface.co/InternRobotics/VL-LN-Bench-basemodel) | ||
|
|
||
| After unzipping the base model, scene datasets, and trajectory data, put everything under VL-LN-Bench/ in the layout below. | ||
| ```bash | ||
| VL-LN-Bench/ | ||
| ├── base_model/ | ||
| │ └── iion/ | ||
| ├── raw_data/ | ||
| │ └── mp3d/ | ||
| │ ├── scene_summary/ | ||
| │ ├── train/ | ||
| │ │ ├── train_ion.json.gz | ||
| │ │ └── train_iion.json.gz | ||
| │ └── val_unseen/ | ||
| │ ├── val_unseen_ion.json.gz | ||
| │ └── val_unseen_iion.json.gz | ||
| ├── scene_datasets/ | ||
| │ └── mp3d/ | ||
| │ ├── 17DRP5sb8fy/ | ||
| │ ├── 1LXtFkjw3qL/ | ||
| │ ... | ||
| └── traj_data/ | ||
| ├── mp3d_split1/ | ||
| ├── mp3d_split2/ | ||
| └── mp3d_split3/ | ||
| ``` | ||
|
|
||
| ### 2. Environment Setup | ||
| Here we set up the Python environment for VL-LN Bench and InternVLA-N1. If you've already installed the InternNav Habitat environment, you can skip those steps and only run the commands related to VL-LN Bench. | ||
|
|
||
| - Get Code | ||
| ```bash | ||
| git clone git@github.com:InternRobotics/VL-LN.git # code for data collection | ||
| git clone git@github.com:InternRobotics/InternNav.git # code for training and evaluation | ||
| ``` | ||
|
|
||
| - Create Conda Environment | ||
| ```bash | ||
| conda create -n vlln python=3.9 -y | ||
| conda activate vlln | ||
| ``` | ||
|
|
||
| - Install Dependencies | ||
| ```bash | ||
| conda install habitat-sim=0.2.4 withbullet headless -c conda-forge -c aihabitat | ||
| cd VL-LN | ||
| pip install -r requirements.txt | ||
| cd ../InternNav | ||
| pip install -e . | ||
| ``` | ||
|
|
||
| ### 3. Guidance for Data Collection | ||
| This step is optional. You can either use our collected data for policy training, or follow this step to collect your own training data. | ||
|
|
||
|
|
||
| - Prerequisites: | ||
| - Get pointnav_weights.pth from [VLFM](https://github.com/bdaiinstitute/vlfm/tree/main/data) | ||
| - Arrange the Directory Structure Like This | ||
| ```bash | ||
| VL-LN | ||
| ├── dialog_generation/ | ||
| ├── images/ | ||
| ├── VL-LN-Bench/ | ||
| │ ├── base_model/ | ||
| │ ├── raw_data/ | ||
| │ ├── scene_datasets/ | ||
| │ ├── traj_data/ | ||
| │ └── pointnav_weights.pth | ||
| ... | ||
| ``` | ||
|
|
||
| - Collect Trajectories | ||
| ```bash | ||
| # If having slurm | ||
| sbatch generate_frontiers_dialog.sh | ||
|
|
||
| # Or directly run | ||
| python generate_frontiers_dialog.py \ | ||
| --task instance \ | ||
| --vocabulary hm3d \ | ||
| --scene_ids all \ | ||
| --shortest_path_threshold 0.1 \ | ||
| --target_detected_threshold 5 \ | ||
| --episodes_file_path VL-LN-Bench/raw_data/mp3d/train/train_iion.json.gz \ | ||
| --habitat_config_path dialog_generation/config/tasks/dialog_mp3d.yaml \ | ||
| --baseline_config_path dialog_generation/config/expertiments/gen_videos.yaml \ | ||
| --normal_category_path dialog_generation/normal_category.json \ | ||
| --pointnav_policy_path VL-LN-Bench/pointnav_weights.pth\ | ||
| --scene_summary_path VL-LN-Bench/raw_data/mp3d/scene_summary\ | ||
| --output_dir <PATH_TO_YOUR_OUTPUT_DIR> \ | ||
| ``` | ||
|
|
||
| ### 4. Guidance for Training and Evaluation | ||
| Here we show how to train your own model for the IIGN task and evaluate it on VL-LN Bench. | ||
|
|
||
| - Prerequisites | ||
| ```bash | ||
| cd InternNav | ||
| # Link VL-LN Bench data into InternNav | ||
| mkdir projects && cd projects | ||
| ln -s /path/to/your/VL-LN-Bench ./VL-LN-Bench | ||
| ``` | ||
| - Write your OpenAI API key to api_key.txt. | ||
| ```bash | ||
| # Your final repo structure may look like | ||
| InternNav | ||
| ├── assets/ | ||
| ├── internnav/ | ||
| │ ├── habitat_vlln_extensions | ||
| │ │ ├── simple_npc | ||
| │ │ │ ├── api_key.txt | ||
| │ ... ... ... | ||
| ... | ||
| ├── projects | ||
| │ ├── VL-LN-Bench/ | ||
| │ │ ├── base_model/ | ||
| │ │ ├── raw_data/ | ||
| │ │ ├── scene_datasets/ | ||
| │ │ ├── traj_data/ | ||
| ... ... | ||
| ``` | ||
|
|
||
| - Start Training | ||
| ```bash | ||
| # Before running, please open this script and make sure | ||
| # the "llm" path points to the correct checkpoint on your machine. | ||
| sbatch ./scripts/train/qwenvl_train/train_system2_vlln.sh | ||
| ``` | ||
|
|
||
| - Start Evaluation | ||
| ```bash | ||
| # If having slurm | ||
| sh ./scripts/eval/bash/srun_eval_dialog.sh | ||
DuangZhu marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # Or directly run | ||
| python scripts/eval/eval.py \ | ||
| --config scripts/eval/configs/habitat_dialog_cfg.py | ||
| ``` | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add benchmark results. Please follow the readme for each model in mmdet or InternNav readme to add more reference for reimplementing a work. |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| --- | ||
| myst: | ||
| html_meta: | ||
| "description lang=en": | | ||
| Documentation for users who wish to build sphinx sites with | ||
| pydata-sphinx-theme. | ||
| --- | ||
|
|
||
| # Projects | ||
|
|
||
| ```{toctree} | ||
| :caption: Projects | ||
| :maxdepth: 2 | ||
| benchmark | ||
| ``` |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
benchmarks