真机松灵机械臂数采

[fengsvkn]

真机进行数采，数据转换，训练的流程是什么样子的？
可否提供一些真机的叠衣服数据集？
未来是否支持lerobot格式数据集直接训练？

[jin-yiwei]

Hi, @fengsvkn.

1. The data collection can be referenced https://github.com/HorizonRobotics/RoboOrchard/tree/master/projects/HoloBrain, https://horizonrobotics.github.io/robot_lab/holobrain/real_env/modules/data_collection.html;
2. The training process can be referenced https://github.com/HorizonRobotics/RoboOrchardLab/tree/master/projects/holobrain;

[AlanHuangT]

@fengsvkn robo_orchard_lab 并不强制用户使用RODataset， 实际上用户可以用任意的dataset做训练， 只要自己适配好Dataset . 对于 HoloBrain提供的pipeline， 不会提供lerobot数据。 有需要可以用户自己转换。

[fengsvkn]

HorizonRobotics/RoboOrchard#1
真机上也遇到了一些问题，卡到手眼标定这里了

[fengsvkn]

HoloBrain提供的pipeline在哪里？我已经把lerobot格式的数据集转成RODataset格式，手眼标定也完成了。
我只看到了robotwin格式的数据集的配置，有没有RODataset格式数据集的配置。

[AlanHuangT]

@linxuewu 一作出来解答一下。

[linxuewu]

HoloBrain提供的pipeline在哪里？我已经把lerobot格式的数据集转成RODataset格式，手眼标定也完成了。 我只看到了robotwin格式的数据集的配置，有没有RODataset格式数据集的配置。

你好。目前rodataset的代码还没release，预计这几天会放出来。（现在如果想训，只能打包成lmdb dataset。）

[fengsvkn]

@linxuewu 你好，rodataset训练holobrain的代码预计什么时候放出来？

[Muyun99]

@linxuewu 你好，rodataset训练holobrain的代码预计什么时候放出来？

你好，我们这周会对 rodataset 使用的接口做一些合并，可以先看下这里的单元测试的 dataset 拿数据的流程哈，如果着急的话需要对 config 的 dataset 部分自行做一些修改

• link

[crayonsea]

@linxuewu 你好，rodataset训练holobrain的代码预计什么时候放出来？

你好，ro dataset 训练 holobrain 的代码已经更新，可以参考数据配置文件 projects/holobrain/configs/config_agilex_ro_dataset.py，替换自己的 RO 数据路径即可

[fengsvkn]

真机训练跑不通，遇到了很多问题，最后卡到了预训练模型与Qwen2.5VL模型不一致（flash-attention环境配不好，我给关了）
(RoboOrchardLab) root@hhws-System-Product-Name:/data/RoboOrchardLab/projects/holobrain# python3 train.py --config ${CONFIG}
/data/RoboOrchardLab/projects/holobrain/train.py:221: DeprecationWarning: Call to deprecated function (or staticmethod) basic_config. (Use robo_orchard_core.utils.logging module instead.) -- Deprecated since version 0.2.0.
log_basic_config(
Rank[0/1] 03/13/2026 15:47:36 INFO train.py:225 | if accelerator initialized:True
Rank[0/1] 03/13/2026 15:47:36 INFO train.py:226 | accelerator state: {'_cpu': False, 'backend': None, 'device': device(type='cuda'), 'debug': False, 'distributed_type': <DistributedType.NO: 'NO'>, 'num_processes': 1, 'process_index': 0, 'local_process_index': 0, 'fork_launched': False, 'deepspeed_plugins': None, 'torch_tp_plugin': None, 'parallelism_config': None, 'device_mesh': None, 'dynamo_plugin': TorchDynamoPlugin(backend=<DynamoBackend.NO: 'NO'>, mode='default', fullgraph=False, dynamic=None, options=None, disable=False, use_regional_compilation=False), '_mixed_precision': 'no'}
Rank[0/1] 03/13/2026 15:47:36 INFO train.py:88 |
{
"hist_steps": 1,
"pred_steps": 64,
"chunk_size": 4,
"embed_dims": 256,
"with_depth": false,
"with_depth_loss": false,
"min_depth": 0.01,
"max_depth": 1.2,
"num_depth": 128,
"batch_size": 16,
"max_step": 100000,
"step_log_freq": 50,
"save_step_freq": 5000,
"num_workers": 16,
"lr": 0.0001,
"training_with_subtask": false,
"with_cot": false,
"training_datasets": [
"piper_clothes_or"
],
"deploy_datasets": [
"piper_clothes_or"
],
"vlm_pretrain": "/data/models/Qwen2.5-VL-3B-Instruct",
"num_vlm_layers": 1,
"freeze_vlm": false,
"checkpoint": "/data/models/HoloBrain_v0.0_Qwen/pretrain/model.safetensors"
}
Rank[0/1] 03/13/2026 15:47:36 WARNING action_decoder.py:250 | Use default teacher_forcing_mean_steps: 16
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 824/824 [00:00<00:00, 7721.99it/s]
Rank[0/1] 03/13/2026 15:47:37 WARNING structure.py:118 | Deprecated, please use transformers >= 4.57.1.
/data/miniforge3/envs/RoboOrchardLab/lib/python3.10/site-packages/datasets/utils/experimental.py:37: UserWarning: 'register_feature' is experimental and might be subject to breaking changes in the future.
warnings.warn(
Loading dataset from disk: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 8244.33it/s]
Rank[0/1] 03/13/2026 15:47:40 INFO dataset_wrapper.py:72 | dataset length: 58493, number of batches: 3655, dataset flags: [35574]
Rank[0/1] 03/13/2026 15:47:40 INFO train.py:121 | number of parameters: 1079.75M, trainable: 411.07M, non-trainable: 668.69M
Rank[0/1] 03/13/2026 15:47:40 INFO utils.py:111 | load checkpoint: /data/models/HoloBrain_v0.0_Qwen/pretrain/model.safetensors
/data/RoboOrchardLab/robo_orchard_lab/models/holobrain/structure.py:414: FutureWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
state_dict = super().state_dict(*args, destination, prefix, keep_vars)
Traceback (most recent call last):
File "/data/RoboOrchardLab/projects/holobrain/train.py", line 228, in
main(args, accelerator)
File "/data/RoboOrchardLab/projects/holobrain/train.py", line 130, in main
load_checkpoint(model, config.get("checkpoint"), accelerator)
File "/data/RoboOrchardLab/projects/holobrain/utils.py", line 114, in load_checkpoint
missing_keys, unexpected_keys = load_model(
File "/data/miniforge3/envs/RoboOrchardLab/lib/python3.10/site-packages/safetensors/torch.py", line 224, in load_model
missing, unexpected = model.load_state_dict(state_dict, strict=False)
File "/data/RoboOrchardLab/robo_orchard_lab/models/holobrain/structure.py", line 424, in load_state_dict
return super().load_state_dict(state_dict, strict=strict, **kwargs)
File "/data/miniforge3/envs/RoboOrchardLab/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2635, in load_state_dict
raise RuntimeError(
RuntimeError: Error(s) in loading state_dict for HoloBrain_Qwen2_5_VL:
size mismatch for spatial_enhancer.fusion_fc.0.layers.0.0.weight: copying a param with shape torch.Size([1024, 512]) from checkpoint, the shape in current model is torch.Size([1024, 384]).
size mismatch for spatial_enhancer.fusion_fc.0.layers.1.weight: copying a param with shape torch.Size([512, 1024]) from checkpoint, the shape in current model is torch.Size([384, 1024]).
size mismatch for spatial_enhancer.fusion_fc.0.layers.1.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([384]).
size mismatch for spatial_enhancer.fusion_fc.1.weight: copying a param with shape torch.Size([256, 512]) from checkpoint, the shape in current model is torch.Size([256, 384]).

[fengsvkn]

RoboOrchardLab.zip

[fengsvkn]

真机手眼标定的数据也不知道放到哪里。

[jin-yiwei]

@fengsvkn

1. flash-attn 可以用 whl 安装：https://github.com/Dao-AILab/flash-attention/releases/download/fa4-v4.0.0.beta4/flash_attn_4-4.0.0b4-py3-none-any.whl
2. 手眼标定的数据具体在 config 的 calibration 字段，通过 config 的 dataset_config calibration 字段传入

[fengsvkn]

我就是用的这个whl
这是报错信息：
Rank[0/1] 03/13/2026 15:37:09 WARNING action_decoder.py:250 | Use default teacher_forcing_mean_steps: 16 Traceback (most recent call last): File "/data/RoboOrchardLab/projects/holobrain/train.py", line 228, in main(args, accelerator) File "/data/RoboOrchardLab/projects/holobrain/train.py", line 90, in main model = build_model(config) File "/data/RoboOrchardLab/projects/holobrain/configs/config_holobrain_qwen_common.py", line 130, in build_model model = model_class( File "/data/RoboOrchardLab/robo_orchard_lab/models/holobrain/structure.py", line 105, in init self.vlm = Qwen2_5_VLForConditionalGeneration.from_pretrained( File "/data/miniforge3/envs/RoboOrchardLab/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4094, in from_pretrained model = cls(config, *model_args, **model_kwargs) File "/data/miniforge3/envs/RoboOrchardLab/lib/python3.10/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1451, in init super().init(config) File "/data/miniforge3/envs/RoboOrchardLab/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1260, in init self.config._attn_implementation_internal = self._check_and_adjust_attn_implementation( File "/data/miniforge3/envs/RoboOrchardLab/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1893, in _check_and_adjust_attn_implementation lazy_import_flash_attention(applicable_attn_implementation) File "/data/miniforge3/envs/RoboOrchardLab/lib/python3.10/site-packages/transformers/modeling_flash_attention_utils.py", line 171, in lazy_import_flash_attention _flash_fn, _flash_varlen_fn, _pad_fn, _unpad_fn = _lazy_imports( File "/data/miniforge3/envs/RoboOrchardLab/lib/python3.10/site-packages/transformers/modeling_flash_attention_utils.py", line 96, in _lazy_imports from flash_attn import flash_attn_func, flash_attn_varlen_func ImportError: cannot import name 'flash_attn_func' from 'flash_attn' (unknown location)

dataset_config calibration 字段只能传eye in hand的数据

high相机有两个标定的数据呢？传入eye to hand的标定，论文里面也没看懂