Simlab

A simulation platform for autonomous laboratory environments that enables research into fully closed-loop scientific experimentation at massive scales.

This PhD research project integrates Isaac Sim (NVIDIA's 3D simulation software) with MADSci (Argonne National Labotory's experiment orchestration software) to investigate how autonomous agents can convert scientific protocols into executable robot commands and handle execution failures. By using MADSci, protocols run with the same execution and orchestration software that operates real laboratory robots, ensuring direct transferability between simulation and physical systems.

Research Focus:

• Protocol automation
• Failure handling
• Scalable laboratory simulation

Key Capabilities

• Isaac Sim Integration: Full 3D simulation environment for laboratory robotics with physics-based interactions
• Isaac Lab Integration: Reinforcement learning framework for robot control
• cuRobo Integration: GPU-accelerated collision-aware motion planning
• MADSci Integration: Laboratory experiment orchestration with standardized device interfaces
• ZMQ Communication: Real-time bidirectional communication between simulation and orchestration systems
• Protocol Conversion: Autonomous agent-based conversion of scientific protocols into executable robot workflows
• Scalable Architecture: Designed for warehouse-scale laboratory automation research

Quick Start

Environment Setup

This project uses two separate UV-managed virtual environments.

Versions installed by setup-isaacsim.sh:

Component	Version
Python	3.11
Isaac Sim	5.1.0
Isaac Lab	2.3.1
cuRobo	0.7.7
OpenUSD tools	24.11

Versions installed by setup-madsci.sh:

Component	Version
Python	3.12
MADSci	0.6.0
Opentrons	8.5.1

Prerequisites:

sudo apt install git-lfs  # Required for cuRobo

Automated setup (recommended for first-time setup):

./setup-isaacsim.sh  # Creates Isaac Sim environment + Isaac Lab + cuRobo + USD tools
./setup-madsci.sh    # Creates MADSci environment + Opentrons patches

Activate environments:

source activate-isaacsim.sh
source activate-madsci.sh

Note: The activation scripts validate that the virtual environment exists before activating. If the venv is missing, you'll see an error message directing you to run the setup script first.

Upgrading Dependencies

To upgrade Isaac Sim version:

1. Edit requirements-isaacsim.in and update isaacsim[all,extscache]==X.X.X
2. Run:

   source activate-isaacsim.sh
   uv pip compile requirements-isaacsim.in --extra-index-url https://pypi.nvidia.com -o requirements-isaacsim.txt --upgrade
   uv pip sync requirements-isaacsim.txt
   deactivate

To upgrade MADSci version:

1. Edit requirements-madsci.in and update madsci-*==X.X.X versions
2. Run:

   source activate-madsci.sh
   uv pip compile requirements-madsci.in --override overrides-madsci.txt -o requirements-madsci.txt --upgrade
   uv pip sync requirements-madsci.txt
   deactivate

Running the System

The orchestrator coordinates startup and shutdown of all system components:

# Full system with REST Gateway
python tools/orchestrate.py \
    --isaac-cmd "source activate-isaacsim.sh && cd projects/my-project && python run.py" \
    --gateway-cmd "source activate-madsci.sh && python -m slcore.gateway.rest_gateway --num-envs 5" \
    --madsci-cmd "cd projects/my-project/madsci/ && ./run_madsci.sh" \
    --workflow-cmd "source activate-madsci.sh && cd projects/my-project && python run_workflow.py workflow.yaml"

# Minimal (Isaac Sim + direct test script, no MADSci)
python tools/orchestrate.py \
    --isaac-cmd "source activate-isaacsim.sh && cd projects/my-project && python run.py" \
    --workflow-cmd "source activate-madsci.sh && python my_test_script.py"

All output is logged to /tmp/simlab/<timestamp>/. The --gateway-cmd and --madsci-cmd arguments are optional for simpler testing scenarios.

Architecture Overview

Core Components

• Isaac Sim Integration (slcore/common/): Shared utilities, simulation scripts, and ZMQ communication interface
• MADSci Integration (projects/<project>/madsci/): Per-project laboratory orchestration configuration and services
• Robot Modules (slcore/robots/): Per-robot directories containing both Isaac Sim ZMQ servers and MADSci node files

Communication Pattern

The system uses ZMQ ROUTER-DEALER pattern for Isaac Sim ↔ MADSci communication:

• Isaac Sim: Runs a single ZMQ ROUTER server on port 5555
• Robot nodes: Connect as DEALER clients with identity-based routing (e.g., env_0.pf400, env_1.thermocycler)
• Multiplexing: Multiple robot instances share a single ZMQ port, with routing based on client identity

REST Gateway (Simulation):

For simulation, the REST Gateway consolidates all robot REST nodes into a single FastAPI process with path-based routing. This replaces individual node processes/ports with a single gateway on port 8000.

MADSci Workcell Manager
    |
    | HTTP POST /env_0/pf400/action/transfer
    v
REST Gateway (port 8000) --> ZMQ DEALER --> Isaac Sim ZMQ ROUTER (port 5555)

Workcell configs use path-based URLs: http://127.0.0.1:8000/env_0/pf400 instead of individual ports per robot.

Workflow Pattern

1. Define laboratory layout in YAML (robots, locations, resources)
2. Define workflow steps in YAML (robot actions, parameters)
3. Submit workflow to MADSci workcell manager
4. MADSci coordinates robot nodes to execute workflow steps
5. Robot nodes communicate with Isaac Sim via ZMQ for physical simulation

Project Structure

simlab/
├── .venv-isaacsim/   # Isaac Sim environment (includes Isaac Lab, cuRobo, and USD tools)
├── .venv-madsci/     # MADSci environment (includes patched Opentrons)
├── IsaacLab/         # Isaac Lab (cloned by setup-isaacsim.sh)
├── curobo/           # cuRobo (cloned by setup-isaacsim.sh)
├── slcore/           # Python package (import as: from slcore.robots.common import ...)
│   ├── common/           # Shared utilities (utils.py, primary_functions.py)
│   ├── gateway/          # REST Gateway for consolidated robot node routing
│   └── robots/           # Per-robot directories
│       ├── common/           # Shared robot utilities (ZMQ base classes, config)
│       ├── ur5e/             # UR5e arm
│       ├── ot2/              # Opentrons OT-2 liquid handler
│       ├── pf400/            # Brooks PF400 plate handler
│       ├── hidex/            # Hidex plate reader
│       ├── sealer/           # Plate sealer
│       ├── peeler/           # Plate peeler
│       └── thermocycler/     # Thermocycler
├── assets/           # 3D simulation assets
│   ├── scenes/           # Laboratory layouts
│   ├── architecture/     # Structural elements (walls, floors, ceilings)
│   ├── robots/           # Robot models
│   ├── labware/          # Experimental apparatus (tips, plates, reagents)
│   ├── props/            # Environmental objects
│   └── tools/            # Asset processing tools (.blend to .usd conversion)
├── docs/             # Documentation and examples
│   └── examples/         # Code snippets and reference examples
├── forks/            # Patches for third-party libraries (applied during setup)
│   └── *.patch           # Unified diff patches for opentrons simulation support
├── tools/            # User tools and utilities for the command line
│   └── usd/              # USD command-line tools built from source
├── projects/         # Self-contained experimental projects
│   ├── _template/        # Template for new projects (madsci/, workflow.yaml, run_workflow.py)
│   ├── scaling-mvp/      # Parallel environment testing (5 simultaneous workcells)
│   └── [custom]/         # Custom projects, each with optional madsci/ subdirectory
├── specs/            # Specifications, design docs, and issue reports
├── requirements-isaacsim.in  # Isaac Sim dependencies
├── requirements-madsci.in    # MADSci dependencies
└── activate-{isaacsim,madsci}.sh  # Environment activation scripts

Contributing & Collaboration

Research Collaboration

This project supports research into:

• Autonomous agent-based protocol conversion, generation, and execution
• Failure handling and recovery in automated experimentation
• Scalable simulation for laboratory automation

Researchers interested in collaboration on laboratory automation, autonomous agent-driven experimentation, or related areas are welcome to reach out.

Technical Contributions

• Issue Reports: Bug reports, feature requests, and documentation improvements
• Protocol Examples: Additional scientific protocol implementations
• Device Integration: New robot or laboratory device integrations
• Simulation Assets: Laboratory environments, robot models, or consumables