Human Operator is a human augmentation tool that allows AI to briefly take control of your body to help you learn and do things you normally cannot do. To do this, it uses a Vision-Language Model for human motor control through Electrical Muscle Stimulation (EMS). Vision-based commands are generated via open-ended speech input through the Claude API to control finger and wrist stimulation for intuitive on-body interaction.
🏆 Winning project at MIT Hard Mode 2026 (Learn Track) 🏆
Left to right: AI stimulates wrist muscle to say 'Hello' back • AI stimulates fingers in sequence to play melody • AI stimulates fingers to form 'OK' sign
- Peter He – Portfolio | GitHub | LinkedIn
- Ashley Neall – Portfolio | GitHub | LinkedIn
- Valdemar Danry – Portfolio | GitHub | LinkedIn
- Daniel Kaijzer - Portfolio | Github | Linkedin
- Yutong Wu - Portfolio | Github | Linkedin
- Sean Lewis - Portfolio | Github | Linkedin
- Python 3.10.x
- Conda, Miniconda or other environment setup
- Arduino IDE 2.3.x (for firmware setup)
- Claude API key (from Anthropic)
EMS & Control:
- TENS/EMS Gel Electrodes
- Arduino-compatible microcontroller (Arduino Micro recommended)
- 5V Relays for finger control (pinky, middle, index)
- Custom or commercial TENS/EMS unit with electronic control capability
Sensing:
- Camera module (integrated or USB) for POV video capture
-
Clone the repository:
git clone <repository-url> cd Human-Operator
-
Create Python environment:
conda env create -f environment.yml conda activate human-operator
-
Configure API key:
cp .env_empty .env # Edit .env and add your Anthropic API key: # ANTHROPIC_API_KEY=your_api_key_here
-
Upload Arduino firmware:
- Open Arduino IDE
- Load
firmware/human_operator_ems/human_operator_ems.ino - Upload to your Arduino board
- Verify serial communication works (115200 baud)
Start the real-time EMS control system:
python app.pyThis launches:
- Video capture from your camera
- AI processing via Claude API for vision-based motor commands
- Hardware control for EMS stimulation of pinky, middle, and index fingers
- Voice trigger ("Hey Operator") to activate commands
Commands are triggered by physical interaction detected in your POV video.
Test and calibrate EMS parameters with the GUI:
python utils/stimGUIfingersHardMode.pyThis tool allows you to:
- Manually trigger finger stimulation (pinky, middle, index)
- Adjust amplitude, frequency, and pulse width
- Visualize stimulation parameters in real-time
- Perfect for hardware calibration and testing
app.py (Main Application)
├── Video Capture (OpenCV)
├── Vision-Language Model (Claude API)
├── Command Planning (Motor control sequences)
└── Hardware Interface (receiver.py)
└── Flask Server (receiver.py)
└── Serial Communication → Arduino Firmware
└── Relay Control (Finger EMS)
stimGUIfingersHardMode.py (Manual Testing GUI)
└── Direct Hardware Interface (PyQt5 + Serial)
Run a ball-detection demo that triggers avoidance responses:
python utils/ball_demo.pyArduino not detected:
- Check USB cable connection
- Verify correct board selection in Arduino IDE
- Confirm serial port settings (115200 baud)
No camera detected:
- Verify camera permissions (especially on macOS)
Missing API key:
- Ensure
.envfile exists in the root directory - Verify
ANTHROPIC_API_KEYis correctly set
Inspired by research and systems from the Human Computer Integration Lab at UChicago on neuromuscular interfaces and electrode placement optimization: