Transform your Balatro gameplay with JokerNet 🤖✨
JokerNet represents a foundational framework for AI-powered game automation that can be extended far beyond Balatro. This section outlines potential enhancements and broader applications that demonstrate the scalability and versatility of the current multiagent architecture. 🤖✨
The system integrates Azure OpenAI for intelligent reasoning, Docker for scalable deployment, and custom mods for enhanced game interaction, making it a comprehensive solution for automated gaming.
Built with state-of-the-art AI frameworks like LangChain and LangGraph, this project exemplifies expertise in AI agent orchestration, full-stack development, and containerized gaming environments. As a software engineer specializing in AI and automation, I've crafted JokerNet to demonstrate innovative approaches to game AI - from sophisticated multiagent coordination to real-time visual analysis and precise control simulation.
- 🚀 Key Features
- 🏗️ System Architecture
- 🤖 Multiagent System with LangGraph
- 🌐 Streamlit Web Interface
- 🐳 Docker Environment: Balatro with Enhanced Automation
⚠️ Mouse Controller Status- 📦 Installation and Setup
- 📁 Project Structure
- 🔧 Configuration
- 🚀 Future Work & Extensions
- 📄 License
| 🎯 AI-Powered Gameplay | 🎮 Dual Control Methods | 👁️ Real-Time Vision |
|---|---|---|
| Multiagent orchestration using LangGraph for coordinated strategic decisions | Gamepad control (primary) + Mouse control (under development) | Computer vision-powered game state recognition and dynamic planning |
| Learn more → | See interface → | Explore architecture → |
| 🌐 Modern Web UI | 🐳 Containerized | 🔧 Custom Mods |
|---|---|---|
| Responsive Streamlit interface for configuration and real-time monitoring | Fully containerized Balatro environment with noVNC remote access | Enhanced game experience with auto-start and automation features |
| View interface → | Setup guide → | Mod details → |
- 🔗 MCP Integration: Model Context Protocol server for seamless AI tool integration
- 🌐 REST API: Comprehensive HTTP endpoints for programmatic control
- 🧠 Azure OpenAI Integration: Advanced reasoning capabilities for strategic gameplay
- ⚡ Performance Optimized: GPU support, model caching, and low-latency controls
JokerNet operates through a sophisticated multi-layered architecture designed for maximum scalability and modularity:
graph TB
subgraph "User Interface Layer"
A[Streamlit Interface<br/>Port 8501]
end
subgraph "AI Orchestration Layer"
C[Multiagent System<br/>LangGraph]
end
subgraph "🐳 Docker Container<br/>Complete Game Environment"
subgraph "API Services"
F[REST API<br/>Port 8000]
G[MCP Server<br/>Port 8001]
end
subgraph "Game Runtime"
B[noVNC<br/>Port 6080]
D[Balatro Game<br/>Love2D Engine]
E[Custom Mods<br/>BalatroLogger]
end
subgraph "Display System"
H[Xvfb<br/>Virtual Display]
I[x11vnc<br/>VNC Server]
end
B --> I
I --> H
H --> D
F --> D
G --> D
E --> D
end
A --> F
A --> B
A --> C
C --> A
C --> G
style A fill:#e1f5fe
style C fill:#f3e5f5
style D fill:#e8f5e8
style F fill:#fff3e0
Architecture Highlights:
- 🔄 Multi-layered Design: Clear separation between external UI, AI orchestration, and containerized game environment
- 🐳 Docker Encapsulation: All game execution components (display, APIs, game engine, mods) are contained within a single Docker image for easy deployment and portability
- 📡 Service Mesh: Streamlit interfaces with API, VNC, and multiagent system independently
- 🛡️ Fault Tolerance: Isolated components with graceful error handling
- ⚡ Performance: Optimized for responsive game automation and AI processing
- 🔗 Dual Control Paths: Both REST API and MCP server provide game control capabilities
- 🔄 Game Agnostic Design: With minimal modifications, the same Docker environment can support different games by swapping the game engine and mods
- 🔗 Bidirectional Communication: Multiagent system communicates directly with both Streamlit interface and MCP server for seamless coordination
For detailed agent orchestration, see Multiagent System
At the heart of JokerNet lies a sophisticated multiagent system built using LangGraph, a powerful framework for orchestrating complex AI workflows. This system intelligently decomposes the challenging task of autonomous Balatro gameplay into coordinated, specialized agents that collaborate seamlessly.
The multiagent architecture features three specialized agents, each with distinct responsibilities:
| Agent | Role | Specialization | Key Functions |
|---|---|---|---|
| 🧠 Planner Agent | Strategic Director | High-level decision making | Task decomposition, strategic planning |
| 👁️ Visualizer Agent | Vision Specialist | Computer vision analysis | Screenshot processing, game state extraction |
| ⚙️ Worker Agent | Execution Expert | Action implementation | Gamepad control (primary), mouse interaction (under development) |
---
config:
flowchart:
curve: linear
---
graph TD;
__start__([<p>__start__</p>]):::first
planner_visualizer(planner_visualizer)
planner(planner)
worker_visualizer(worker_visualizer)
worker(worker)
tool(tool)
output(output)
__end__([<p>__end__</p>]):::last
__start__ --> planner_visualizer;
planner -. end .-> output;
planner -.-> worker_visualizer;
planner_visualizer --> planner;
tool --> worker_visualizer;
worker -.-> planner;
worker -.-> tool;
worker_visualizer --> worker;
output --> __end__;
classDef default fill:#f2f0ff,line-height:1.2
classDef first fill-opacity:0
classDef last fill:#bfb6fc
🔍 Workflow Breakdown:
- 🚀 START Node: Initial user input processing and system initialization
- 🧠 Planner-Visualizer Node: Strategic analysis with visual context capture
- 📋 Planner Node: High-level task decomposition and strategic planning
- 🔀 Conditional Routing: Intelligent decision branching based on planner actions
- 👁️ Worker-Visualizer Node: Context-aware screenshot capture for execution
- ⚙️ Worker Node: Precise action planning and tool orchestration
- 🎮 Tool Node: Direct game control execution (gamepad actions - mouse under development)
- 🔄 Conditional Worker Routing: Adaptive loop control with success/failure handling
- 📤 Output Node: Final result formatting and user response delivery
🛡️ Advanced Flow Control Features:
- ⏱️ Max Step Limits: Prevents infinite loops (5 planner steps, 3 worker steps)
- 💾 State Persistence: Maintains conversation history and game state across iterations
- 🛟 Error Handling: Graceful degradation and intelligent completion detection
- 🔧 Tool Integration: Exclusive MCP server integration for game control (no direct API usage)
This architecture enables complex decision-making while ensuring system reliability and user control. For the web interface that interacts with these agents, see Streamlit Interface
JokerNet features a modern, responsive Streamlit application that provides an intuitive and powerful interface for configuration, monitoring, and interaction with the AI agents.
| Feature | Description | Benefits |
|---|---|---|
| 📺 Live Game View | Embedded noVNC viewer for game monitoring | Visual feedback and control |
| ⚙️ Agent Configuration | Direct API calls for game and agent settings | Flexible automation strategies |
| 💬 Chat Interface | Natural language interaction with AI agents | Intuitive user experience |
| 🎯 Run Configuration | Deck selection, stake adjustment via API | Customized gameplay scenarios |
| 📊 Progress Monitoring | Live updates on agent actions and game state | Transparent automation process |
JokerNet includes a comprehensive Docker setup that containerizes the entire Balatro gaming environment, complete with custom mods, API servers, and remote access capabilities.
The Docker environment provides a fully isolated and optimized gaming ecosystem with all services running within a single container:
graph TB
subgraph "🐳 Docker Container<br/>Internal Architecture"
subgraph "API Services"
F[REST API<br/>Port 8000]
G[MCP Server<br/>Port 8001]
end
subgraph "Game Runtime"
B[noVNC<br/>Port 6080]
D[Balatro Game<br/>Love2D Engine]
E[Custom Mods<br/>BalatroLogger]
end
subgraph "Display System"
H[Xvfb<br/>Virtual Display]
I[x11vnc<br/>VNC Server]
end
B --> I
I --> H
H --> D
F --> D
G --> D
E --> D
end
style D fill:#e8f5e8
style F fill:#fff3e0
🔧 Internal Service Flow:
- 🌐 External Access: noVNC (Port 6080) provides web-based game interface, while VNC (Port 5900) offers native client access
- 🚀 API Layer: REST API (Port 8000) handles HTTP requests, MCP Server (Port 8001) manages AI agent integration
- 🎮 Game Control: Both API services communicate directly with the Balatro game engine for programmatic control
- 🖥️ Display Pipeline: Xvfb creates virtual display → x11vnc serves VNC → noVNC provides web proxy
- 🔧 Mod Integration: Custom mods enhance game functionality and enable automation features
🐳 Docker Container Benefits:
- 🎮 Complete Game Environment: Everything needed to run and interact with the game is encapsulated in a single container
- 🔧 Easy Deployment: One-command deployment with all dependencies and services pre-configured
- 🔄 Game Swapping: Simple replacement of game files and mods enables support for different games
- 📦 Portability: Consistent environment across different host systems
- 🛠️ Isolation: Clean separation between host system and game environment
All services are expertly managed by Supervisor for reliable, production-grade operation:
| Service | Port | Purpose | Technology |
|---|---|---|---|
| 🖥️ Xvfb | N/A | Virtual X server for headless display | X11 |
| 📡 x11vnc | 5900 | VNC server for remote desktop access | VNC Protocol |
| 🌐 noVNC | 6080 | Web-based VNC client (no installation needed) | WebRTC |
| 🚀 FastAPI | 8000 | REST API server for game control | Python/FastAPI |
| 🤖 MCP Server | 8001 | AI agent integration and tool orchestration | Python/MCP |
JokerNet includes several custom modifications to dramatically enhance automation capabilities:
- 💉 Lovely Injector: Core mod loading framework for Love2D games
- 🎮 Steamodded (SMODS): Comprehensive modding framework for Balatro
- 📊 BalatroLogger: Custom mod for automated game state logging and control
- ⭐ Saturn Mod: Enhanced UI and gameplay features
I've developed a custom mod (BalatroLogger) that significantly improves automation capabilities:
-- Key features from auto_start.lua
function love.update(dt)
if config and config.auto_start then
if G then
-- Automatic deck and stake configuration
if config.deck and G.P_CENTERS[config.deck] then
G.GAME.viewed_back = G.P_CENTERS[config.deck]
end
-- Programmatic game start
if G.FUNCS and G.FUNCS.start_run then
G.FUNCS.start_run(nil, {
stake = config.stake or 1,
seed = (config.seed and config.seed ~= "random") and config.seed or nil
})
end
end
end
end🎮 Mod Capabilities:
- 🚀 Auto-start functionality: Programmatic game initialization with custom parameters
- 💾 Configuration persistence: Settings saved across game sessions
- 📈 Enhanced logging: Detailed game state tracking for AI analysis
- 🔗 API integration: Direct communication with external automation systems
| Service | URL | Description | Primary Use |
|---|---|---|---|
| 🌐 noVNC Direct | http://localhost:6080 | Primary web-based game access | Main game interface |
| 📚 REST API | http://localhost:8000/docs | Interactive API documentation | Development & integration |
| 🤖 MCP Server | http://localhost:8001 | AI agent integration endpoint | Agent communication |
| 💻 VNC Native | localhost:5900 | Traditional VNC client access | Advanced users |
The REST API provides comprehensive control with intuitive endpoints:
# 🎮 Game control via gamepad
curl -X POST "http://localhost:8000/gamepad/buttons" \
-H "Content-Type: application/json" \
-d '{"sequence": "A RIGHT B", "duration": 0.1}'
# ⚙️ Auto-start configuration
curl -X POST "http://localhost:8000/auto_start" \
-H "Content-Type: application/json" \
-d '{"auto_start": true, "deck": "b_magic", "stake": 5}'
# 📸 Screenshot capture
curl "http://localhost:8000/screenshot" > game_state.pngThe Model Context Protocol server enables seamless AI agent integration with powerful tools:
# Available tools for AI agents
- press_buttons(sequence): Gamepad control
- mouse_click(x, y): Mouse interaction (under development)
- get_screen(): Screenshot capture
- locate_element(description): UI element detectionFor setup instructions, see Installation Guide
The system supports both gamepad and mouse control modes, but mouse functionality is not operational. The gamepad controller works reliably for automated gameplay, while mouse interaction requires additional development work.
I attempted to implement mouse control using AI vision models (HuggingFace ASKUI/PTA-1) for object detection, but encountered significant challenges:
- Card Detection: The AI model couldn't reliably identify Balatro's playing cards
- UI Recognition: Difficulty detecting buttons and interactive elements
- Visual Complexity: Balatro's artistic designs and animations made detection challenging
- Real-time Performance: Game state changes faster than AI processing
- Use Gamepad Control: Primary method that works reliably
- Manual Mouse: Direct interaction through noVNC interface when needed
- Future Development: Custom vision models and computer vision pipelines for mouse control
# 🖥️ Host system requirements
sudo modprobe uinput
sudo chmod 666 /dev/uinputgit clone https://github.com/Javier-Jimenez99/JokerNet.git
cd JokerNet
uv sync # Install dependencies
cp .env.example .env # Configure environment variablescd BalatroDocker
docker-compose up -dcd src
streamlit run app.py| Service | URL | Purpose |
|---|---|---|
| 🎮 Game | http://localhost:6080 | noVNC interface |
| 🌐 Interface | http://localhost:8501 | Streamlit web app |
| 📚 API Docs | http://localhost:8000/docs | REST API documentation |
Ejecuta ./startup.sh para iniciar en modo CPU. Usa ./startup.sh --gpu para activar GPU si está disponible.
JokerNet/
├── 🔧 .env # Environment configuration
├── 📦 pyproject.toml # Project dependencies
├── 🔒 uv.lock # Dependency lock file
├── 📁 src/
│ ├── 🌐 app.py # Streamlit application
│ ├── 🔗 api.py # API client
│ ├── 🤖 agents/
│ │ ├── 🧠 planner.py # Strategic planning agent
│ │ ├── ⚙️ worker.py # Execution agent
│ │ └── 📋 models/ # Pydantic models
│ └── 🎨 ui_components/ # Streamlit UI modules
├── 🐳 BalatroDocker/
│ ├── 📄 Dockerfile # Container definition
│ ├── 🔄 docker-compose.yml # Service orchestration
│ ├── 📜 scripts/ # Setup and startup scripts
│ ├── ⚙️ config/ # Configuration files
│ └── 🔧 src/ # API and MCP servers
├── 📓 notebooks/ # Development and testing
└── 🎭 assets/ # Static resources
Configure your .env from .env.example file:
# Azure OpenAI Configuration (required for AI agent)
AZURE_OPENAI_ENDPOINT=https://your-azure-endpoint.openai.azure.com/
AZURE_OPENAI_API_KEY=your_api_key_here
AZURE_OPENAI_MODEL=gpt-4.1
AZURE_OPENAI_API_VERSION=2025-01-01-preview
# LangSmith Configuration
LANGCHAIN_TRACING_V2=true
LANGCHAIN_PROJECT=BalatroAgent-Worker
LANGCHAIN_API_KEY=your_langchain_api_key_here
# Docker/Steam Configuration
STEAM_USER=your_steam_username
STEAM_PASS=your_steam_password
GAME_SPEED=16
Available Deck Options:
🔴 b_red- Red Deck🔵 b_blue- Blue Deck (recommended)🪄 b_magic- Magic Deck🌌 b_nebula- Nebula Deck👻 b_ghost- Ghost Deck⚡ b_plasma- Plasma Deck
Stake Levels: 1-8 (1=easiest, 8=hardest)
JokerNet's modular architecture enables easy extension to other games and advanced AI capabilities.
- Strategy Games: Civilization VI, Factorio
- Action/RPG: Slay the Spire, Hades
- Simulation: Cities: Skylines, The Sims 4
- Reinforcement Learning: PPO/DQN for complex decisions
- Multi-Modal AI: Visual + textual + audio processing
- Performance: Edge computing, model quantization
- Controls: Voice control, gesture recognition, mouse control
This project showcases professional-level software engineering with a focus on AI-driven automation, game development integration, and scalable system design.
