Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 76 additions & 70 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,84 +16,99 @@ docs/planning/ # Numbered phase directories (00-15)

## Build & Run

### Standard Workflow: Boot Stages Testing
## 🚨 CRITICAL: QEMU MUST RUN INSIDE DOCKER ONLY 🚨

For normal development, use the boot stages test to verify kernel health:
**NEVER run QEMU directly on the host.** This is an absolute, inviolable requirement.

```bash
# Run boot stages test - verifies kernel progresses through all checkpoints
cargo run -p xtask -- boot-stages

# Build only (no execution)
cargo build --release --features testing,external_test_bins --bin qemu-uefi
```
Running QEMU directly on macOS causes **system-wide instability**:
- Hypervisor.framework resource leaks when QEMU is killed
- GPU driver destabilization affecting ALL GPU-accelerated apps
- Crashes in terminals (Ghostty), browsers (Chrome/Brave), and Electron apps (Slack)
- Memory pressure cascades from orphaned QEMU processes

The boot stages test (`xtask boot-stages`) monitors serial output for expected markers at each boot phase. Add new stages to `xtask/src/main.rs` when adding new subsystems.
**The ONLY acceptable ways to run QEMU:**
1. `./docker/qemu/run-boot-parallel.sh N` - Run N parallel boot tests
2. `./docker/qemu/run-kthread-parallel.sh N` - Run N parallel kthread tests
3. `./docker/qemu/run-kthread-test.sh` - Run single kthread test
4. `./docker/qemu/run-dns-test.sh` - Run DNS resolution test
5. `./docker/qemu/run-keyboard-test.sh` - Run keyboard input test
6. `./docker/qemu/run-interactive.sh` - Interactive session with VNC display

### GDB Debugging (For Deep Technical Issues)

Use GDB when you need to understand **why** something is failing, not just **that** it failed. GDB is the right tool when:
- You need to examine register state or memory at a specific point
- A panic occurs and you need to inspect the call stack
- You're debugging timing-sensitive issues that log output can't capture
- You need to step through code instruction-by-instruction

**Do NOT use GDB** for routine testing or to avoid writing proper boot stage markers. If you find yourself adding debug log statements in a loop, that's a sign you should use GDB instead.
**For interactive use (framebuffer + keyboard):**
```bash
./docker/qemu/run-interactive.sh
# Then connect with TigerVNC:
open '/Applications/TigerVNC Viewer 1.15.0.app'
# Enter: localhost:5900
```

**PROHIBITED commands (will destabilize the host system):**
```bash
# Start interactive GDB session
# ❌ NEVER DO THIS:
cargo run -p xtask -- boot-stages
cargo run -p xtask -- dns-test
cargo run --release --bin qemu-uefi
./breenix-gdb-chat/scripts/gdb_session.sh start
./breenix-gdb-chat/scripts/gdb_session.sh cmd "break kernel::kernel_main"
./breenix-gdb-chat/scripts/gdb_session.sh cmd "continue"
./breenix-gdb-chat/scripts/gdb_session.sh cmd "info registers"
./breenix-gdb-chat/scripts/gdb_session.sh cmd "backtrace 10"
./breenix-gdb-chat/scripts/gdb_session.sh serial
./breenix-gdb-chat/scripts/gdb_session.sh stop
qemu-system-x86_64 ...
```

### Logs
All runs are logged to `logs/breenix_YYYYMMDD_HHMMSS.log`
### One-Time Docker Setup

Before running any tests, build the Docker image:
```bash
# View latest log
ls -t logs/*.log | head -1 | xargs less
cd docker/qemu
docker build -t breenix-qemu .
```

### Docker-Based Testing (Recommended for Parallel Execution)
### Standard Workflow: Docker-Based Testing

For isolated, parallel test execution, use Docker containers. Each container runs its own QEMU instance with no host interference.
For normal development, use Docker-based boot tests:

**One-time setup - build the Docker image:**
```bash
cd docker/qemu
docker build -t breenix-qemu .
# Build the kernel first
cargo build --release --features testing,external_test_bins --bin qemu-uefi

# Run boot tests in Docker (isolated, safe)
./docker/qemu/run-boot-parallel.sh 1

# Run multiple parallel tests for stress testing
./docker/qemu/run-boot-parallel.sh 5
```

**Parallel kthread-only tests (fast - focused testing):**
For kthread-focused testing:
```bash
# Build the kthread_test_only kernel (runs only kthread tests, then exits)
# Build kthread-only kernel
cargo build --release --features kthread_test_only --bin qemu-uefi

# Run 10 parallel tests
./docker/qemu/run-kthread-parallel.sh 10
# Run kthread tests in Docker
./docker/qemu/run-kthread-parallel.sh 1
```

**Parallel full boot tests:**
**Why Docker is mandatory:**
- Complete isolation from host Hypervisor.framework
- No GPU driver interaction (uses software rendering)
- Clean container lifecycle - no orphaned processes
- No risk of destabilizing user's system
- Containers auto-cleanup with `--rm`

### Logs
Docker test output goes to `/tmp/breenix_boot_N/` or `/tmp/breenix_kthread_N/`:
```bash
# Build full test kernel
cargo build --release --features testing,external_test_bins --bin qemu-uefi
# View kernel log from test 1
cat /tmp/breenix_boot_1/serial_kernel.txt

# Run 5 parallel full boot tests
./docker/qemu/run-boot-parallel.sh 5
# View user output from test 1
cat /tmp/breenix_boot_1/serial_user.txt
```

**Why Docker?**
- Each container is fully isolated - no lock contention on disk images
- Can run N tests in parallel without QEMU process conflicts
- Containers clean up automatically with `--rm`
- Uses TCG (software emulation) - slower than native but reliable
### GDB Debugging - DISABLED

GDB debugging requires native QEMU which is prohibited. For debugging:
1. Add strategic `log::info!()` statements
2. Run in Docker and examine serial output
3. Use QEMU's `-d` flags for CPU/interrupt tracing (inside Docker)

**Note:** Docker tests use software CPU emulation (TCG) rather than hardware virtualization, so they run slower than native QEMU with Hypervisor.framework. Use Docker for parallel stress testing; use native QEMU for interactive debugging.
If you absolutely must use GDB for a critical issue, **ask the user first** and explain why Docker-based debugging is insufficient.

## Development Workflow

Expand Down Expand Up @@ -602,36 +617,27 @@ log::debug!("clock_gettime called"); # This changes timing!
./breenix-gdb-chat/scripts/gdb_session.sh stop
```

## QEMU Process Cleanup - MANDATORY
## Docker Container Cleanup

**Agents MUST clean up stray QEMU processes.** This is non-negotiable.
**Docker containers auto-cleanup with `--rm`, but if needed:**

QEMU processes frequently get orphaned during testing, debugging, or when agents are interrupted. These orphaned processes:
- Hold locks on disk images, preventing new QEMU instances from starting
- Consume system resources
- Cause confusing errors like "Failed to get write lock"
```bash
# Kill any running breenix-qemu containers
docker kill $(docker ps -q --filter ancestor=breenix-qemu) 2>/dev/null || true

### Cleanup Requirements
# Verify no containers running
docker ps --filter ancestor=breenix-qemu
```

1. **Before handing control back to the user**: Always run QEMU cleanup
2. **Before running any QEMU command**: Kill any existing QEMU processes first
3. **When debugging fails or times out**: Clean up QEMU before reporting results
### If You See Orphaned Host QEMU Processes

### Cleanup Command
If you see `qemu-system-x86_64` processes running directly on the host (not in Docker), this means someone violated the Docker-only rule. Clean them up:

```bash
pkill -9 qemu-system-x86_64 2>/dev/null; pgrep -l qemu || echo "All QEMU processes killed"
```

**CRITICAL: QEMU will ALWAYS hang beyond the bounds of a run.** You MUST kill the previous QEMU before running any test, or it will fail due to disk image lock contention.

### When to Clean Up

- **BEFORE every test run** - This is mandatory, not optional
- After any `xtask boot-stages` or `xtask interactive` run
- After GDB debugging sessions
- When the user reports "cannot acquire lock" errors
- When handing results back to the user after kernel work
**Then investigate how they got there** - all QEMU execution must go through Docker.

This is the agent's responsibility - do not wait for the user to ask.

Expand Down
2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ testing = ["kernel/testing"]
kthread_test_only = ["kernel/kthread_test_only"] # Run only kthread tests and exit
kthread_stress_test = ["kernel/kthread_stress_test"] # Run kthread stress test (100+ kthreads) and exit
workqueue_test_only = ["kernel/workqueue_test_only"] # Run only workqueue tests and exit
dns_test_only = ["kernel/dns_test_only"] # Run only DNS test and exit (fast network debugging)
blocking_recv_test = ["kernel/blocking_recv_test"] # Run only blocking recvfrom test and exit
test_divide_by_zero = ["kernel/test_divide_by_zero"]
test_invalid_opcode = ["kernel/test_invalid_opcode"]
test_page_fault = ["kernel/test_page_fault"]
Expand Down
8 changes: 7 additions & 1 deletion docker/qemu/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,13 +1,19 @@
# Dockerfile for running Breenix tests in isolated QEMU
# Includes tools for keyboard input testing via QEMU monitor
# Supports X11 and VNC display modes for interactive use

FROM ubuntu:24.04

# Install QEMU and OVMF
# Install QEMU with display support and OVMF
RUN apt-get update && apt-get install -y \
qemu-system-x86 \
ovmf \
expect \
# X11 and SDL support for graphical display
libsdl2-2.0-0 \
libgtk-3-0 \
libvte-2.91-0 \
x11-apps \
&& rm -rf /var/lib/apt/lists/*

# Create working directory
Expand Down
129 changes: 129 additions & 0 deletions docker/qemu/run-blocking-recv-test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
#!/bin/bash
# Run blocking recvfrom test in isolated Docker container
# Usage: ./run-blocking-recv-test.sh
#
# This script runs the blocking_recv_test kernel build in Docker,
# then sends a UDP packet to wake the blocking recvfrom().

set -e

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BREENIX_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"

# Build Docker image if needed
IMAGE_NAME="breenix-qemu"
if ! docker image inspect "$IMAGE_NAME" &>/dev/null; then
echo "Building Docker image..."
docker build -t "$IMAGE_NAME" "$SCRIPT_DIR"
fi

echo "Building blocking_recv_test kernel..."
cargo build --release --features blocking_recv_test --bin qemu-uefi

# Check for blocking_recv_test kernel build
UEFI_IMG=$(ls -t "$BREENIX_ROOT/target/release/build/breenix-"*/out/breenix-uefi.img 2>/dev/null | head -1)
if [ -z "$UEFI_IMG" ]; then
echo "Error: UEFI image not found."
exit 1
fi

if [ ! -f "$BREENIX_ROOT/target/test_binaries.img" ]; then
echo "Error: test_binaries.img not found. Build the test disk first."
exit 1
fi

# Create output directory
OUTPUT_DIR=$(mktemp -d)
trap "rm -rf $OUTPUT_DIR" EXIT

# Create empty output files
touch "$OUTPUT_DIR/serial_kernel.txt"
touch "$OUTPUT_DIR/serial_user.txt"

echo "Running blocking recvfrom test in Docker container..."
echo " UEFI image: $UEFI_IMG"
echo " Output dir: $OUTPUT_DIR"
echo ""

# Copy OVMF files to writable location
cp "$BREENIX_ROOT/target/ovmf/x64/code.fd" "$OUTPUT_DIR/OVMF_CODE.fd"
cp "$BREENIX_ROOT/target/ovmf/x64/vars.fd" "$OUTPUT_DIR/OVMF_VARS.fd"

# Run QEMU inside Docker
# - Uses SLIRP networking (user mode) with UDP forward from host port 55556
timeout 120 docker run --rm \
-p 55556:55556/udp \
-v "$UEFI_IMG:/breenix/breenix-uefi.img:ro" \
-v "$BREENIX_ROOT/target/test_binaries.img:/breenix/test_binaries.img:ro" \
-v "$BREENIX_ROOT/target/ext2.img:/breenix/ext2.img:ro" \
-v "$OUTPUT_DIR:/output" \
"$IMAGE_NAME" \
qemu-system-x86_64 \
-pflash /output/OVMF_CODE.fd \
-pflash /output/OVMF_VARS.fd \
-drive if=none,id=hd,format=raw,media=disk,readonly=on,file=/breenix/breenix-uefi.img \
-device virtio-blk-pci,drive=hd,bootindex=0,disable-modern=on,disable-legacy=off \
-drive if=none,id=testdisk,format=raw,readonly=on,file=/breenix/test_binaries.img \
-device virtio-blk-pci,drive=testdisk,disable-modern=on,disable-legacy=off \
-drive if=none,id=ext2disk,format=raw,readonly=on,file=/breenix/ext2.img \
-device virtio-blk-pci,drive=ext2disk,disable-modern=on,disable-legacy=off \
-machine pc,accel=tcg \
-cpu qemu64 \
-smp 1 \
-m 512 \
-display none \
-boot strict=on \
-no-reboot \
-no-shutdown \
-monitor none \
-device isa-debug-exit,iobase=0xf4,iosize=0x04 \
-netdev user,id=net0,hostfwd=udp::55556-:55556 \
-device e1000,netdev=net0,mac=52:54:00:12:34:56 \
-serial file:/output/serial_user.txt \
-serial file:/output/serial_kernel.txt \
&

QEMU_PID=$!

echo "Waiting for blocking recvfrom test..."
TIMEOUT=90
ELAPSED=0
SENT_PACKET=0

while [ $ELAPSED -lt $TIMEOUT ]; do
sleep 1
ELAPSED=$((ELAPSED + 1))

if [ -f "$OUTPUT_DIR/serial_user.txt" ]; then
USER_OUTPUT=$(cat "$OUTPUT_DIR/serial_user.txt" 2>/dev/null)

if [ $SENT_PACKET -eq 0 ] && echo "$USER_OUTPUT" | grep -q "BLOCKING_RECV_TEST: waiting for packet..."; then
SENT_PACKET=1
echo "Blocking recvfrom waiting; sending UDP packet..."
echo "wakeup" | nc -u -w1 localhost 55556 || true
fi

if echo "$USER_OUTPUT" | grep -q "BLOCKING_RECV_TEST: PASS"; then
echo ""
echo "========================================="
echo "BLOCKING RECV TEST: PASS"
echo "========================================="
docker kill $(docker ps -q --filter ancestor="$IMAGE_NAME") 2>/dev/null || true
exit 0
fi
fi
done

echo ""
echo "========================================="
echo "BLOCKING RECV TEST: TIMEOUT"
echo "========================================="
echo ""
echo "User output (COM1):"
cat "$OUTPUT_DIR/serial_user.txt" 2>/dev/null | grep -E "BLOCKING_RECV_TEST" || echo "(no blocking recv output)"
echo ""
echo "Last 20 lines of kernel output (COM2):"
tail -20 "$OUTPUT_DIR/serial_kernel.txt" 2>/dev/null || echo "(no output)"

docker kill $(docker ps -q --filter ancestor="$IMAGE_NAME") 2>/dev/null || true
exit 1
Loading
Loading