Device Runtime Eval: Execute expressions on Unity player builds via ADB

[Niaobu]

Device Runtime Eval: Execute expressions on Unity player builds via ADB

Context

unityctl's script.execute / script.eval commands are among the most powerful tools for AI agents controlling Unity — they enable arbitrary C# execution inside the Editor via Roslyn. However, this capability stops at the Editor boundary. Once an app is built and running on an Android device, agents have no way to inspect or manipulate the running player.

This proposal adds a reflection-based expression evaluator that runs inside Unity player builds, exposed over HTTP via ADB port forwarding. It reuses the same CLI → Bridge → Device communication pattern and gives agents a runtime inspection and remote control capability for deployed builds.

Design

Architecture

CLI: unityctl device eval "<expr>"
  │
  ▼
Bridge ─── adb forward tcp:N tcp:N ───► Android Device
                                            │
                                            ▼
                                     Unity Player (IL2CPP)
                                     ┌─────────────────────────────┐
                                     │ DeviceServer                │
                                     │   (HTTP, background thread) │
                                     │         │                   │
                                     │         ▼                   │
                                     │ ExpressionEvaluator         │
                                     │   ├── Tokenizer             │
                                     │   ├── Parser → AST          │
                                     │   └── Evaluator (reflection)│
                                     │         ├── TypeResolver    │
                                     │         ├── MemberResolver  │
                                     │         └── ValueConverter  │
                                     │         │                   │
                                     │         ▼                   │
                                     │   Main thread dispatch      │
                                     │   (for Unity API calls)     │
                                     └─────────────────────────────┘

Components

1. DeviceServer — Lightweight HTTP listener embedded in the player build. Runs on a background thread, dispatches work to Unity's main thread for API calls. Included in builds via a runtime MonoBehaviour that auto-starts (similar to how the Editor plugin bootstraps via [InitializeOnLoad]).

2. ExpressionEvaluator — A small recursive-descent parser + reflection-based evaluator. Accepts a C#-like expression language and translates it to reflection calls. Runs entirely on compiled types already present in the IL2CPP binary — no JIT, no Emit, no Roslyn.

3. ADB integration — The Bridge (or CLI directly) manages adb forward to map the device's HTTP port to localhost, making the device server accessible as if it were local.

Expression Language

A deliberately constrained subset of C# expressions. The mental model: anything you could type into a debugger watch window.

Grammar

ExpressionList = Expression (';' Expression)*     // last value is returned
Expression     = Assignment | BinaryExpr
Assignment     = '$' Identifier '=' Expression     // evaluator variable
               | Chain '=' Expression              // property/field set
BinaryExpr     = Chain (BinOp Chain)*
Chain          = Atom ('.' Member)*
Member         = Identifier                        // field/property access
               | Identifier '(' ArgList ')'        // method call
               | Identifier '<' TypeList '>' '(' ArgList ')'  // generic method
Atom           = 'new' TypeName '(' ArgList ')'    // constructor
               | Literal                           // int, float, string, bool, null
               | '$' Identifier                    // evaluator variable
               | TypeName                          // static type access
               | '(' Expression ')'                // grouping
Literal        = Number | QuotedString | 'true' | 'false' | 'null'
BinOp          = '+' | '-' | '*' | '/' | '==' | '!=' | '>' | '<' | '>=' | '<='

What it supports

// Property/field access
Camera.main.transform.position.x
Application.targetFrameRate
Screen.width

// Method calls
GameObject.Find("Player").GetComponent<Health>().currentHp
Object.FindObjectsOfType<Camera>().Length
PlayerPrefs.GetFloat("volume")

// Generic methods
obj.GetComponent<Rigidbody>()
Resources.Load<Material>("MyMaterial")

// Assignment — property/field set
Application.targetFrameRate = 30
Camera.main.fieldOfView = 90

// Constructors
new Vector3(1, 2, 3)
new Color(1, 0, 0, 1)

// Arithmetic and comparison on results
player.transform.position.x + 5.0f
Screen.width / 2
Time.time > 10

// Evaluator variables ($ prefix, not C# variables)
$player = GameObject.Find("Player"); $player.transform.position

// Array/collection indexing via reflection
transform.GetChild(0).name

What it does NOT support

Feature	Reason
Lambdas / closures	Requires code generation — impossible on IL2CPP
Variable declarations (var x = ...)	Use $x = ... instead
Control flow (if/for/while)	Not an expression — out of scope
LINQ (.Where(), .Select())	Requires delegate creation
Anonymous types / dynamic	Requires Emit
String interpolation ($"...")	Use string concat with +
typeof() operator	Use Type.GetType("Name") instead
Multi-line method bodies	Not a REPL — single expression chains only

Arithmetic Without Emit

Binary operators are implemented as a type-dispatch switch — no expression trees or JIT needed:

static object EvalBinaryOp(object left, string op, object right)
{
    return (left, op, right) switch
    {
        (int a, "+", int b) => a + b,
        (float a, "+", float b) => a + b,
        (float a, "+", int b) => a + (float)b,
        (int a, "+", float b) => (float)a + b,
        (string a, "+", _) => a + right?.ToString(),
        (Vector3 a, "+", Vector3 b) => a + b,
        (Vector3 a, "*", float b) => a * b,
        // ... other combinations
        _ => throw new EvalException($"Operator '{op}' not supported for {left?.GetType().Name} and {right?.GetType().Name}")
    };
}

Type Resolution

The evaluator needs to resolve short type names to full System.Type references. Strategy:

1. Prebuilt lookup table — On startup, scan all loaded assemblies and build a Dictionary<string, Type> for common Unity types (GameObject, Transform, Vector3, Camera, Application, Time, etc.)
2. Namespace search order — For unqualified names, search: UnityEngine → UnityEngine.UI → System → all loaded assemblies
3. Fully qualified fallback — UnityEngine.Rendering.Volume works if the short name is ambiguous
4. Caching — All resolved types, methods, and properties are cached after first lookup

Main Thread Dispatch

Most Unity APIs must run on the main thread. The DeviceServer receives HTTP requests on a background thread and must marshal execution:

HTTP thread                          Main thread
    │                                    │
    ├── Parse expression                 │
    ├── Queue evaluation ───────────────►├── Evaluate via reflection
    │                                    ├── Serialize result
    ├── ◄─────────────── Return result ──┤
    ├── Send HTTP response               │

Use a TaskCompletionSource + main-thread update loop (same pattern as the Editor plugin's UnityCtlClient).

Error Messages

Errors should guide the user toward what works, not just say what failed:

> unityctl device eval "objects.Where(o => o.name == \"foo\")"
Error: Lambda expressions are not supported on IL2CPP.
  Hint: Use a built-in query instead:
    GameObject.Find("foo")
    Object.FindObjectsOfType<MyType>()

> unityctl device eval "var x = 5"
Error: Variable declarations use $ prefix:
    $x = 5

> unityctl device eval "if (health < 0) { Die(); }"
Error: Control flow (if/for/while) is not supported.
  Only expressions and assignments are allowed.

> unityctl device eval "SomeStrippedType.DoThing()"
Error: Type 'SomeStrippedType' not found in loaded assemblies.
  It may have been stripped by IL2CPP managed code stripping.
  Preserve it by adding to your project's link.xml.

CLI Interface

# Evaluate an expression
unityctl device eval "Camera.main.transform.position"

# Set a value
unityctl device eval "Application.targetFrameRate = 30"

# Chain with variables
unityctl device eval '$cam = Camera.main; $cam.fieldOfView = 90; $cam.fieldOfView'

# Check device connection
unityctl device status

# List connected devices (via adb devices)
unityctl device list

Response Format

Consistent with existing script.eval responses:

{
  "success": true,
  "result": "(1.0, 2.5, -3.0)",
  "resultType": "UnityEngine.Vector3"
}

{
  "success": false,
  "error": "Type 'Foo' not found in loaded assemblies.",
  "hint": "It may have been stripped by IL2CPP. Add to link.xml to preserve it."
}

IL2CPP Stripping Considerations

IL2CPP's managed code stripping removes types and members not referenced in compiled code. This directly affects what the evaluator can access at runtime.

Mitigation strategies:

• Document which stripping level is required (recommend Minimal for best compatibility)
• Provide a link.xml template that preserves common Unity types used in eval
• Surface clear errors when a type/member is not found, with actionable fix (add to link.xml)
• The DeviceServer component itself, by existing in the build, preserves the types it references

Build Integration

The device runtime components should be:

• Conditionally compiled — Only included in Development Builds, or behind a scripting define (UNITYCTL_DEVICE)
• Strippable — When the define is absent or it's a release build, zero overhead
• Auto-starting — A RuntimeInitializeOnLoadMethod creates the server GameObject on player start
• Part of the existing UPM package — Lives alongside the Editor plugin in com.dirtybit.unityctl, but under a Runtime/ assembly

Scope

Phase 1: Foundation

• Expression tokenizer + parser
• Reflection-based evaluator (property/field access, method calls, constructors)
• Type resolver with namespace search
• HTTP server for player builds (background thread + main thread dispatch)
• ADB port forwarding integration in Bridge/CLI
• device eval, device status, device list CLI commands
• Error messages with hints

Phase 2: Ergonomics

• Evaluator variables ($var)
• Arithmetic and comparison operators
• Generic method support (GetComponent<T>())
• Member resolution caching
• Result serialization for complex Unity types (Vector3, Color, etc.)
• link.xml template for common preserved types

Phase 3: Extended Commands

• device screenshot — capture screenshot from device
• device log — stream device logs (logcat filtered)
• device scene — scene queries (list, active scene)
• device find — GameObject queries with filters
• Parity with a subset of Editor-side commands where applicable

Constraints

• IL2CPP only — Must work without Mono runtime or JIT compilation
• No external dependencies — No MoonSharp, no Roslyn, no third-party scripting engines
• Minimal binary size impact — The evaluator + HTTP server should be small
• Development builds only — Default to stripping from release builds
• Android first — ADB is the initial transport; iOS/desktop can follow later