API improvements based on real-world agent usage data (1,270 commands analyzed)

[Niaobu]

API Improvements Based on Real-World Agent Usage Data

Context

Analysis of 1,270 real-world commands across 30 Claude Code sessions over one month, using three worktrees of the same Unity project.

Current reliability: 89% overall (1,134 OK / 1,270 total)

Failures break down into three categories:

• C# compilation errors — 44 (33% of failures)
• Timeouts — 40 (30%)
• Wrong command/flag names — 21 (16%)
• Other — 29 (21%)

This issue tracks the quick-win improvements (items 1–4, 8) that address 50+ failures with low effort, plus longer-term proposals (items 5–10).

---

1. Command Aliases

Problem: Agents tried non-existent commands 21 times. These are natural guesses from an LLM that knows Unity terminology but not the exact CLI surface.

Tried	Should be	Occurrences
editor launch	editor run	2
compile	asset refresh	1
exec <expr>	script eval <expr>	1
assets refresh	asset refresh	1
script run	script execute	2
play start	play enter	1
play stop	play exit	0 (lucky)
log	logs	1
start	editor run	1
screenshot game	screenshot capture	1

Proposal: Add hidden aliases using System.CommandLine's .AddAlias() so these resolve silently without polluting --help:

// editor
runCommand.AddAlias("launch");          // editor launch → editor run
runCommand.AddAlias("start");           // editor start → editor run

// play
enterCommand.AddAlias("start");         // play start → play enter
exitCommand.AddAlias("stop");           // play stop → play exit

// script
executeCommand.AddAlias("run");         // script run → script execute

For top-level aliases (compile, exec, log), implement as thin wrapper commands marked IsHidden = true since System.CommandLine doesn't natively support aliasing across command groups.

Impact: Eliminates ~21 wasted tool calls.

---

2. Universal --timeout / -t Flag

Problem: Agents naturally try -t on commands that don't support it:

• scene load -t 120 → "Unrecognized command or argument '120'"
• asset refresh -t 120 → "Unrecognized command or argument '-t'"

Currently only script eval and script execute accept -t. The bridge already has per-command timeout defaults and supports caller-specified overrides — the CLI just doesn't expose this.

Proposal: Add --timeout / -t as a global option, passed through to the bridge's RpcRequest.Timeout:

var timeoutOption = new Option<int?>(
    ["-t", "--timeout"],
    "Timeout in seconds (overrides default for this command)");
rootCommand.AddGlobalOption(timeoutOption);

Bridge defaults are already sensible (30s for most, 60s for asset refresh, 300s for tests, 600s for recording). The -t flag lets the caller override when needed.

Impact: Eliminates timeout failures on scene load and asset refresh, and makes the API consistent.

---

3. logs Flag Aliases

Problem: Agents tried --count N, --last N, and -f (follow) — none of which work as expected. Actual flag is -n N:

Tried	Actual	Occurrences
logs --count 20	logs -n 20	4
logs --last 20	logs -n 20	1
logs -f	(not working)	1

Proposal:

• Add --count and --last as aliases for -n
• Verify -f / --follow implementation works (it's documented in SKILL.md but doesn't work as expected in practice)

Impact: Eliminates 5 failures.

---

4. screenshot capture Flag Alias

Problem: Agent used --output which doesn't exist. The path argument is positional:

# Agent tried:
unityctl screenshot capture --output /path/to/file.png
# Actual syntax:
unityctl screenshot capture /path/to/file.png

Proposal: Add --output / -o as an option alias for the positional path argument.

---

5. scene load During Play Mode

Problem: Two failures from scene load while in play mode:

Error: This cannot be used during play mode, please use SceneManager.LoadScene() instead.

The error tells the user which Unity API to use, but the agent has to translate that into a script eval call.

Proposal: Either auto-switch to runtime scene loading when in play mode, or improve the error message to include the exact unityctl command to use instead.

---

6. Better C# Compilation Error Feedback

Problem: C# compilation errors are the #1 failure cause (44/134). For script eval, the error references a temp file path and line numbers are offset by the wrapper code.

Proposal A — Better eval error output: Show the generated wrapper code alongside the error so the agent can see what was actually compiled, with a caret pointing to the error location.

Proposal B — script discover command: Add a command to introspect available types and members at runtime via reflection:

unityctl script discover --type "SomeType"
unityctl script discover --members "UnityEditor.EditorPrefs"
unityctl script discover --search "BuildPlayer"

This would let agents verify APIs exist before writing code.

---

7. Timeout UX Improvements

Problem: The 30s bridge timeout is the second-biggest failure category (40 failures). Patterns observed:

• Long builds (7): Agent starts a build, gets timeout, then has to wait + retry
• Cascading timeouts (8): After a timeout, Unity may still be blocked. Agent retries immediately, gets another timeout, cascading 3–5 times
• Unity stuck (5): After certain operations, Unity becomes unresponsive for minutes

Proposal A — Better timeout error message: Include actionable guidance (wait and retry, use longer timeout, check status).

Proposal B — wait --retry: Atomic "wait for Unity, then run command":

unityctl wait --retry "script eval '1+1'"

Proposal C — Background execution mode: Fire-and-forget with polling for long operations like builds:

unityctl script eval --background 'BuildPipeline.BuildPlayer(...)'
unityctl job status <id>
unityctl job wait <id> --timeout 600

---

8. SKILL.md Improvements

The SKILL.md is what agents read before using the CLI. Based on the failure data, add:

• Common pitfalls section — mapping wrong commands to correct ones
• Timeout guidance — default timeouts, when to use -t, recovery after timeout
• Shell quoting examples — for script eval expressions

---

9. asset refresh Improvements

Problem: 13 non-timeout failures, mostly compilation errors. Agent sometimes doesn't realize it needs to fix code and retry.

Proposal: After compilation failure, add a hint suggesting the fix-compile-retry workflow.

---

10. Exit Code Granularity

Problem: Every failure returns exit code 1. Agents can't distinguish between fixable, retryable, and setup errors.

Proposal:

Exit Code	Meaning
0	Success
1	Command error (bad args, not found)
2	Compilation error (C# failed to compile)
3	Runtime error (C# threw exception)
4	Timeout
5	Not connected (bridge down or Unity not connected)

---

Priority

#	Proposal	Impact	Effort
1	Command aliases	Eliminates 21 failures	Low
2	Universal -t flag	Enables timeout control everywhere	Low
3	logs flag aliases	Eliminates 5 failures	Trivial
8	SKILL.md improvements	Prevents future wrong-name errors	Trivial
4	screenshot --output	Eliminates 1 failure	Trivial
10	Exit code granularity	Better automation	Low
7A	Better timeout error msg	Reduces cascading retries	Low
6A	Better eval error output	Faster fix iteration	Medium
5	Scene load in play mode	Eliminates 2 failures	Medium
9	Asset refresh hints	Better error UX	Low
6B	script discover	Major reduction in C# errors	Medium-High
7B	wait --retry	Eliminates cascading timeouts	Medium
7C	Background job execution	Solves build timeouts	High

Quick wins (all low effort, address 50+ failures): #1, #2, #3, #4, #8