AN EMBODIED, GAME-INDEPENDENT AI TEAMMATE
Player 2 has entered the game.
Eyes, hands, ears, and memory — wired to ordinary player I/O. It watches the screen and holds a controller, the same way you do. No game APIs. No memory reading. No injection. If it can't see it or press it, it doesn't know it.
Two reasons to plug in
Play together, or measure against it
An AI teammate, not a walkthrough
Local co-op, campaigns, mod-friendly worlds. It accumulates a shared history with you across a session, argues about strategy, develops favorite tactics, and occasionally saves you — and occasionally does not. The framing is closer to "I gave an AI a controller and taught it how to play" than to a bot that beats a game for you.
Not built for ranked competitive multiplayer, and not intended for it. This is for private matches, co-op, campaigns, local play — the parts of gaming that are about the people (or the teammate) in the room with you.
A motor cortex for evaluating agents
Screen capture, a virtual Xbox pad, one shared clock, a deadman safety boundary, and
a recorder that logs what the controller actually did — not what a model
proposed. Point any vision-capable model at it through IFastPolicy and
you get a real perception-to-action harness with no game-specific glue to write.
Transport is the real cost. One decision through a CLI-shelled agent runs ~12.5k tokens before a single pixel is added — a downscaled JPEG frame is nearly free by comparison. Full numbers in docs/STORAGE.md.
The body, not the brain
Eyes. Hands. Ears. Voice. Memory.
Every capability maps to a channel a human player already has. None of it knows what game it's in — game understanding lives entirely in the model sitting behind these channels.
Windows Graphics Capture into a ring buffer, downscaled and perceptually deduplicated before anything reads it.
doneA virtual Xbox pad over ViGEm. One thread is the sole writer, always — nothing else touches the device.
doneGame audio and microphone input, timestamped on the same clock as everything else.
plannedLocal faster-whisper and Piper for the loop that must never stall; a cloud model for the words themselves. Opt-in — nothing in core imports it.
plannedSQLite plus model-curated rules — what it learned about a game and about playing with you, kept across sessions.
plannedNo game-specific knowledge in the runtime
The hard constraint
There is no heal_party(). No seek_cover(). No
aim_at_enemy(). The lowest abstraction the runtime knows is the human input
device: make the pad have these states at these times.
Facts about a specific game — which window to
capture, that controller-x mines in Factorio, that ^/ is a
forbidden chat prefix — live in a profile TOML file, never in code. A test greps the
entire runtime for game names on every change. It has already caught three real leaks.
The question for every new piece of code is the same one: could this run unmodified in
Factorio, Baldur's Gate 3, and a game that doesn't exist yet?
Five threads, one queue, one bus
Nothing cognitive can stall the hands
Cognition only ever submits. A dedicated scheduler thread is the only thing that ever writes to the controller, which is what makes "it keeps moving while it talks" true by construction — not by hope.
- The scheduler is the sole writer to the controller. Everything else submits through its queue.
- A new chunk resolves against neutral, not live state. Anything it doesn't mention gets released — nothing inherits a held stick by accident.
- Absent is not empty.
buttons=Noneinherits;buttons=frozenset()releases everything. - One clock. Everything timestamps from
player2.clock; audio drifts against measured sample count, never arrival time. - The recorder captures what happened, not what was proposed. Proposals get preempted, rejected, and overridden constantly.
- MCP is the control plane, never the servo bus. Frames and action chunks never travel over it.
- Voice is opt-in. Nothing in core imports
player2.audio. - Silence means stop. No fresh chunk, no held input — the deadman always wins.
Milestone 1 — reached 2026-08-09
A real game, a real controller, on the record
An agent process manipulated a live copy of Factorio 2.0.7 through the same virtual-pad path a human uses — no API, no memory reading, no injection. In the recorded run: 6 chunks accepted, 5 preemptions as each new intent replaced the last, and the deadman releasing the pad at 6706ms the moment the agent went quiet. The property that matters most held in a real game, not just in tests.
| Subsystem | State |
|---|---|
| Clock & action-chunk contracts | Done, hardened |
| Motor system — scheduler, virtual pad | Done, hardened, verified on hardware |
| Profiles — game facts as config | Done |
| Window find / focus | Done |
| Eyes — capture, ring buffer, fingerprint | Done, hardened |
| Recorder — synchronized demonstrations | Done, hardened |
| Agent seam — Observation, policy, loop | Done, hardened |
| Direct SDK transport — Anthropic, OpenAI, CLI, Ollama | Done, not yet live-tested |
| Budget governor — token-rate degrade ladder | Done, not yet live-tested |
| Keyboard / mouse + text entry | Planned |
| Ears, voice, memory, MCP control plane | Planned |
Built by adversarial review, not vibes
Every line has an author and a reviewer
Owns architecture, the tests, the final diff, and validation. Writes the test file first, always — it becomes the spec the implementer works against, not documentation written after the fact.
Three tiers routed by stakes: cheap and tightly-specified work, a default implementer, and an escalation for concurrency, lifetimes, and anything that must be released correctly. Implements against Claude's tests, or reviews a diff — never told which model wrote it.
Every finding gets adjudicated — reproduced, or rejected with a one-line reason — and every round is logged with its token cost in docs/DEV-PROCESS.md. That log is treated as a deliverable, not overhead: the goal is a routing policy backed by real numbers instead of vendor benchmarks.