Contents
Chi (χ, pronounced “kai”) is an open-source autoresearch harness for LLM coding agents, and the pitch fits in one sentence: point agents — any vendor — at a problem with a programmatic evaluator (a build, a correctness check, a score) and let them iterate unattended until the champion stops improving. The code is at github.com/kdpisda/chi under Apache-2.0, the package is getchi on PyPI, and there’s a short overview on my project page.
For a week in July I ran six coding agents from five different vendors — claude, codex, pi, deepseek, glm, nemotron — against one problem: a batched Cholesky kernel on a B200 GPU, chasing a GPU MODE leaderboard. There was no framework. Coordination was a hand-rolled blackboard of markdown files on a shared sandbox, steering was me editing STEERING.md at odd hours, and monitoring was me, reading transcripts. The fleet found real wins. It also produced the most instructive failure I’ve seen in an agent system: one agent cycled through three near-identical “optimizations” for roughly thirty hours, posted seven near-identical “final submission” messages to the blackboard, and never produced a single GPU benchmark datapoint. Nothing noticed, because nothing was watching.
Chi is the harness I wished I’d had that week, rebuilt from the postmortem.
The gap: chat loops don’t research
There’s a category missing between “prompt a model once and hope” and “a human babysitting a coding agent all afternoon.” Plenty of tools give you an agent that edits code. Almost none give you unattended iteration: a loop that proposes a candidate, measures it, records the result durably, keeps the best one, and keeps going after you close the laptop — without drifting, looping, or silently burning your API budget.
The linchpin is the programmatic evaluator. If you can express “better” as a program — compile it, check correctness against held-out seeds, produce a number — then the LLM’s job changes character completely. It no longer has to be right on the first try; it has to propose candidates faster than the evaluator rejects them. That’s not code generation anymore, it’s search, and search is something you can harness: dedup identical candidates, gate on correctness, track a champion, detect stalls. Chi calls a problem exactly that — a directory with a problem.yaml naming entrypoints for correctness and benchmark, held-out seeds, and a score metric with a direction. Correctness is a hard gate: an incorrect candidate is recorded but can never become champion, and candidates never see the reference outputs they’re graded against.
And the harness is deliberately vendor-agnostic. The manual experiment ran five vendors side by side, and the heterogeneity was a feature, not an accident — different models stall in different places and unstick each other’s dead ends. A harness that only speaks one vendor’s protocol turns your research loop into a procurement decision. Chi treats the model as a replaceable part.
Try the whole machine with no API key
Before the architecture, the demo — because the demo is an architectural statement.
uv tool install getchi
chi run examples/offline.yaml # from a checkout of the repo
That runs with no provider, no key, and no network. A scripted adapter replays three canned candidates for the bundled optimize_function problem, and the harness really evaluates each one: the O(n²) prefix-sum baseline gets measured, then displaced by an O(n) itertools.accumulate rewrite:
"baseline_score": 25.18, # O(n²) prefix sums
"champion_score": 0.055, # O(n) itertools.accumulate
"status": "done"
The conventional move for an agent framework is to make the LLM load-bearing everywhere, which means your test suite is either mocked into meaninglessness or costs money and flakes with the provider. I made the model a pluggable adapter instead, so the scripted adapter that powers this demo is the same one that powers the test suite — evaluator, run store, champion selection, and watchdog all execute deterministically in CI with zero network. If a framework can only be exercised against a live LLM, it isn’t tested; it’s rehearsed.
What a week of manual fleet-running actually teaches you
Chi’s v1 design doc opens with a forensic review of the July experiment, and every structural decision in the harness answers a specific observed failure. The short list:
- The thirty-hour loop. An agent that looks busy but produces no evaluated candidates is looping by definition — and no one had encoded that definition. → Chi ships a deterministic watchdog from day one.
- Voluntary protocol compliance. Strong models followed the blackboard conventions; weak ones didn’t. One agent recorded 1 claim where 560+ had existed. → The run store is enforced, not conventional.
- A non-durable bus. An epoch reset on the shared directory destroyed 1,100+ messages of coordination history. → Append-only storage, mirrored to plain files.
- Confounded rule-outs. Two “this approach is dead” verdicts were later reversed — they’d been tested in the wrong regime, and the unscoped negatives nearly killed the winning direction. → Negative results carry scope, evidence, and confidence, and can be formally challenged.
That’s the honest origin story: Chi is not a speculative agent framework, it’s a postmortem with a CLI.
The run store is enforced, not suggested
Every run gets an append-only SQLite database (WAL mode) at runs/<run_id>/chi.db, and every insert to the events, experiments, and negative-ledger tables also appends one line to a matching JSONL mirror — greppable, rsync-able, vendor-neutral. SQLite is the source of truth; the mirrors are the escape hatch.
The part that matters is who gets to write. The conventional move here is a CONVENTIONS.md and good intentions: ask the agents to log their experiments, record their dead ends, update the shared state. I did the opposite — agents can only write through chi subcommands, and the adapter (not the model) emits the iteration events, heartbeats, and experiment records — because the manual run proved that voluntary compliance selects for exactly the agents you least need bookkeeping from. Bookkeeping is structural or it is fiction.
The store carries more than results. Every evaluated candidate is recorded under a normalized code_hash, so an agent re-proposing an identical candidate gets the cached result back — and gets told it’s repeating itself. The negative ledger records ruled-out approaches with the regime they were ruled out in, and an agent that believes its case is different must file a challenge with a distinguishing hypothesis rather than silently retrying.
One invariant governs all of it: any agent must be reconstructable from the store alone. When an agent’s session ends or the watchdog kills it, Chi doesn’t compact its context — it respawns the agent with a fresh seed context built purely from the store: problem statement, current champion, relevant findings, the negative-ledger slice for its strategy, and the steering in force. The design doc’s list of non-goals says it plainly: context compaction, “never.” Fresh context from durable state beats a lossy summary of a rotting transcript, and the field experiment already proved the recovery path works — an agent came back from a cleared context using the store alone.
Adapters: meet the agents where they already are
A coder in Chi is a process behind one of two production adapters. litellm_loop is a deliberately minimal tools-in-a-loop over LiteLLM — read_file, write_file, run_eval, query_knowledge, report_deadend, nothing fancier — which routes to any LiteLLM-supported model: Anthropic, OpenAI, DeepSeek, GLM, MiniMax. cli_subprocess goes the other way and drives a headless vendor CLI per iteration, so if a vendor’s agentic CLI is where the capability lives, Chi wraps it rather than reimplementing it:
run_name: toy
problem: problems/optimize_function
budgets:
total_usd: 2.0
per_role_usd: { coder: 1.5 }
coders:
- id: c1
model: claude
adapter: cli_subprocess
command: "claude -p --permission-mode acceptEdits {prompt_file}"
policies:
max_iterations: 10
That budgets block is not advisory. Every LLM call’s cost is captured from LiteLLM’s usage data, checked against run and per-role caps before the call is made, and an over-cap call raises, emits a BUDGET_BLOCK event, and ends the run gracefully with a clean final report. An unattended loop with a soft budget is just an expensive loop; the entire premise of walking away depends on the cap being hard.
The watchdog is deterministic on purpose
The obvious way to supervise LLM agents in 2026 is with another LLM. Chi’s watchdog contains zero model calls, by design: a monitor that can hallucinate is a monitor you also have to monitor. Instead it evaluates cheap signals every iteration against thresholds from fleet.yaml:
- Eval recency — more than N iterations without a new row in the experiments table means the agent is looping, whatever its transcript says. Internally this is called the nemotron rule, after the agent that earned it over those thirty hours.
- Repeat detection — the same normalized candidate hash K times in a row earns a “try a different approach” mutation of the next prompt; twice that earns a kill and a fresh respawn.
- Doomed-edit timeouts — an iteration that times out with no measured result triggers an immediate steer toward a smaller, faster edit rather than waiting out the recency counter.
- Heartbeats and leases — a stale heartbeat gets probed then killed, and the task lease expires back to the pool instead of requiring a human audit.
Escalation is graduated — nudge, then mutate the prompt, then kill and respawn — and every kill writes a WATCHDOG_KILL event carrying the triggering signal and its evidence, so a run is auditable after the fact. Boring rules, durably logged, beat clever judgment you can’t replay.
Steering: the human is optional, not required
Steering was the highest-leverage act in the manual experiment — one well-timed epistemic reset in STEERING.md rescued the whole run — but the human was the steering engine, and that doesn’t scale past one sleep-deprived operator. So Chi’s steering is two-layer. The harness itself maintains the directive agents actually run under: an auto-generated digest of run state — champion and score, active hypothesis, dead-end slice, next focus — refreshed at every safe point, so a fully unattended run needs no human input at all. On top of that sits the operator override channel: edit runs/<run_id>/steering.md, or run chi steer runs/<run_id> "stop micro-tuning; try itertools", or — inside an interactive session — just type. Plain text while a run is active becomes a steering directive, folded in at the next iteration boundary, never mid-eval.
And steering is accounted for. Every change emits a STEER_UPDATE event carrying the full content and its hash; every result records the steering hash it ran under; adapters emit STEER_ACK the first time an agent runs under a new hash. An agent ignoring your directives stops being a suspicion and becomes a queryable fact.
One harness, three faces
Day to day, chi opens a full-terminal session built on Textual, with a Claude Code-style interface: scrolling transcript, a bottom input with a slash-command dropdown, a live status bar, and modal fuzzy pickers for vendors and models. /run examples/offline.yaml streams the ★ new best lines in live. chi --plain gives a minimal line-based REPL — also used automatically when stdout isn’t a terminal — and every operator verb works fully non-interactively for scripts and CI:
chi providers --enable anthropic,deepseek
chi models --pick anthropic/claude-sonnet-5,claude
chi validate examples/fleet.yaml
chi run examples/fleet.yaml
chi steer runs/<run_id> "stop micro-tuning; try itertools"
chi champion runs/<run_id> --export best.py
The TUI is sugar; the harness underneath is the same either way. If it only worked with a human watching a pretty terminal, it wouldn’t be an autoresearch harness.
Where it goes from here
Honesty about status: v1 covers Phases 0–1 of the product spec — a single coder agent, both production adapters, the enforced store, budgets, steering, and the watchdog. The store schema, though, was designed for every later phase up front, so nothing that follows needs a migration.
The later phases are already written down in the repo’s design docs: Phase 2 brings orchestrated parallel coders — atomic task claims under contention, a worktree per agent, query-before-work enforcement across the fleet. Phase 3 is where steering grows an LLM brain: plateau detection triggering generated epistemic resets, islands and bandit-style strategy allocation, an escalation ladder, and a negative ledger that blocks (with the challenge mechanism as the appeal process) once multiple agents are running. Phase 4 adds two-tier evaluation — calibrating cheap proxy scores against rationed authoritative benchmarks, which answers the manual run’s most expensive mistake, a sub-noise local “win” that burned a real leaderboard submission — plus remote eval backends and approval checkpoints.
The repo also documents the autonomous director: hand Chi a goal in plain language and it runs the fleet in rounds — run, review, classify, research, steer — with deterministic rules (not the LLM’s mood) deciding whether the run is improving, plateaued, or stuck, researching only when stuck, and stopping itself on a target score or cost ceiling if you gave it one. Even at full autonomy it never fires a ranked leaderboard submission on its own; irreversible public actions stay human, deliberately.
I built Chi because I spent a week being the harness myself, and the parts of that week worth keeping — the blackboard, the negative ledger, the steering file, the champion pointer — deserved to be structural instead of heroic. If you have a problem with a programmatic evaluator and agents you’d like to stop babysitting, uv tool install getchi, run the offline demo, and tell me what breaks: getchi.dev · github.com/kdpisda/chi.