🚀 New: chi (χ) — an open-source autoresearch harness for fleets of LLM coding agents. Read the announcement.
Kuldeep Pisda

chi

χ — an autoresearch harness: point a fleet of LLM coding agents at a problem with a programmatic evaluator and let them iterate unattended.

Stack
Python · LiteLLM · Textual
Status
active
Source
https://github.com/kdpisda/chi
Website
https://getchi.dev

Chi (“kai”) is an open-source autoresearch harness: point a fleet of LLM coding agents — any vendor — at a problem with a programmatic evaluator (a build, a correctness check, a score) and let them iterate unattended until the champion stops improving. The design is grounded in a postmortem of a hand-run multi-agent fleet: everything that failed by convention there is enforced by structure here.

How it works

A problem is a directory with a problem.yaml: entrypoint commands for correctness and benchmark, held-out seeds, and a score metric with a direction. Correctness is a hard gate, and candidates never see reference outputs. Coders plug in through three adapters — litellm_loop (tools-in-a-loop over any LiteLLM-routable model), cli_subprocess (drives a headless vendor CLI per iteration), and scripted (deterministic playback, powering the test suite and the no-key demo):

coders:
  - id: c1
    model: claude
    adapter: cli_subprocess
    command: "claude -p --permission-mode acceptEdits {prompt_file}"

Runs live in a full-terminal Textual session — scrolling transcript, slash commands, live status bar — where plain text typed during a run becomes a steering directive. Everything also works non-interactively for scripts and CI:

uv tool install getchi
chi run examples/fleet.yaml
chi status runs/<run_id>
chi champion runs/<run_id> --export best.py

Under the hood

  • The run store is enforced SQLite + JSONL — append-only by construction, not by convention, so no agent can rewrite history or lose another’s results.
  • Hard budget caps bound every run, and a deterministic watchdog kills stuck agents instead of trusting an LLM to notice it’s looping.
  • Two-layer steering keeps the operator in control mid-run without restarting: structured commands plus free-text directives, both audit-logged.
  • The offline demo is a design statement about testability: a scripted adapter replays canned candidates and really evaluates each one — the champion replaces an O(n²) prefix-sum baseline with an O(n) itertools.accumulate rewrite, no API key involved.
  • v1 ships Phases 0–1 (single coder agent); the phased design doc in the repo maps the road to parallel fleets and auto-steering.

Published on PyPI as getchi (Apache-2.0), with docs at getchi.dev. Full story in the deep dive.