Turn behavior descriptions into interactive games for studying AI behavior.
Fig selects games, designs interventions, hands implementation to a coding agent, and builds evaluators grounded in the resulting game rules and traces. It keeps construction, player execution, and behavioral judgment separate, with saved prompts, source hashes, validation results, and model provenance.
Behavior description → game selection → intervention → implementation
→ evaluator → player trajectories → judgments
Requires Python 3.11 or later. From a terminal:
git clone https://github.com/fleet-ai/fig.git
cd fig
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
fig --help
python examples/reproduction/offline.py --out runs/offline-demo
fig check-implementation runs/offline-demo/build/implementation --execution localOn Windows, activate with .venv\Scripts\Activate.ps1 in PowerShell.
The package name is fig-behavior; the Python import and command are fig.
The demo includes a small claims game and handwritten model responses. It
intentionally creates a broken adapter, repairs it after a failed check, and
builds and validates an evaluator. The final report should have "status": "success".
Inspect runs/offline-demo/reproduction.json and its build/ and evaluators/
artifacts. This demonstrates the pipeline without network calls; it does not
measure model quality or reproduce the paper's benchmark results.
For just the evaluator builder:
python examples/evaluator/build_demo.py --out runs/evaluator-demoSet OPENAI_API_KEY in your environment, then edit the design, mutation, and
evaluator model IDs in examples/reproduction/live.json to models available
through your provider. The .env.example file documents configuration; Fig does
not automatically load .env files.
fig reproduce examples/reproduction/live.json --out runs/live-demo --execution localThis makes paid model calls and executes generated code. Local execution uses your OS permissions. For isolated checks with a pinned Docker image, see the reproduction guide. Builders stop before player evaluation.
To design interventions using the included catalog descriptions, without installing the native game engines:
fig build --behavior-file examples/secret_channel.txt \
--catalog-file examples/active-behavior-suite/catalog.json \
--model YOUR_MODEL_ID --stop-after design --out runs/secret-channelTo implement catalog interventions, supply your compatible game checkout with
--games /path/to/gameable_games. The full native game catalog, its engine
installations, and historical experiment outputs are not included. The portable
claims game is included. See catalog setup and source locks
and the workflow reference for implementation, generated games,
multiple behaviors, and evaluation APIs.
| Task | Guide |
|---|---|
| Reproduce, resume, or relocate a build | Reproduction |
| Configure implementation agents and budgets | Implementation |
| Build and validate an evaluator | Evaluator builder |
| Run selection from frozen catalog inputs | Selection reruns |
| Configure program, LLM, and hybrid opponents | Opponent controllers |
| Understand behavioral labels and effect measurements | Measurement |
| Browse all stages and Python interfaces | Workflow reference |
| Develop and test Fig | Contributing |
python -m pytest
ruff check src tests
python -m buildThe CI template runs the core suite on Python 3.11–3.13
and verifies offline reproduction from built distributions. To enable it, copy it
to .github/workflows/ci.yml and push with a credential authorized to manage
workflows. Native-engine integration tests are optional; see
Contributing. The wheel contains the library, prompts, and
evaluator recipes. Examples, research tools, tests, and guides are included in
the repository and source distribution.
MIT. External game engines retain their own licenses.