Skip to content
fleet-aiPublic

About

No description, website, or topics provided.

Resources

Contributing

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

Fig

Turn behavior descriptions into interactive games for studying AI behavior.

Fig selects games, designs interventions, hands implementation to a coding agent, and builds evaluators grounded in the resulting game rules and traces. It keeps construction, player execution, and behavioral judgment separate, with saved prompts, source hashes, validation results, and model provenance.

Behavior description → game selection → intervention → implementation
                    → evaluator → player trajectories → judgments

Quick start: no API key required

Requires Python 3.11 or later. From a terminal:

git clone https://github.com/fleet-ai/fig.git
cd fig
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
fig --help
python examples/reproduction/offline.py --out runs/offline-demo
fig check-implementation runs/offline-demo/build/implementation --execution local

On Windows, activate with .venv\Scripts\Activate.ps1 in PowerShell. The package name is fig-behavior; the Python import and command are fig.

The demo includes a small claims game and handwritten model responses. It intentionally creates a broken adapter, repairs it after a failed check, and builds and validates an evaluator. The final report should have "status": "success". Inspect runs/offline-demo/reproduction.json and its build/ and evaluators/ artifacts. This demonstrates the pipeline without network calls; it does not measure model quality or reproduce the paper's benchmark results.

For just the evaluator builder:

python examples/evaluator/build_demo.py --out runs/evaluator-demo

Generate with a model

Set OPENAI_API_KEY in your environment, then edit the design, mutation, and evaluator model IDs in examples/reproduction/live.json to models available through your provider. The .env.example file documents configuration; Fig does not automatically load .env files.

fig reproduce examples/reproduction/live.json --out runs/live-demo --execution local

This makes paid model calls and executes generated code. Local execution uses your OS permissions. For isolated checks with a pinned Docker image, see the reproduction guide. Builders stop before player evaluation.

To design interventions using the included catalog descriptions, without installing the native game engines:

fig build --behavior-file examples/secret_channel.txt \
  --catalog-file examples/active-behavior-suite/catalog.json \
  --model YOUR_MODEL_ID --stop-after design --out runs/secret-channel

To implement catalog interventions, supply your compatible game checkout with --games /path/to/gameable_games. The full native game catalog, its engine installations, and historical experiment outputs are not included. The portable claims game is included. See catalog setup and source locks and the workflow reference for implementation, generated games, multiple behaviors, and evaluation APIs.

Documentation

Task Guide
Reproduce, resume, or relocate a build Reproduction
Configure implementation agents and budgets Implementation
Build and validate an evaluator Evaluator builder
Run selection from frozen catalog inputs Selection reruns
Configure program, LLM, and hybrid opponents Opponent controllers
Understand behavioral labels and effect measurements Measurement
Browse all stages and Python interfaces Workflow reference
Develop and test Fig Contributing

Development

python -m pytest
ruff check src tests
python -m build

The CI template runs the core suite on Python 3.11–3.13 and verifies offline reproduction from built distributions. To enable it, copy it to .github/workflows/ci.yml and push with a credential authorized to manage workflows. Native-engine integration tests are optional; see Contributing. The wheel contains the library, prompts, and evaluator recipes. Examples, research tools, tests, and guides are included in the repository and source distribution.

License

MIT. External game engines retain their own licenses.

About

No description, website, or topics provided.

Resources

Contributing

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages