Open, independent LLM inference provider benchmark: serverless endpoints measured on end-to-end latency, p99 tail latency, streaming tokens per second and exact task accuracy. Every provider serves the same model (GLM 5.3 Flash) on the same paired micro-tasks, with the same request settings, at the same moment. Providers measured: Baseten, DeepInfra, Fireworks AI, Modal, Nebius, Novita AI, Parasail, Telnyx, Together AI and Z.AI.
Live leaderboard: openbenchmarks.com/inference · Public dataset: openbenchmarks/OB-Inference-Microtasks
Published and maintained by OpenBenchmarks Labs.
The benchmark compares single-user serverless inference endpoints across three short, deterministic task families:
- Meeting-notes lookup (
meeting-notes-lookup) - Ticket triage (
ticket-triage) - Contract-terms extraction (
contract-terms-extraction)
Every provider receives paired streaming requests with fixed parameters (harness/spec.yaml), and judge/ grades exact task correctness deterministically alongside latency (median and p99), throughput (tokens/s), and failure rates. It does not measure model quality across different models, high-concurrency load testing, dedicated endpoints, or long-form generation.
10 serverless inference providers configured in tools/:
baseten, deepinfra, fireworks, modal, nebius, novita, parasail, telnyx, together, zai
System prompt templates for all three task families are committed in data/prompt_templates/.
- Public split: Published on Hugging Face at
openbenchmarks/OB-Inference-Microtasks. - Held-out split: The frozen evaluation set used for the published leaderboard stays private. Place local dataset files under
data/micro-single-task/(gitignored) before running live benchmarks.
See data/README.md for the directory layout, manifest format, and JSONL schema.
data/: System prompt templates (prompt_templates/), dataset layout, and schema documentation.harness/: CLI runner (openbench-inference), run spec (harness/spec.yaml), and provider adapters.tools/: One<tool>.yamlconfig per inference provider (10 tools).judge/: Deterministic answer evaluator, latency/throughput/failure metrics, and report generator.benchmark.yaml: Benchmark contract, scope, and refresh cadence.AGENTS.md: Entry guide for coding agents.
Run the offline test suite (no API keys required):
uv venv && uv pip install -e ".[dev]"
.venv/bin/pytestUse one virtual environment per OB-* repository (all OB-* repos install top-level harness, judge, and tools packages).
To run against live providers, copy .env.example to .env, add your keys, place the dataset as described in data/README.md, and inspect the CLI:
cp .env.example .env
.venv/bin/openbench-inference --help- Add or edit
tools/<tool>.yaml(and.env.exampleif adding a new API key). For a new provider, also add its filename toTOOLSinharness/providers.pyand update the expectedPROVIDER_COUNTand request totals. - Run
.venv/bin/pytest. - Open a pull request.
- OB-Voice-Latency: Real PSTN voice-agent latency (leaderboard)
- OB-Document-Parsing-LegalQA: Document parsers on redlined legal contracts (leaderboard)
- OB-Code-Websearch: Web search APIs for coding agents (leaderboard)
- All OpenBenchmarks leaderboards
MIT. See LICENSE.