Skip to content

About

Open, independent benchmark on LLM inference providers: serverless endpoints serving the same model (GLM 5.3 Flash) measured on latency, p99 tail latency, tokens per second and task accuracy. Baseten, DeepInfra, Fireworks AI, Modal, Nebius, Novita AI, Parasail, Telnyx, Together AI, Z.AI.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

OB-Inference

Open, independent LLM inference provider benchmark: serverless endpoints measured on end-to-end latency, p99 tail latency, streaming tokens per second and exact task accuracy. Every provider serves the same model (GLM 5.3 Flash) on the same paired micro-tasks, with the same request settings, at the same moment. Providers measured: Baseten, DeepInfra, Fireworks AI, Modal, Nebius, Novita AI, Parasail, Telnyx, Together AI and Z.AI.

Live leaderboard: openbenchmarks.com/inference · Public dataset: openbenchmarks/OB-Inference-Microtasks

Published and maintained by OpenBenchmarks Labs.

Methodology

The benchmark compares single-user serverless inference endpoints across three short, deterministic task families:

  • Meeting-notes lookup (meeting-notes-lookup)
  • Ticket triage (ticket-triage)
  • Contract-terms extraction (contract-terms-extraction)

Every provider receives paired streaming requests with fixed parameters (harness/spec.yaml), and judge/ grades exact task correctness deterministically alongside latency (median and p99), throughput (tokens/s), and failure rates. It does not measure model quality across different models, high-concurrency load testing, dedicated endpoints, or long-form generation.

Tools evaluated

10 serverless inference providers configured in tools/:

baseten, deepinfra, fireworks, modal, nebius, novita, parasail, telnyx, together, zai

Dataset

System prompt templates for all three task families are committed in data/prompt_templates/.

  • Public split: Published on Hugging Face at openbenchmarks/OB-Inference-Microtasks.
  • Held-out split: The frozen evaluation set used for the published leaderboard stays private. Place local dataset files under data/micro-single-task/ (gitignored) before running live benchmarks.

See data/README.md for the directory layout, manifest format, and JSONL schema.

Repository structure

  • data/: System prompt templates (prompt_templates/), dataset layout, and schema documentation.
  • harness/: CLI runner (openbench-inference), run spec (harness/spec.yaml), and provider adapters.
  • tools/: One <tool>.yaml config per inference provider (10 tools).
  • judge/: Deterministic answer evaluator, latency/throughput/failure metrics, and report generator.
  • benchmark.yaml: Benchmark contract, scope, and refresh cadence.
  • AGENTS.md: Entry guide for coding agents.

Quick start

Run the offline test suite (no API keys required):

uv venv && uv pip install -e ".[dev]"
.venv/bin/pytest

Use one virtual environment per OB-* repository (all OB-* repos install top-level harness, judge, and tools packages).

To run against live providers, copy .env.example to .env, add your keys, place the dataset as described in data/README.md, and inspect the CLI:

cp .env.example .env
.venv/bin/openbench-inference --help

Adding or updating a tool

  1. Add or edit tools/<tool>.yaml (and .env.example if adding a new API key). For a new provider, also add its filename to TOOLS in harness/providers.py and update the expected PROVIDER_COUNT and request totals.
  2. Run .venv/bin/pytest.
  3. Open a pull request.

Links

Related benchmarks

License

MIT. See LICENSE.

About

Open, independent benchmark on LLM inference providers: serverless endpoints serving the same model (GLM 5.3 Flash) measured on latency, p99 tail latency, tokens per second and task accuracy. Baseten, DeepInfra, Fireworks AI, Modal, Nebius, Novita AI, Parasail, Telnyx, Together AI, Z.AI.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages