evaluation-harness

AuditPilot: auditable enterprise AI agents for evidence-grounded workflows, governed tools, evaluation harnesses, human review, and remediation delivery.

An evaluation harness for medical/health AI agents — reproduce and cover multiple benchmarks under one scoring discipline

HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time

The open-source MultiAgentOps evaluation and verification harness for any industry business workflow.

Coding Agent Runtime & Evaluation Harness for controlled repository-level software repair

Teach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it tells you. Deterministic-first, tool-agnostic.

An end-to-end framework for running, sandboxing, and scoring agentic LLMs on complex data-science and econometric replication tasks.

Claude Code/Codex skills, evals, harnesses/context tools, and learning resources that make it easy to build with Jev, TypeSafe's System One model. All materials pull directly from their canonical cookbooks and documentation.

Run 23 speech language models on your own audio task through one interface: vLLM, transformers, and API backends behind a single JSONL contract. Ships HEAR, the speaker-attribution benchmark from our EMNLP 2026 paper.

Agent Reliability Toolkit for LLMs - Test, evaluate, stress-test, and red-team your AI applications with scenario-based testing, multiple evaluators, and multi-provider support.

An sdk and framework for evaluating and comparing multiple model outputs using configurable LLM-based jurors

The luck-robust benchmark for trading agents.

Detecting Relational Boundary Erosion in AI systems. A framework for testing whether models maintain honest, calibrated, and appropriate boundaries.

MODA_NER: open fashion attribute extraction — three-track benchmark suite and models by Hopit AI. Companion to hopit-ai/Moda.

QwerySmith: open text-to-SQL models that answer from retrieved evidence with verifiable citations - plus the harness that trains, evaluates, and releases them (Qwen3-8B QLoRA, 3 seeds, measured-evaluation gate)

A contract-first PDF extraction and document intelligence evaluation harness.

VLA ≠ VLM. Side-by-side viewer running NVIDIA Alpamayo R1 (vision-language-action) alongside Qwen2.5-VL (vision-language) on the same 44-sec SF dashcam clip at 5 Hz. 220 paired traces. Surfaces what an action-trained model sees that a scene-trained model doesn't, and vice versa.

Continuous Evaluation Infrastructure for Production AI

agent-attack-atlas：面向工具型 LLM 智能体的攻击方法图鉴与实测：188 张方法卡+ 13 条确定性判据 + 36 批次 1664 条可离线复算的证据束。

Field-level accuracy evaluation of LLM document extraction — POs, packing slips, bills of lading to schema-validated JSON, with committed scoring reports