ai-eval

This course uses a deliberately vulnerable banking application to demonstrate common security vulnerabilities, their impact, and how to fix them. The application is built with Flask (backend) and React (frontend).

Official codebase for "CulturalRiddles: A multicultural benchmark of riddles"

An sdk and framework for evaluating and comparing multiple model outputs using configurable LLM-based jurors

LocalBench is a professional offline-first desktop toolkit for benchmarking, comparing, and profiling local & cloud LLMs. Evaluate models with smart hardware compatibility checks, sequential benchmarking, live performance metrics, dynamic scoring, and side-by-side playground testing. 📊

Agentic Workflow × AI Eval 세미나 자료 — 32장 HTML deck, workflow-orchestrator 개념도, 발표에서 쓴 스킬 3종(workflow-orchestrator·eval-writer·skill-evaluator)

🧪 Evaluation framework for testing Claude Code skills at scale. Run regression suites across model versions.

AI agent evaluation harness: paired per-case diffs, McNemar significance testing, Cohen's kappa judge calibration & bootstrap CIs

AI 聊天教练 MVP：Spring Boot、DeepSeek、结构化输出、两段式分析和轻量评测体系。

Competitive prompt optimization and LLM adversarial evaluation benchmark

Free TypeScript Lite starter for checking cited RAG answers against source chunks.

Socratic reasoning, moral philosophy dilemma benchmark & dialectical essay evaluation engine (Kant, Sartre, Plato, Hegel)

LLM evaluation platform with rubric scoring, A/B judging, and position-bias auditing

Can LLMs write benchmarks other LLMs cannot solve? Creator vs solver evaluation framework for hard-but-fair AI benchmarks.

Local-first, collaborative prompt-engineering platform. Versioned prompts, variables, test cases, moderation. Vanilla JS, zero dependencies, IndexedDB.

Audits AI answers by scoring confidence language against claim verifiability, flagging fluent wrong answers that standard evals miss.

Local Harness

Config-driven CLI that runs promptfoo evals, identifies low-scoring prompts, rewrites them via Claude API, and re-evaluates.

🚀 基于Java的开源AI自动化评测框架 / An open source AI automation evaluation framework based on Java

Scores AI-to-human handoffs on context transfer and redundant questions, catching escalations where the customer has to repeat themselves.