LLMQA

Why it exists

Shipping with LLMs means quality is a moving target — a new model version, a reworded prompt, or a fresh temperature setting can quietly break behavior that used to work. LLMQA makes that regression visible and blockable: define a golden dataset once, score every change against it, and fail CI when quality drops. It's the automated version of the kind of manual, case-by-case model evaluation that otherwise doesn't scale.

Design principles

• Free by default. Deterministic mock providers mean the full test suite and CI gates run without any API key or cost.
• Provider-agnostic. Bring your own Anthropic, OpenAI, or xAI key for live evals — the harness doesn't care which model you test.
• Reviewable. Every result is a readable table: per-case pass/fail, gating metric, latency, and cost.
• CI-first. Quality gates and regression checks return real exit codes, so they actually stop a bad build.