LLMQA

Why it exists

Shipping with LLMs means quality is a moving target — a new model version, a reworded prompt, or a fresh temperature setting can quietly break behavior that used to work. “Looks better” isn't a diff you can gate a deploy on. LLMQA makes that regression visible and blockable: define a golden dataset once, score every change against it, and fail CI when quality drops. It's the automated version of the kind of manual, case-by-case model evaluation that otherwise doesn't scale.

Who it's for

Anyone shipping a feature backed by an LLM — an app team that reworded a prompt, a platform picking between models, or a library author who needs a regression test for generated output. If you'd write a unit test for a function, LLMQA is how you write one for a model.

Three ways to run it

1. 1
   CLI. pip install llmqa, then llmqa run --provider mock. Quality and regression gates return real exit codes for any pipeline.
2. 2
   Web dashboard. Pick a dataset, watch runs stream in case by case, compare two providers, and track a quality trend — the same app you're reading this in.
3. 3
   GitHub Action. A reusable, drop-in composite Action gates a pull request on quality, writes a JUnit report, and annotates failing cases inline on the diff.

What's included

• Six golden datasets (78 cases). Factual QA, summarization, RAG grounding, code QA, safety refusals, and a mixed baseline — each case declares which metric gates its pass/fail.
• Four metrics. Exact match, similarity, LLM-as-judge, and a hallucination/grounding check.
• Production runner. Parallel execution, per-call retries and timeouts, a cost ceiling, and a persistent response cache.
• Typed & tested. Ships a py.typed marker, is type-checked with mypy, and CI runs the suite across Python 3.11–3.13.

Design principles

• Free by default. Deterministic mock providers mean the full test suite and CI gates run without any API key or cost.
• Provider-agnostic. Bring your own Anthropic, OpenAI, or xAI key for live evals — the harness doesn't care which model you test.
• Reviewable. Every result is a readable table: per-case pass/fail, gating metric, latency, and cost.
• CI-first. Quality gates and regression checks return real exit codes, so they actually stop a bad build.
• Extensible. Adding a dataset, metric, or provider is a small, documented change — see the extending guide.