Open Source · MIT Licensed · v0.2
LLMQA is an open-source harness for making LLM quality measurable, reviewable, and gate-able in CI — free, key-optional, and provider-agnostic. Point it at a golden dataset, pick your metrics, and get a pass/fail report with quality gates and regression detection so a model or prompt change can't silently degrade quality.
The hosted dashboard runs the free, deterministic mock
provider only — no API key, no cost. Clone the repo and add your own
ANTHROPIC_API_KEY, OPENAI_API_KEY, or
XAI_API_KEY to evaluate live models.
How it works
-
1
Define a golden dataset. A YAML file of test cases — input, expected answer, tags, and optional grounding context. Each case declares which metric(s) gate its pass/fail.
-
2
Run the evaluation. LLMQA sends every case through your provider and scores each with the metrics you pick — exact match, similarity, LLM-as-judge, hallucination.
-
3
Gate it in CI. A pass-rate threshold or regression check returns a non-zero exit code, so a bad model or prompt change fails the build instead of shipping.
Quality gates
Set --min-pass-rate and the run exits non-zero when quality
drops below the bar. Drop it into any CI pipeline in one line.
Regression detection
Compare against a stored baseline. If the average score falls beyond your tolerance, the build fails with a clear diff of what regressed.
Provider-agnostic
Deterministic mock providers for free, key-optional testing — or bring your own Anthropic, OpenAI, or xAI key for live model evals.
Reviewable reports
Per-case results, gating metrics, latency, and cost — as a console table, Markdown export, or the interactive dashboard.
Get started
Try the interactive dashboard, read the docs, or star the repo.