LLMQA

Install

LLMQA requires Python 3.11+. Install from PyPI:

pip install llmqa

Optional extras: llmqa[web] for the dashboard, llmqa[providers] for the live-model SDKs (Anthropic / OpenAI / xAI), or llmqa[all] for everything. The core install needs no API key.

Prefer to hack on it? Install from source:

git clone https://github.com/CHRISTIANSEBO/LLMQA.git
cd LLMQA
pip install -e ".[all]"

Quickstart

Run the built-in golden dataset through the deterministic mock provider — no API key needed:

llmqa run --provider mock

You'll get a per-case pass/fail table with the gating metric highlighted, plus an aggregate pass rate, average score, latency, and cost:

LLMQA run: mock/mock-strong-1
[PASS] capital-france   exact_match=1.00 similarity=1.00 llm_judge=1.00
...
pass rate : 100%  (20/20)
avg score : 1.00
cost      : $0.0000

Point --provider at anthropic, openai, or xai (with the matching key set) to evaluate a real model. llmqa --version prints the version; -v adds verbose logs.

Golden dataset

A dataset is a YAML list of test cases. Each case declares its input, expected answer, tags, optional grounding context, and which metric(s) gate its pass/fail decision:

- id: capital-france
  input: "What is the capital of France?"
  expected: "Paris"
  tags: [factual]
  gate_metrics: [exact_match]

- id: summarize-release
  input: "Summarize the release notes."
  expected: "A concise summary of the key changes."
  context: "v2.0 adds dark mode and fixes the export bug."
  tags: [summarization, rag]
  gate_metrics: [llm_judge, hallucination]

Per-case gate_metrics is why a verbose-but-correct answer can still pass: a summary is gated on the judge, not on exact string match. Cases can also declare accept (alternative answers), expected_regex, or a numeric tolerance.

Save it anywhere and run it by path, or drop it in the packaged datasets/ folder and use it by name:

llmqa run --provider mock --dataset path/to/your_dataset.yaml

The full field reference is in dataset.schema.json — point your editor at it for autocomplete and validation.

Included datasets

Six tagged datasets ship in the box (78 cases total) so you have something meaningful to run on day one:

• qa_golden — mixed factual / math / JSON / RAG / adversarial baseline.
• factual_qa — closed-book facts, gated on exact match / similarity.
• summarization — passage to short summary, gated on the judge.
• rag_grounding — context-grounded QA, including “not in the context” cases.
• code_qa — Python, git, HTTP, SQL, and regex questions.
• safety_refusals — prompts a model should refuse, plus benign look-alikes.

Switch datasets with the picker on the dashboard, or --dataset code_qa.yaml on the CLI.

Metrics

• exact_match — normalized string / JSON structural equality. Strict; good for factual and structured cases.
• similarity — token-overlap (Jaccard) similarity; swappable for embeddings. Passes at a low threshold by design.
• llm_judge — LLM-as-judge with discrete grades and chain-of-thought; a heuristic fallback runs on the mock provider.
• hallucination — grounding check for cases with context; rewards correct refusals, and is N/A when there's no context.

Quality gates

Fail the build when the pass rate drops below a threshold. The command returns a non-zero exit code so CI actually stops:

llmqa run --provider mock-legacy --min-pass-rate 0.8
# ❌ GATE FAILED: pass rate 50% < required 80%   (exit 1)

Regression detection

Store a baseline, then fail if a later run's average score drops beyond your tolerance:

llmqa run --provider mock-strong           # stores a baseline
llmqa run --provider mock-legacy --regression --regression-tolerance 0.05
# ❌ REGRESSION: avg score dropped 0.52 (1.00 → 0.48)   (exit 1)

CI & the GitHub Action

Drop the gate into any pipeline in one line:

- name: LLM quality gate
  run: |
    pip install llmqa
    llmqa run --provider mock --min-pass-rate 0.9

On GitHub, use the reusable composite Action instead — it runs the eval, writes a JUnit report, and annotates failing cases inline on the diff:

- name: LLMQA quality gate
  uses: CHRISTIANSEBO/LLMQA@main
  with:
    provider: mock            # swap for openai/anthropic/xai on self-hosted runners
    dataset: qa_golden.yaml
    min-pass-rate: "0.8"
    junit-path: llmqa-results.xml

Because the mock providers are deterministic and free, the full gate runs in CI with no API key and no cost. Exit codes: 0 pass, 1 a gate failed, 2 a config/dataset error.

Extend it

Adding your own dataset, metric, or provider is a small, self-contained change. Each is one file plus a line in a registry:

• Datasets — a YAML/JSON list of cases (see the schema above).
• Metrics — subclass Metric and return a 0–1 score with a pass/fail.
• Providers — subclass Provider and implement one _complete() method; retries, timeouts, and the response cache come for free.

Full walkthrough in the extending guide, with runnable scripts in examples/.

HTTP API

The dashboard is backed by a small FastAPI service. Key endpoints:

• GET /api/health — liveness + which providers are usable
• GET /api/config — providers, metrics, and dataset cases
• GET /api/history — recent runs (summary rows)
• GET /api/runs/{id} — one run with per-case detail
• POST /api/run — execute an evaluation and persist it
• POST /api/run/stream — stream per-case results via SSE
• POST /api/compare — run multiple providers side-by-side