LLMQA

Install

LLMQA requires Python 3.11+. Install from source:

git clone https://github.com/CHRISTIANSEBO/LLMQA.git
cd LLMQA
pip install -e ".[all]"

Extras: [providers] for live-model SDKs, [web] for the dashboard, [dev] for tests, [all] for everything. Core install needs no API key.

Quickstart

Run the built-in golden dataset through the deterministic mock provider — no API key needed:

llmqa run --provider mock-strong

You'll get a per-case pass/fail table with the gating metric highlighted, plus an aggregate pass rate, average score, latency, and cost.

Golden dataset

A dataset is a YAML list of test cases. Each case declares its input, expected answer, tags, optional grounding context, and which metric(s) gate its pass/fail decision:

- id: capital-france
  input: "What is the capital of France?"
  expected: "Paris"
  tags: [factual]
  gate_metrics: [exact_match]

- id: summarize-release
  input: "Summarize the release notes."
  expected: "A concise summary of the key changes."
  context: "v2.0 adds dark mode and fixes the export bug."
  tags: [summarization, rag]
  gate_metrics: [llm_judge, hallucination]

Per-case gate_metrics is why a verbose-but-correct answer can still pass: a summary is gated on the judge, not on exact string match.

Metrics

• exact_match — normalized string / JSON structural equality. Strict; good for factual and structured cases.
• similarity — token-overlap (Jaccard) similarity; swappable for embeddings. Passes at a low threshold by design.
• llm_judge — LLM-as-judge with discrete grades and chain-of-thought; a heuristic fallback runs on the mock provider.
• hallucination — grounding check for cases with context; rewards correct refusals, and is N/A when there's no context.

Quality gates

Fail the build when the pass rate drops below a threshold. The command returns a non-zero exit code so CI actually stops:

llmqa run --provider mock-legacy --min-pass-rate 0.8
# ❌ GATE FAILED: pass rate 50% < required 80%   (exit 1)

Regression detection

Store a baseline, then fail if a later run's average score drops beyond your tolerance:

llmqa run --provider mock-strong           # stores a baseline
llmqa run --provider mock-legacy --regression --regression-tolerance 0.05
# ❌ REGRESSION: avg score dropped 0.52 (1.00 → 0.48)   (exit 1)

CI integration

Drop the gate into any pipeline. Example GitHub Actions step:

- name: LLM quality gate
  run: |
    pip install -e ".[all]"
    llmqa run --provider mock-strong --min-pass-rate 0.9

Because the mock providers are deterministic and free, the full gate runs in CI with no API key and no cost.

HTTP API

The dashboard is backed by a small FastAPI service. Key endpoints:

• GET /api/health — liveness + which providers are usable
• GET /api/config — providers, metrics, and dataset cases
• GET /api/history — recent runs (summary rows)
• GET /api/runs/{id} — one run with per-case detail
• POST /api/run — execute an evaluation and persist it
• POST /api/run/stream — stream per-case results via SSE
• POST /api/compare — run multiple providers side-by-side