Eval-Framework — LLM-as-a-Judge with bias correction
Our framework for systematic evaluation of LLM outputs: pairwise comparisons, bias corrections (position swap, verbosity, self-preference), calibration against a human reference rating with Spearman ρ as the acceptance threshold. As of July 2026: developed internally and calibrated once, in March 2026. It does not yet run as a standing quality gate and has not been used in a customer mandate.
Why LLM-as-a-Judge — and why with bias correction
Subjective output quality (helpfulness, completeness, tone) does not scale through humans — a sample of 200 answers needs a day of human reviewer time, daily quality gates are economically unworkable. LLM-as-a-Judge is the answer but has three systematic bias problems: position bias (the first comparison entry is preferred), verbosity bias (longer answers are overrated), self-preference (a model prefers its own outputs). Our framework corrects all three with documented methods — position-swap averaging, verbosity normalisation, generator/judge separation.
Calibration against human baselines
A judge model is only usable if its judgments correlate with human judgments. We measure that with Spearman rank correlation (acceptance threshold ρ ≥ 0.7). In the calibration run of 20 July 2026, our default judge qwen2.5:14b reached ρ = 0.81 for relevance and ρ = 0.90 for faithfulness (mean of three runs) — across twenty anchor samples against the ratings of a single human rater. These are lower than the 0.89 and 0.78 we published before, and the reason is worth stating: until then we calibrated against a narrower prompt than the one that runs in production. Measured on the real path, the number drops. We had learned this once already — in RAG-Wissen a faithfulness baseline fell from 0.900 to 0.650 once the runner was pointed at the production path. The rule applies to us too: measure on the production path or do not measure. Further limits, named rather than hidden: twenty samples and one rater are a starting point, not a robust baseline; only two of five dimensions are calibrated, the other three contribute 60 per cent of the overall score and are flagged as unvalidated in the report; and the Krippendorff α measures agreement between the judge models, not agreement with the human. The design provides for injecting 5 calibration samples every 50 evaluations to detect model drift — implemented, but not yet in use for lack of standing production runs.
What the framework is meant for
Anyone running AI pipelines in SME production cannot get by without objective quality measurement — and subjective reviews stop scaling by the second productive pipeline at the latest. The eval framework is designed as a starting point for that: configurable rubrics for common use cases (customer service, knowledge answering, content generation), example scripts, documented bias corrections. The plan is to set it up against the use cases of a mandate and hand it over as an in-house capability — the goal is not vendor lock-in, but internal AI competence. We have not taken that step yet; anyone working with us today gets the method, not a fully bedded-in product.