Prüfstand — AI Testing Framework
Prüfstand is our test bench for software and AI projects: thirteen specialised agents covering security (SAST, dependency and secret scanning, OWASP), code quality and accessibility, function (unit, E2E, API), load, plus macOS signing and notarisation. An LLM selects the agents that fit a given target, classifies the findings and compiles them into a report. As of July 2026 it is an internal prototype — no CI integration and no customer deployment yet.
Why systematic testing looks different for AI systems
Classical software is deterministic: same input, same output. AI systems are not. The same prompt yields different answers over days, a model update can quietly shift quality, and subjective dimensions (tone, helpfulness, completeness) cannot be covered with assert statements. We therefore keep the two layers apart on purpose: Prüfstand covers what is machine-decidable — vulnerabilities, dependencies, secrets, accessibility, endpoints, load. Judging subjective answer quality lives in our separate eval framework, which uses LLM-as-a-Judge with bias correction.
Architecture — GUI for sparring, CLI for batch runs
Prüfstand has two tiers: an Electron GUI for the exploratory phase and a CLI for repeatable runs. Both go through the same orchestrator and the same test profiles (quick, security-only, standard, full), so a run started in the GUI is reproducible as a command without a translation step. All agents run locally, the planning model optionally via Ollama — no cloud calls for test data. What is missing, we state plainly: results currently live only in the memory of the running process. Persistence, regression comparison over time and CI integration are the next steps, not today's state.
What we are building Prüfstand for
When we accompany customers on their first AI projects, the question 'how do we measure whether this is good enough' usually arrives too late — once the pilot is already running and subjective impressions diverge. The goal of Prüfstand is to be able to establish a shared measurement setup from day one that objectifies the question: what is the baseline, what is 'good enough', how do we know when it gets worse. A translator between engineering vocabulary ('latency, throughput, p95') and business vocabulary ('quality, trust, brand consistency'). Until persistence and regression comparison are in place, we work in mandates with established standard tooling.