How Wallras measures this

What runs where

Wallras is one container on your server, between your app and your AI provider. Prompts, answers and every record stay on your server. On a trial or paid plan, the only thing sent to Wallras is license metadata: your license key, a random installation id and a monthly count of successful requests. Like any connection over the internet, our server also sees the time of each check and your server’s IP address.

Drift radar

Question: is this model behaving like it used to? Measured separately for every model name, so a change in your traffic mix never looks like drift.

• Baseline: the first 200 answers per model (WALLRAS_BASELINE_SIZE). When several apps use the same model, each app (by X-Wallras-App) gets its own baseline, because different apps ask different things: a new app starting on a model is never mistaken for that model drifting. Without X-Wallras-App, all traffic to a model counts as one app. Checks start after 30 more. Until then the status is “collecting” and shows how many requests are left. POST /wallras/baseline/reset accepts current behaviour as the new normal.
• Answer length and wording: how long answers are and the words they use, compared with the model’s own normal, both in their spread and on average, so a sudden change and a slow stretch are both caught.
• Measured against the model’s own variation: a change has to stand out from how much this model naturally varies, so models that are naturally varied aren’t flagged for being themselves.
• Refusals: a rise in “I can’t help with that” answers, recognised in 13 languages.
• Served model: a model version the provider has never served you before.
• Token certainty: where the provider returns token probabilities, a drop in how sure the model was of its own words.
• Speed: answers getting markedly slower.
• Three levels: watch, medium and high, so you choose what deserves an alert.

All of this is deterministic code: no AI model judges your AI. Tested on 10 Nov 2026 with NVIDIA’s Nemotron 3.5 Bolt: after the app’s instructions were changed, Wallras reported refusals up from 0% to 33% and flagged length, wording and speed.

Drift accuracy benchmark

How often does drift detection catch a real change, and how often does it cry wolf? tools/drift_benchmark.py, shipped in the engine image, simulates answer streams from four kinds of model (terse, chatty, highly variable, mixed tasks), each with natural randomness, then applies real drifts of different sizes, plus controls with no change at all. Every case runs 40 times per model with different random seeds: 1,280 cases in all, 200 baseline and 60 recent answers each. Wallras is compared with the rules generic monitoring uses, on exactly the same data.

Wallras catches far more drifts than the best generic rule (78% vs 45%) while raising false alarms about as rarely (1.9% vs 1.3% to 3.1%), and it’s the only one that sees refusals. Two kinds of silent swap are tested. When the provider reports a new version name, Wallras catches it from that name alone, which generic rules miss. When the name stays the same, only the answers can give it away: here the swapped-in model writes very differently, so length rules catch it too. A swap to a model that writes almost the same way would be harder for every method; that is what the judge and regression sets are for. Error-rate monitoring sees nothing, because drifted answers are still successful responses. Limits: the data is simulated (see real models below), small changes over only 60 answers are hard for any method (refusals up 8 points: 35%), and real traffic will differ. Run it yourself with docker run --rm --entrypoint python ghcr.io/wallras/wallras-engine tools/drift_benchmark.py; the numbers are deterministic.

On real models

The benchmark above uses simulated answers. To check it against real model output, we asked three models hosted on NVIDIA (Nemotron 3.5 Bolt, Ising Calibration 1.5 31B and Nemotron 3 Ultra) the same 30 customer-support questions three times each at temperature 0.7, plus once with a changed system prompt (“answer thoroughly”), on 11 November 2026. Nemotron 3 Ultra spent most of its token budget reasoning and returned text for only 18 of 120 questions, so it was left out. Each case was scored on 20 random draws of 60 baseline and 30 recent answers, with the served model name hidden so only the answers could reveal a swap.

On real models, Wallras caught every silent swap and 97.5% of prompt changes. These real changes were large, so simple length rules caught them too; the simulated benchmark above is where Wallras’s wider coverage shows (refusals, subtle wording, swaps behind the same name). With nothing changed, most alerts were at the lower “watch” level. Error-rate monitoring alone saw none of these changes. The answers are saved, so anyone can re-score them: docker run --rm --entrypoint python ghcr.io/wallras/wallras-engine tools/real_model_benchmark.py --offline. This is real model output with staged changes, not production traffic.

How it decided

A decision record shows how the model arrived at one answer. What can be seen depends on where the model runs:

• Hosted models (OpenAI, NVIDIA and others): the provider doesn’t expose the network. Wallras records each token’s probability and the top alternatives when the API returns them (WALLRAS_CAPTURE_LOGPROBS asks on your app’s behalf), and the reasoning text that reasoning models send.
• Self-hosted models (Hugging Face transformers, through the SDK): Wallras reads the model’s internals directly, so probabilities aren’t distorted by sampling settings, and it can see how early in the network each word was settled and which parts of the prompt the model relied on.

The record highlights forks (close calls between two words), unlikely picks, guessed phrases, words decided late and unsure, what the model leaned on, numbers in the answer that the reasoning never reached, and reasoning that used up the token budget before answering.

Built to be trusted: probabilities are exact; layer and attention signals show where the model looked and how settled it was; and reasoning text is checked against the answer rather than taken at face value. Tested on 11 Nov 2026 with Pythia-14M and tiny-GPT2 (GPT-NeoX and GPT-2 architectures) and NVIDIA Nemotron 3.5 Bolt, 3.5 Lightning and 3 Ultra.

Confidence check

Question: when the AI sounds sure, is it right? It needs your feedback (POST /wallras/feedback) on which answers were correct.

• Stated confidence is read from the answer: explicit percentages first, then wording such as “might”, “I think” or “definitely”. Wallras weighs it against the model’s real track record from your feedback, and prefers token probabilities and real outcomes where they exist. Politeness like “I’m sure you’ll love it” doesn’t count.
• For each level of confidence: how often those answers were actually right, with a range that shows how sure that number is.
• The AI is called over- or under-confident only once there’s enough feedback to say so reliably.
• Where the provider exposes token probabilities, the same check runs on those, which are harder to fake than wording.

Blind spot map

Question: where shouldn’t we trust it? Answers are grouped by the X-Wallras-Topic header you send or, without it, by topics Wallras finds itself, on your server with no outside services. A topic is flagged as a blind spot only once there’s enough data to be sure, when:

• accuracy is clearly below the rest, shown with its range;
• refusals are clearly above the rest;
• consistency checks often contradict each other; or
• hidden tests often fail.

Agent traces

Question: where did the agent go wrong? Built only from records already on your server.

• Grouping: requests with the same X-Wallras-Trace header are one run. Without it, Wallras joins the steps of each run automatically by following the conversation, keeping only fingerprints of the messages, and two conversations that start with the same prompt stay separate. If your app rewrites earlier messages between steps, send X-Wallras-Trace instead. One-off chats aren’t shown as runs.
• Per step: model asked for and model served, status, time taken, input and output tokens, tool calls with arguments, and the tool results sent back.
• Flagged: repeated identical tool calls (a likely loop), failed steps, tool results reporting an error, unusually slow steps, unusually long runs, runs that end on a tool request, and answers your feedback marked wrong.
• Cost is shown only when you set WALLRAS_PRICES, from the provider’s own token counts.
• Steps outside Wallras: traces show every model call through Wallras. Tools your app runs, database lookups and API calls appear when your app reports them (POST /wallras/traces/{id}/steps, or one line with the SDK): name, kind, timing and outcome, never inputs or outputs. They’re shown in order with the model calls, and failed ones are flagged.

Regression tests

Question: did our change break anything that used to work?

Wallras uses the right tool for each kind of answer: rules wherever an answer can be checked by rule, similarity to screen, people for the close calls, and an AI judge where quality is a matter of judgement. Similar wording doesn’t make an answer true and different wording doesn’t make it wrong, so no single score decides alone.

• Each item is a request with the answer you accept: usually a real one that went through Wallras, or your own examples imported as JSON Lines. Real items come from a trace, chosen answers, or answers your feedback marked correct (and never marked wrong).
• Default check (similar): the new answer must be close to the accepted one and keep its facts: the same numbers, names and dates, the same yes or no, no negation flipped (“you can” to “you cannot”) and no new refusal (or the reverse). Closeness is by meaning when WALLRAS_EMBED_MODEL is set, otherwise by wording in any word order. Tool-call steps compare the tool and arguments.
• Close calls go to a person: similarity screens answers; it doesn’t prove them. When no rule decided and the score is close to the bar either way, the run lists the item under needs_review with both answers, so someone checks it instead of trusting the number.
• Calibrated: with calibrate, the original model is asked each question again, and the bar for that item is set from how close it gets to its own accepted answer. Questions with many correct phrasings stop failing on wording alone. Costs one extra call per item.
• Judge: for quality that similarity can’t capture, a separate judge model grades each answer against your written rubric, with temperature 0, and returns PASS or FAIL with a reason. Use a different, stronger model than the one being tested; a run with judge checks refuses to start without one. Judges can be wrong too, so check a sample of their reasons.
• With a judge model: pass judge_model to a run and a separate model decides whether each answer says the same thing as the accepted one (similar checks) or whether every claim is supported by the context (grounded checks), so correct paraphrases pass and fluent answers with a different fact fail. Changed numbers and refusals are still caught by rule. Without a judge or an embedding model, runs compare wording and say so in a warning.
• JSON: for structured outputs, the answer must be valid JSON with the keys you list. Exact, contains, regex and number checks are also available.
• Each run is stored with which items passed (not the answers), so the next run reports what newly broke and what was fixed. The pass rate comes with a 95% range.
• Comparisons: the same set run on several models or system prompts at once, each with its pass rate and 95% range and the items they disagree on. Overlapping ranges mean the difference may be chance. Comparisons aren’t added to the set’s run history.
• CI gate: with fail_below, the verdict is fail if the pass rate is below it or anything newly broke.
• Runs call your provider once per item, on your key, and aren’t billed by Wallras.

Hidden tests

Questions you write, with known answers, checked as exact, contains, regex or number. They run roughly hourly at random times (±50%) with the same system prompt and settings as your recent real traffic, so they test the AI as your users get it. They go straight to your provider with your key, aren’t stored as customer traffic, and aren’t billed. Only the questions you add are tested.

Consistency checks

When you turn it on, a small share of real questions (WALLRAS_CONSISTENCY_RATE, off by default; 0.02 is typical) is asked again in different words, after the original answer has been sent. If the answers disagree, it’s asked once more to separate real contradictions from randomness. Changed numbers, or a refusal on one side, count as contradictions.

Model swap diff

Question: what breaks if we switch? Real past prompts (you choose how many) are replayed on the candidate. The current model is also re-asked a sample of them (12 by default) to measure its own natural randomness, and the candidate is judged only on differences beyond that noise:

• similarity of each answer pair, by wording, or by meaning if WALLRAS_EMBED_MODEL is set;
• the share of answers that differ more than the current model differs from itself;
• refusal flips beyond the current model’s own flip rate, and the answer length ratio.

Results include the worst examples side by side and a confidence level, which is low for small samples. Replays cost provider calls on your key, and the key is only ever sent to your own provider.

Tamper evidence

Wallras records are tamper-evident: any change to past records shows.

• Detected: every record (answers, feedback, deletions, the oversight log) carries a hash of the one before it. Editing, inserting or deleting any record breaks the chain, and GET /wallras/integrity shows where.
• Receipts, for the strongest proof: someone with full server access could try to rebuild the whole history, and receipts catch that: a receipt is the latest record number and its fingerprint. Keep it where the server can’t change it (a Slack channel, your log system, a report you sent). A rebuilt history won’t match an older receipt (POST /wallras/integrity/verify).
• Automatic receipts: set WALLRAS_ANCHOR_URL and Wallras sends one every hour (paid feature), signed with WALLRAS_ANCHOR_SECRET if you set one.
• Deletion requests remove the content of matching records but keep their place in the chain, so the check still passes, and the deletion itself is logged.

Speed and failure

Measured with tools/bench.py, shipped in the engine image, which anyone can rerun (docker run --rm --entrypoint python ghcr.io/wallras/wallras-engine tools/bench.py). Setup: a 4-core Windows laptop, Python 3.14, the test suite’s fake provider on the same machine, 400 requests each, 10 Nov 2026.

Expect about 5 to 7 ms per request on modest hardware, next to AI answers that usually take 0.5 to 5 seconds. With 10 requests at once, the median added about 20 ms, because the provider, Wallras and the test shared the same 4 cores; on its own server it’s lower.

• Recording never blocks answering: with Wallras’s database locked by another process, 20 of 20 requests were still answered.
• Provider down: Wallras returned a clear 502 error in about 3.5 seconds instead of hanging.
• Always available: run several containers on Postgres (active-active) so losing one loses nothing, or use automatic restarts with a fallback to your provider (see Running it reliably).

Tested providers

Wallras speaks the OpenAI chat completions API, Anthropic’s Messages API and Google’s Gemini API. “Tested with real traffic” means we sent real requests to the provider through Wallras. “Tested against a test server” means it runs on every release against test servers we built from the provider’s published formats; it hasn’t yet been run against that provider with live keys. The rest are expected to work, but we haven’t verified them ourselves. Each real-traffic row names the engine version it was run on; every release since runs the full test suite, including the test servers.

Not supported in their own formats: other native APIs, such as Cohere’s, or Bedrock’s Converse API. Use them through an OpenAI-compatible address or gateway. Anthropic and Gemini traffic gets every feature, including those that ask the model again (replays, regression runs, swap diff, hidden tests, consistency checks): each request goes back in its own format to its own provider.

What you’re billed for

• One request is one successful model call through Wallras: /v1/chat/completions, /anthropic/v1/messages, or a Gemini generateContent or streamGenerateContent. A streamed answer counts once. An agent that makes five model calls for one task makes five requests.
• Not counted: provider errors and rate limits, failed retries, answers served from the response cache, hidden tests, consistency re-asks, swap-diff replays, step replays, regression test runs and comparisons, and SDK reports about answers that already went through Wallras.
• SDK reports for models running inside your app (not through Wallras) count as one request each.
• Going over: extra requests cost your plan’s per-request rate, billed after the month ends, and a month never costs more than the next plan up. On Free, recording pauses at 10,000 requests; your AI keeps working.
• Yearly billing: pay for 11 months, get 12.
• After cancelling: the plan runs to the end of the paid period, then Wallras steps down to Free. Your records stay on your server.
• Trial: one 7-day trial per email inbox, no card.

Extra provider calls

Wallras bills only your own requests, but two features call your provider on your key, and your provider charges for those:

• Consistency checks: off until you set a rate. At 2% of questions re-asked, plus a follow-up when answers disagree: about 2 to 4 extra calls per 100 requests.
• Hidden tests: one call per test you add, about once an hour each.
• Retries (only if you set WALLRAS_RETRIES): a failed call is tried again, up to the number you set. A response cache, if you switch one on, does the opposite and saves calls.

Example: with consistency checks at 2%, 100,000 requests a month adds roughly 2,000 to 4,000 provider calls, plus about 720 a month per hidden test. By default neither runs, trials included, so Wallras makes no calls of its own until you set WALLRAS_CONSISTENCY_RATE or add a hidden test. The overview suggests turning consistency checks on when your plan includes them. Swap diff runs only when you ask.

Where Wallras fits

Wallras covers monitoring, agent debugging, a gateway, evaluation and prompt releases in one container on your own servers. Specialist tools in each category go deeper in places. So you can decide, here is what’s included and what isn’t:

Getting the most from Wallras

• Accuracy comes from evidence. Drift shows when behaviour changes; your feedback, hidden tests and regression sets prove correctness for the questions that matter to you. Rule checks catch changed numbers, names, dates, yes/no answers, negations and refusals; add meaning comparison or a judge model for open-ended quality, and close calls are listed for a person to check.
• Scale as you grow. The default SQLite file serves one container with several workers. For several containers and no single point of failure, use Postgres (active-active): every container writes to one chain, in order.
• Complete agent traces. Every model call through Wallras is traced; add your app’s own steps (tools, lookups, API calls) directly or through OpenTelemetry for the full picture.
• Decision records show how sure the model was and where it looked; together with your feedback they show where to trust it.
• Confidence sharpens with feedback. Every number comes with a range, so you always see how sure it is.
• Languages: refusals are recognised in 13 languages and hedging in 6, and every language is measured for length, wording and speed.
• Audit-ready evidence. Reports document monitoring and human oversight for your reviews and auditors; certification itself is your auditor’s call.