# Wallras: the whole site as Markdown Current engine version: 0.19.0. Rebuilt from the site and the engine on every release. --- # Wallras: catch AI regressions before your customers do Source: index.html Free 7-day trial, no card needed # Catch AI regressions before your customers do. Wallras watches your AI in production and tells you when it starts drifting, when it’s confidently wrong, and what will break before you switch models. It runs on your servers: prompts, answers and records never leave them. [Start free 7-day trial](checkout?tier=growth) [Watch it catch problems](demo) Example: **support-assistant** on llama-3.1-70b Baseline This week Works with **Anthropic’s and Google Gemini’s own APIs** and any OpenAI-compatible API, such as **OpenAI, Azure OpenAI, NVIDIA NIM, Groq, Together, OpenRouter, DeepSeek, Ollama and vLLM**. [See what we’ve tested, and how](methodology#compat). ## Your AI changes. Nobody tells you. The model you tested is not the model answering your customers today. Most teams find out from a complaint. ### Providers update quietly The same model name can serve a different model next month. Answers shift in length, tone and what the AI is willing to do. ### Confidence isn't accuracy AI sounds just as sure when it's wrong. Without a track record, you can't tell a good answer from a convincing one. ### Switching models is a gamble A cheaper or faster model might handle 90% of your traffic perfectly and quietly break the other 10%. You only find out after. ## Running in about ten minutes Wallras sits between your app and your AI provider as a transparent proxy. Requests pass through untouched. By default Wallras only watches. 1. ### Run it on your servers One Docker container on your own infrastructure. Logs and analysis stay there. 1. ### Change one line Point your AI client at Wallras instead of the provider. Streaming, tool calls and rate limits keep working exactly as before. 1. ### Let it learn normal Wallras builds a baseline for each model from real traffic, then flags anything that moves away from it. ``` # before client = OpenAI(base_url="https://integrate.api.nvidia.com/v1") # after: one line, nothing else changes client = OpenAI(base_url="http://wallras.internal:8080/v1") # ask what changed GET /wallras/drift → status: drifting → "AI is refusing more requests than its baseline" ``` ## Eight ways to check your AI Each one answers a question your team can’t answer today. [How each one is measured](methodology), with thresholds and limits. ### Drift radar **Is my AI still behaving like it did last month?** Tracks answer length, wording, refusals, certainty and the model actually being served, per model, against its own baseline.Live ### Model swap diff **What breaks if we switch models?** Replays your real past prompts on the new model and reports only the differences beyond normal run-to-run randomness.Live ### Confidence check **When it sounds sure, is it right?** Compares how certain your AI sounds with how often it has actually been correct, using feedback from your app.Live ### Blind spot map **Where should we not trust it?** Groups failures by topic and task so you can see exactly where your AI keeps getting things wrong.Live ### Agent traces **Where did the agent go wrong?** Every step of an agent run, with tools called, results, timing, tokens and cost. Flags loops, failing tools and runaway runs, with no code changes.Live ### Regression tests **Did our change break anything?** Turns real answers you trust into a test set, re-runs it on any model or prompt, and shows what newly broke. Pass or fail, ready for CI.Live ### How it decided **Why did it say that?** For each answer: how sure it was of every word, where it nearly said something else, which phrases it guessed, and whether its reasoning matches its answer. With self-hosted models, which layer of the network made each decision.Live ### Compliance reports **Can we prove our AI is monitored?** Turns your local records into evidence reports for audits, clients and your own board. Evidence for reviews, not a certificate of compliance. [See a sample](sample-report).Live ## One container. Four jobs. Teams usually buy a separate tool for each of these. Wallras does all four from the same records, on your own servers. ### General-purpose AI observability **Is our AI healthy today?** - Drift per model against its own baseline: length, wording, refusals, certainty - Catches a different model being served under the same name - Confidence compared with real accuracy, and blind spots by topic - Errors, latency, tokens and cost by model or app, with a Prometheus feed for Grafana - Alerts to Slack, PagerDuty or any webhook when something changes - Many containers, one view: nodes report to a hub, and prompts never leave each node - Active-active: many containers on one database, writing one tamper-evident chain [The eight features](#features) ### Agent development and debugging **Where did the agent go wrong?** - Every step of a run with tools, results, timing, tokens and cost, with no code changes - OpenTelemetry from LangChain, the OpenAI Agents SDK and others, joined to the model calls - Flags loops, failing tools and runaway runs - Replay any step with another model or prompt, and see how each answer was decided [Connect your framework](docs#otel) ### AI gateway and request monitoring **What is going through, and what does it cost?** - One-line switch for Anthropic, Gemini or any OpenAI-compatible provider, self-hosted models included - A searchable log of every request, with usage by model or app - Optional retries, a fallback provider, a response cache and per-app limits, all off by default - Every request recorded in a tamper-evident log - Built-in HTTPS, and nothing leaves your servers [Gateway controls](docs#gateway) ### Evaluation **Is it still right, and will the change break it?** - Regression sets built from real answers you trust, run in CI - Rule-based checks first: JSON Schema, business rules, tool calls, citations, grounding, exact text, numbers with tolerances; meaning or an AI judge where rules can’t decide - A review queue and labelling screen: a person decides close calls, and can accept the better answer - A judge model scores a sample of live answers against your rubric - Works the same on Anthropic, Gemini and OpenAI-compatible traffic - Hidden known-answer tests and consistency checks on live traffic - Compare models or prompts side by side; import and export sets as JSON Lines [How it’s measured](methodology#regression) ### Prompt management **Which prompt is live, and is the new one safe to ship?** - Every prompt version saved, attributed and kept in the tamper-evident log - Production and staging labels your apps read, with no redeploy - A new version goes live only if it passes your regression set; one-call rollback - A browser workspace where your team edits, compares and releases them - Live results per version: errors, latency, cost and accuracy [Prompt management](docs#prompts) ## Wallras doesn't believe your AI. It verifies it. An AI's description of itself is the easiest evidence to get wrong. So every signal is weighted by how hard it is to fake. Real outcomesWas the answer actually right? Hidden known-answer testsSent with the same settings as real traffic Consistency checksSame question, asked differently Token probabilitiesWhere the provider exposes them The AI's stated confidenceDiscounted by its track record The AI's own explanationOften written after the fact If your AI says it's 95% sure but has been right 60% of the time on that topic, Wallras treats it as 60%. ## Sees as deep as your setup allows Wallras works out how much of your AI it can observe, and adapts what it checks. ### Outside-in Hosted APIs Learns from behaviour over time: what goes in, what comes out, and how that changes. ### Hybrid APIs with token probabilities, or self-hosted models Adds the model’s own word-by-word probabilities and checks them against how often the model is actually right. ### Inside-out Self-hosted models, with the Wallras SDK Reads the network itself through the SDK (probabilities from raw logits, the layer where each word was settled, attention) to catch confident-sounding answers the model was actually unsure of. Hosted APIs don’t expose this. ## Many containers. One chain. Put several Wallras containers behind your load balancer, on one Postgres database. Any of them answers any request, and together they keep a single tamper-evident record, as if they were one. ### Any container, any request Feedback, replays and lookups work from whichever container your load balancer picks. Lose one, and the others carry on with every record. ### One write at a time Every container takes the same database lock before it writes, so records join the chain strictly in order. No forks to merge, no gaps to explain. ### Shared, not copied Rate and spend limits, the wrong-token lockout, background checks and jobs hold for the whole fleet, not per container. ### See the fleet The fleet view lists every container and worker, its version and what it served. Mixed versions during a rolling upgrade are flagged. ## Leave any time. Nothing breaks. Most tools hold your data or your workflow hostage. Wallras was built so that walking away costs you nothing. - **Your data stays on your servers**Prompts, answers and reports never leave your infrastructure. - **Only a license key and a count go out**Paid plans send their key and a monthly count of requests, for billing. No prompts, no answers, no logs. Ever. Free sends nothing. - **Stop paying and it steps down**Wallras moves to the Free plan. Your AI keeps running exactly as it did, and your records stay yours. - **Watching never blocks answering**Recording and checks happen after each answer is sent, so a slow or failed analysis never touches your traffic. Run Wallras with automatic restarts, or add a short fallback straight to your provider. **Example weekly report**Status drifting - High New model version served under the same namellama-3.1-70b now resolves to a different build - High AI is refusing more requests than its baseline4% of requests last month, 11% this week - Medium Answer length is shiftingAnswers are 1.6x longer on billing questions - Medium Wording style is driftingMore hedging phrases in support replies [See a full sample report](sample-report), made by the real engine from staged traffic. ## Priced per request. Nothing random. Every price is your plan’s included requests times a per-request rate, and the rate never rises as you grow. Go over and nothing stops: extra requests cost your plan’s rate, and a month never costs more than the next plan up. Every plan starts with a free 7-day trial, no card needed. ### Free $0 forever **10,000** requests a month, free forever - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue [Start free](#how) ### Team $50 / month **25,000** requests × **$0.002** each. Extra requests at the same rate. - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue - Confidence check - Agent traces [Start 7-day trial](checkout?tier=team) ### Starter $200 / month **100,000** requests × **$0.002** each. Extra requests at the same rate. - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue - Confidence check - Agent traces - Model swap diff - Blind spot map - Regression test sets - Add-on: Verification pack - Add-on: Compliance reports - Add-on: Automatic signed receipts [Start 7-day trial](checkout?tier=starter) ### Growth $1,000 / month **1,000,000** requests × **$0.001** each. Extra requests at the same rate. - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue - Confidence check - Agent traces - Model swap diff - Blind spot map - Regression test sets - Hidden tests - Consistency checks - Compliance reports - Add-on: Automatic signed receipts [Start 7-day trial](checkout?tier=growth) ### Scale $4,000 / month **10,000,000** requests × **$0.0004** each. Extra requests at the same rate. - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue - Confidence check - Agent traces - Model swap diff - Blind spot map - Regression test sets - Hidden tests - Consistency checks - Compliance reports - Automatic signed receipts [Start 7-day trial](checkout?tier=scale) ### How the bill works One request is one successful AI answer through Wallras; a streamed answer counts once, and an agent making five model calls makes five. Errors, rate limits, hidden tests and consistency checks are never billed. You’ll get a message at 80% and 100% of your included requests, so nothing is a surprise. Consistency checks and hidden tests call your AI provider on your own key (about 2 to 4 extra calls per 100 requests), and you can switch them off. [Exactly what’s counted](methodology#billing). Extra usage is billed after the month ends. If it would cost more than moving up a plan, you pay the next plan’s price instead, and we’ll suggest the upgrade. On Free, recording pauses when you reach 10,000 requests and resumes next month. Your AI keeps working either way. Need more than 10 million requests? [Talk to us](mailto:hello@wallras.com?subject=Wallras%20Enterprise). ### Add a feature without changing plan | Add-on | On | Price | |---|---|---| | Verification packHidden tests, Consistency checks | Starter | $50 / month | | Compliance reports | Starter | $75 / month | | Automatic signed receipts | Starter | $25 / month | | Automatic signed receipts | Growth | $250 / month | Add-ons are priced per request too, on your plan’s included volume. Prices exclude taxes, which are added at checkout where they apply. ## Try it free for 7 days Start any plan’s trial in a minute, no card needed. You get the full plan, and direct access to the person building it. [Start free trial](checkout?tier=growth)[Watch the live demo](demo) - **7 days free.** No card needed, and it ends on its own. - **About 10 minutes** to set up. - **Pricing is public,** so the trial ends with a clear choice. - **Walk away any time** and your AI runs exactly as before. ## Questions ### Does Wallras slow down my AI? Wallras passes each request straight through and records it after the answer has gone back, so it adds one short hop inside your own infrastructure: about 5 to 7 milliseconds per request in our benchmark on a 4-core laptop, next to AI answers that take 0.5 to 5 seconds. If Wallras’s storage fails, answers still go through. [How we measured](methodology#performance). ### Can Wallras see my data? No. Wallras runs on your servers, and prompts, answers and records stay there. On a trial or paid plan it sends us only your license key, a random installation id and a monthly count of requests, so we can renew your license and bill extra usage. Our server also sees the time of each check and your server’s IP address, as with any connection. On Free, it sends nothing at all. Your account and payment details are handled separately, as the [privacy policy](privacy) explains. [See the data flow](methodology#dataflow). ### Which AI providers does it work with? Anthropic’s Messages API and Google’s Gemini API in their own formats, and any OpenAI-compatible API: OpenAI, Azure OpenAI, NVIDIA NIM, Groq, Together, OpenRouter, DeepSeek, Fireworks, and self-hosted models on Ollama, vLLM, LM Studio or llama.cpp. We’ve run real traffic through NVIDIA NIM, Google Gemini, OpenRouter (28 models from 24 companies, including OpenAI and Anthropic Claude, and Claude in Anthropic’s own Messages format) and a self-hosted model on llama.cpp. [Tested providers](methodology#compat). ### Will Wallras add to my AI provider bill? Only if you turn it on. Consistency checks are off until you set `WALLRAS_CONSISTENCY_RATE`: at 0.02 they re-ask 2% of questions (roughly 2 to 4 extra calls per 100 requests). Each hidden test you add runs about once an hour. Both use your own key. Out of the box, trials included, Wallras makes no calls of its own. [Worked example](methodology#extra). ### How does Wallras decide something has drifted? It learns each model’s normal from its first 200 answers, then compares recent answers on length, wording, refusals, the model actually served, certainty and speed. A change is flagged only if it’s both large and bigger than this model’s own random variation, so naturally varied models don’t cry wolf. Every threshold is published. [The method](methodology#drift). ### Does Wallras change how my AI answers? Not unless you ask it to. Out of the box Wallras only observes: it doesn't rewrite prompts, filter answers or modify your model, and removing it changes nothing about your AI's behaviour. If you switch on the optional gateway controls, an identical request can be answered from a short-lived cache and a backup provider can answer when yours is down; responses say when that happened. ### What happens when I stop paying? Your plan stays active until the end of the period you paid for, plus a 7-day grace period if a payment fails. Then Wallras steps down to the Free plan: core features keep working, paid features switch off, and traffic always flows. Your past records stay on your servers. ### Does Wallras make extra calls to my AI provider? A few, and only on your key. To check your AI without trusting it, Wallras can re-ask a small share of real questions in different words (off until you set it; 2% is typical), ask once more when answers disagree, and run hidden tests with known answers. At 2% that's about 2 to 4 extra calls per 100 requests, plus one call an hour for each hidden test you add. Nothing runs until you turn it on. ### Can records be edited after the fact? Not without it showing. Every record, from answers to feedback to the oversight log, is chained to the one before it, so editing, adding or deleting any of them is detected. Someone with full server access could try to rebuild the whole chain, which is why Wallras issues receipts: small fingerprints you keep elsewhere, like a Slack channel or your compliance reports. A rebuilt history won't match them. ### What if someone asks us to delete their data? Search by any detail, like an email address, and Wallras removes that content from its records. The tamper check keeps working, and the deletion is logged so you can show you handled the request. ### How is this different from logging tools? Logging tools show you what your AI said. Wallras tells you whether to believe it: whether its behaviour is drifting, whether its confidence matches its track record, and what will change before you switch models. --- # Docs | Wallras Source: docs.html # Docs From zero to watching your AI in about ten minutes. For how every check works, its thresholds and its limits, see [How Wallras measures this](methodology). ## Quick start Run Wallras next to your app, on your own server. It’s free up to 10,000 requests a month, no account needed. ``` docker run -d --restart unless-stopped -p 8080:8080 -v wallras-data:/data \ -e WALLRAS_UPSTREAM=https://api.openai.com/v1 \ -e WALLRAS_ADMIN_TOKEN=choose-a-long-secret \ ghcr.io/wallras/wallras-engine:0.19 ``` `:0.19` gets every 0.19.x fix automatically. Pin an exact version such as `:0.19.0` to control upgrades, or use `:latest` to always follow the newest release. Then change one line: point your AI client at Wallras instead of your provider. Your provider key stays the same. ``` client = OpenAI(base_url="http://localhost:8080/v1", api_key=YOUR_PROVIDER_KEY) ``` Wallras is built for any OpenAI-compatible API, such as OpenAI, NVIDIA NIM, Groq, Together, OpenRouter, DeepSeek, Fireworks, xAI, and self-hosted Ollama, vLLM, LM Studio or llama.cpp. Streaming, tool calls and rate-limit headers pass through untouched, and every answer comes from your provider unchanged unless you switch on the [gateway controls](#gateway). It learns each model’s normal behaviour from your first 200 answers per model (per app too, when several apps send `X-Wallras-App` to the same model, so one app’s different questions never look like another’s drift), then starts checking after 30 more (set `WALLRAS_BASELINE_SIZE` lower to start sooner). Check it any time; until then it shows how many requests are left: ``` curl localhost:8080/wallras/drift -H "X-Wallras-Token: choose-a-long-secret" ``` ## Anthropic, Google Gemini and other providers Apps that use Anthropic’s or Google’s own APIs point their SDK at Wallras too. The request and the answer pass through byte for byte, including streaming, tool use and thinking, and each call is recorded like any other: drift, request log, usage and costs, agent traces (tool calls are joined into one run), metrics and the tamper-evident ledger. ``` # Anthropic: Wallras sends to WALLRAS_ANTHROPIC_UPSTREAM (default https://api.anthropic.com) client = anthropic.Anthropic(base_url="http://localhost:8080/anthropic", api_key=YOUR_ANTHROPIC_KEY) # Google Gemini: Wallras sends to WALLRAS_GEMINI_UPSTREAM (default https://generativelanguage.googleapis.com) client = genai.Client(api_key=YOUR_GOOGLE_KEY, http_options={"base_url": "http://localhost:8080/gemini"}) ``` - **Anthropic:** `POST /anthropic/v1/messages` is recorded; other paths (token counting, model lists, batches) pass through untouched. The `x-api-key` header goes only to Anthropic and is never stored. Prompt-cache tokens count as input tokens. - **Gemini:** `generateContent` and `streamGenerateContent` are recorded, with or without `alt=sse`. A key in `x-goog-api-key` or `?key=` goes only to Google and is never stored. With `responseLogprobs` on, token probabilities feed the decision record. For Vertex AI, set `WALLRAS_GEMINI_UPSTREAM` to your regional address (for example `https://us-central1-aiplatform.googleapis.com`) and keep the Vertex path. - **Gateway controls:** retries, per-app limits, the response cache and a fallback all apply. The fallback for this traffic is in the same format (another account, region or compatible gateway): see [Gateway controls](#gateway). - **Every evaluation feature works on this traffic too.** Replays, regression runs and comparisons, model swap diff, hidden tests and consistency checks send each request back in the format it came in, to the provider it came from: send the provider’s key the same way your app does (`x-api-key` for Anthropic, `x-goog-api-key` for Gemini). For swap diff, add `"provider": "anthropic"` or `"gemini"`. To try a request on another provider, pass `"provider": "openai"` (or the other two) to a replay or run. Background checks use `WALLRAS_ANTHROPIC_KEY` and `WALLRAS_GEMINI_KEY` if set, otherwise the last key seen, held in memory only. - **Azure OpenAI, AWS Bedrock and other gateways** with an OpenAI-compatible address work through `/v1`: set `WALLRAS_UPSTREAM` to it, for example `https://your-resource.openai.azure.com/openai/v1`. Their own key headers, such as Azure’s `api-key`, pass through. See [which providers we’ve tested](methodology#compat) and how. ## Plans and licenses Start a free 7-day trial from your [account](account), copy your license key, and install it. No restart needed: ``` curl -X POST localhost:8080/wallras/license -H "X-Wallras-Token: …" \ -H "Content-Type: application/json" -d '{"token": "wr1.…"}' ``` Or set `WALLRAS_LICENSE` when starting the container. Renewals install themselves. If a plan ends, Wallras steps down to Free; your AI keeps working. `GET /wallras/plan` shows your plan, this month’s usage and whether recording is on. ## Feedback: tell Wallras what was right Every answer carries an `X-Wallras-Event-Id` header. Send back whether it was right, and the confidence check and blind spot map start working. Give your app the feedback-only token, which can’t read anything: ``` resp = client.chat.completions.with_raw_response.create(model=..., messages=...) event_id = resp.headers["x-wallras-event-id"] # later, when you know requests.post("http://localhost:8080/wallras/feedback", headers={"X-Wallras-Token": WALLRAS_APP_TOKEN}, json={"event_id": event_id, "correct": False, "note": "wrong refund policy"}) ``` ## Topics Add `X-Wallras-Topic: billing` to a request to group it in the blind spot map. Without it, Wallras finds topics itself. The header never reaches your provider. ## Request log, usage and metrics Every request through Wallras is in a log you can filter and search, and usage adds up by model, app or day: requests, errors, p50 and p95 latency, tokens and cost (set `WALLRAS_PRICES` for cost). Name the app sending each request with `X-Wallras-App: support-bot`; the header never reaches your provider. Available on every plan. ``` curl "localhost:8080/wallras/usage?hours=24&by=app" -H "X-Wallras-Token: …" curl "localhost:8080/wallras/requests?status=error&model=gpt-5" -H "X-Wallras-Token: …" curl "localhost:8080/wallras/requests?q=refund&limit=20" -H "X-Wallras-Token: …" # text in the question or answer ``` Search finds requests whose question or answer contains every word, in any order, ignoring case in any language: `q=priya refund`. Put a phrase in double quotes to keep it together: `q="order A-1042"`. It matches text, not meaning. For Grafana, Datadog or any Prometheus scraper, `GET /wallras/metrics` gives requests, tokens, cost, latency, gateway events and drift per model. Prometheus can only send a bearer token, so this one endpoint also accepts your Wallras token there (an admin or viewer token): ``` scrape_configs: - job_name: wallras metrics_path: /wallras/metrics authorization: {credentials: } static_configs: [{targets: ["wallras.internal:8080"]}] ``` ## Gateway controls Out of the box Wallras only watches: every answer comes from your provider, unchanged. These are there when you want them, each off until you set it. Available on every plan, and every control works on OpenAI-compatible, Anthropic and Gemini traffic. | Setting | What it does | |---|---| | `WALLRAS_RETRIES=2` | Retries rate limits, server errors and dropped connections, with backoff (honouring `retry-after` up to 8 seconds). Streams are retried only before the first byte reaches your app. A provider that failed with a 5xx may still have charged for the attempt. | | `WALLRAS_FALLBACK_UPSTREAM`, `WALLRAS_FALLBACK_KEY`, `WALLRAS_FALLBACK_MODELS` | A second OpenAI-compatible provider, used when yours still fails after retries. It gets its own key, never yours; Wallras won’t start with a fallback address but no key. Map models with `{"gpt-5": "other-model", "*": "default"}`. Answers it gives carry `X-Wallras-Fallback`. | | `WALLRAS_ANTHROPIC_FALLBACK_UPSTREAM`, `WALLRAS_ANTHROPIC_FALLBACK_KEY`, `WALLRAS_ANTHROPIC_FALLBACK_MODELS` (and the same three for `GEMINI`) | A fallback for Anthropic or Gemini traffic, with its own key, used when the main one still fails after retries. By default it’s in the same format (another account, region or compatible gateway), streaming included. Answers carry `X-Wallras-Fallback`. | | `WALLRAS_ANTHROPIC_FALLBACK_FORMAT=openai` (or `WALLRAS_GEMINI_FALLBACK_FORMAT`) | The fallback is an OpenAI-compatible provider instead: the request is translated for it, and its answer, tool calls included, is rewritten in Anthropic’s or Gemini’s format, so your app doesn’t notice. A streamed request gets the backup’s whole answer as a stream of events once it’s ready. | | `WALLRAS_CACHE_TTL=300` | Answers an identical request from memory for this many seconds. Only non-streamed requests with `temperature` 0, or any request sent with `X-Wallras-Cache: on`; `X-Wallras-Cache: off` skips it. Kept per provider key, in memory only, and cleared by deletion requests. Works the same for Anthropic and Gemini requests. Hits carry `X-Wallras-Cache: hit` and aren’t billed or recorded again. | | `WALLRAS_LIMITS` | Per-app limits, by `X-Wallras-App`: `{"support-bot": {"requests_per_minute": 60, "tokens_per_day": 2000000, "usd_per_day": 20}, "*": {…}}`. Over a limit, Wallras answers 429 with `retry-after` and doesn’t call your provider. Days are UTC. | ## Agent traces Follow every step of an agent run: which model answered, how long each step took, tokens, the tools it called with their arguments, what the tools sent back, and where it went wrong. Works with no code changes: Wallras groups an agent’s calls automatically by following the conversation, since every step re-sends the earlier messages plus the answer it got. To name runs and steps yourself, add headers (they never reach your provider): ``` client.chat.completions.create(model=..., messages=..., extra_headers={ "X-Wallras-Trace": run_id, # one id per agent run "X-Wallras-Step": "search", # optional step name "X-Wallras-Session": chat_id}) # optional: group runs into a conversation ``` ``` curl "localhost:8080/wallras/traces?status=failed" -H "X-Wallras-Token: …" curl localhost:8080/wallras/traces/{trace_id} -H "X-Wallras-Token: …" curl "localhost:8080/wallras/traces/compare?a={good_run}&b={bad_run}" -H "X-Wallras-Token: …" ``` Each run is marked ok, check or failed. Wallras flags tool loops (the same tool with the same arguments 3 or more times), failed steps, tools that returned errors, unusually slow steps, runs longer than 25 steps, runs that stopped mid-tool-call, and answers your feedback marked wrong. Set `WALLRAS_PRICES` to see each run’s cost. Available on Team and above. Wallras only sees model calls that go through it. Report your app’s own steps (tools it runs, lookups, API calls) so they appear in the same run. Only the name, kind, timing and outcome are sent: ``` from wallras_sdk import Wallras w = Wallras("http://localhost:8080", token=WALLRAS_APP_TOKEN) with w.step(run_id, "search_flights", kind="api"): # timed; a failure is recorded and re-raised results = search(...) # or without the SDK POST /wallras/traces/{trace_id}/steps {"name": "search_flights", "kind": "api", "duration_ms": 840, "status": "ok"} ``` **Replay a step.** Send any recorded request again with a different model, system prompt, messages, temperature or token limit, and see the new answer beside the original, with how similar they are and whether the tool calls match. Replays use your key and aren’t recorded as traffic: ``` curl -X POST localhost:8080/wallras/events/{event_id}/replay -H "X-Wallras-Token: …" \ -H "Authorization: Bearer $PROVIDER_KEY" -H "Content-Type: application/json" \ -d '{"model": "gpt-5-mini", "system": "Always check the refund window first."}' ``` ## Regression test sets Turn real answers you’re happy with into tests, then re-run them after any change: a new model, a new prompt, a new provider version. Add items from a trace, from chosen answers, or from everything your feedback marked correct: ``` curl -X POST localhost:8080/wallras/datasets -H "X-Wallras-Token: …" \ -H "Content-Type: application/json" -d '{"name": "support-golden"}' curl -X POST localhost:8080/wallras/datasets/support-golden/items -H "X-Wallras-Token: …" \ -H "Content-Type: application/json" -d '{"from_feedback": "correct", "limit": 100}' # run it on a candidate model; in CI, fail the build below 95% curl -X POST localhost:8080/wallras/datasets/support-golden/run -H "X-Wallras-Token: …" \ -H "Authorization: Bearer $PROVIDER_KEY" -H "Content-Type: application/json" \ -d '{"model": "gpt-5-mini", "fail_below": 0.95}' ``` For agents, `"check": "tool"` with a value like `search_flights:to=Paris` requires the step to call that tool with those arguments. For RAG, `"check": "grounded"` requires every sentence of the answer to be supported by the context the model was given, with every number in it. Each answer passes when it’s close to the accepted one (by meaning if `WALLRAS_EMBED_MODEL` is set) with the same numbers and no new refusal. Add `"calibrate": true` to set each item’s bar from how much the original model varies. Without an embedding model, answers are compared by wording, in any word order, plus fact checks: a changed number, name, date, yes/no answer, negation or refusal fails even when the rest matches. Every such run says so in a `warning`. Pass a `"judge_model"` to any run and a separate model decides instead: for similar checks, whether each answer says the same thing as the accepted one; for grounded checks, whether every claim is supported by the context. Changed numbers and refusals are still caught by rule. For task-specific quality, create a set with `"check": "judge"` and your rubric as the value, then run it with a `"judge_model"`. `"check": "json"` with a list of keys checks structured outputs, and `"check": "schema"` with a JSON Schema checks types, required fields, enums, ranges and patterns. `"check": "rules"` takes business rules such as `{"contains": ["refund"], "not_contains": ["guarantee"], "max_words": 80}`. `"check": "number"` takes a tolerance: `"3.14 +- 0.01"` or `"100 +- 2%"`. `"check": "citations"` requires every [1], [doc-id] or link in the answer to be one of the sources the model was given, and each cited sentence to be supported by its source (value `"optional"` allows answers with no references). Exact, contains and regex checks work too. Items whose verdict rests on similarity alone and lands close to the bar are listed under `needs_review`, for a person to check. Every run reports the pass rate with its 95% range, what newly broke and what got fixed since the last run, and `"verdict": "pass"` or `"fail"`. Runs make one provider call per item on your key and aren’t billed. Deletion requests also clear matching items. Available on Starter and above. **Bring your own examples** as JSON Lines, one per line, with `question` (or full `messages`), `expected`, and optionally `model`, `system`, `check` and `value`. Export any set the same way, to keep it in git: ``` curl -X POST localhost:8080/wallras/datasets/support-golden/import -H "X-Wallras-Token: …" --data-binary @golden.jsonl curl localhost:8080/wallras/datasets/support-golden/export -H "X-Wallras-Token: …" > golden.jsonl ``` **Compare side by side.** Run one set on up to six models or system prompts at once and see each pass rate with its range, the fastest, and exactly which items they disagree on. Comparisons don’t change the set’s run history. To try a new system prompt in a normal run, add `"system"`: ``` curl -X POST localhost:8080/wallras/datasets/support-golden/compare -H "X-Wallras-Token: …" \ -H "Authorization: Bearer $PROVIDER_KEY" -H "Content-Type: application/json" \ -d '{"candidates": [{"model": "gpt-5"}, {"model": "gpt-5-mini"}, {"model": "gpt-5", "system": "…new prompt…", "label": "new prompt"}]}' ``` ## Prompt management **Question:** which prompt is live, who changed it, and did the change break anything? Available on every plan; gated releases need regression sets (Starter and above). - **Versions:** save a prompt (chat messages or one system text, with `{{variables}}`, plus an optional model and settings) and each save becomes a new numbered version. Versions never change; saving the same content again returns the existing version. Each records who saved it (the token’s name, or the SSO user) and a note, and goes into the tamper-evident ledger, so prompt history can’t be quietly rewritten. - **Environments:** labels such as `production` and `staging` each point at one version. Your app asks for the label; moving it changes what every app gets, with no deploy. - **Gated releases:** promote a version with a regression set as the gate. Wallras runs the set with that version’s system prompt and model first, and moves the label only if it passes. A blocked promotion answers 409 with the failures, and is logged. Every move, gated or not, keeps who, why and the gate result. - **Rollback** puts a label back on the version it pointed at before, in one call. - **Compare versions:** a line-by-line diff of any two versions, and both versions side by side on a regression set as candidates (`{"prompt": "support@3"}`), saved as an experiment. - **Live results per version:** send `X-Wallras-Prompt: support@3` with each model call (the resolve call returns it) and every request is filed under its version: requests, errors, latency, tokens and feedback accuracy for each. - **Roles:** admins save and promote, viewers read, and app tokens can only fetch the prompt to send. - **Variables in a gate:** a regression gate fills the version’s `{{variables}}` with test values you give (`"vars": {"company": "Acme"}`). - **Long gates:** a large regression set can take minutes. Send `"wait": false` and the release answers 202 with a job to check at `GET /wallras/jobs/{id}`; the label moves only if the gate passes, as before. ``` # save a version, then release it only if it passes the regression set curl -X POST localhost:8080/wallras/prompts -H "X-Wallras-Token: …" -H "Content-Type: application/json" \ -d '{"name": "support", "system": "You help {{company}} customers. Be brief.", "model": "gpt-5-mini", "note": "shorter"}' curl -X POST localhost:8080/wallras/prompts/support/promote -H "X-Wallras-Token: …" -H "Authorization: Bearer $PROVIDER_KEY" \ -H "Content-Type: application/json" -d '{"version": 2, "label": "production", "gate": {"dataset": "support-golden", "fail_below": 0.95}}' # in your app (Python SDK): the production version, cached, with a fallback if Wallras can't be reached p = w.prompt("support", company="Acme", fallback={"messages": [{"role": "system", "content": "You help Acme customers."}]}) client.chat.completions.create(model=p["model"] or "gpt-5-mini", messages=p["messages"] + [user_msg], extra_headers=p["header"]) ``` Edit prompts in the [browser workspace](#workspace), through the API or the SDK, or keep them in git and save them from CI. Every change is attributed and audited, however it’s made. ## Human review **Question:** which answers should a person look at? Automatic checks screen; people decide the close calls. The queue fills itself with regression items whose verdict rested on similarity alone near the bar, live answers that contradicted themselves when asked again, and live answers the judge failed. Add any recorded request too. Decide in the [browser workspace](#workspace) or through the API. Available on every plan. - **Decide** correct or wrong, with a note. On a live request the decision is recorded as feedback, so it feeds accuracy, confidence and the blind spot map. - **Accept a better answer:** on a regression item judged correct, `"accept_answer": true` makes this answer the new accepted one. The old item is retired, not edited, so the ledger keeps both. - Every decision is in the audit log with the reviewer’s name. Deletion requests clear the stored text. ``` curl localhost:8080/wallras/review -H "X-Wallras-Token: …" curl -X POST localhost:8080/wallras/review/12 -H "X-Wallras-Token: …" -H "Content-Type: application/json" \ -d '{"verdict": "correct", "accept_answer": true, "note": "same fact, new wording"}' ``` ## Overview and next actions One call shows everything that needs attention: drift, silent model swaps, failed agent runs, failing regression sets and hidden tests, confidence and decision records. It ends with next actions, each with the exact call to make: ``` curl localhost:8080/wallras/overview -H "X-Wallras-Token: …" → "next_actions": [{"severity": "high", "what": "The provider served support-model as different models: v1, v2.", "how": "POST /wallras/datasets/{name}/run …"}, …] ``` Make detection lead straight to verification: watch a regression set and it re-runs itself on a schedule, when the provider silently swaps the model, and when drift starts. Each automatic run is logged with why it ran. ``` curl -X POST localhost:8080/wallras/datasets/golden/watch -H "X-Wallras-Token: …" \ -H "Content-Type: application/json" -d '{"every_hours": 24, "on_swap": true, "on_drift": true}' ``` ## How it decided Each answer can carry a decision record: how the model got there. For every token it shows how likely it was and what else the model nearly said, where it nearly went another way (forks), where sampling picked a less likely word, which phrases it was guessing, and its reasoning checked against the answer. ``` curl localhost:8080/wallras/events/{event_id}/explain -H "X-Wallras-Token: …" → "how_it_got_there": ["Across 52 tokens the model was typically 97% sure …", "It reached 1 fork … it chose “1” (56%) over “Calc” (44%).", …] curl localhost:8080/wallras/decisions -H "X-Wallras-Token: …" # shakiest answers first ``` **Hosted models** (OpenAI, NVIDIA and others) don’t expose their networks, so Wallras records what the API returns: token probabilities with alternatives, and reasoning text from reasoning models. Set `WALLRAS_CAPTURE_LOGPROBS=1` and Wallras asks for probabilities on your app’s behalf and removes them before your app sees the answer (non-streamed requests; if a provider refuses, the original request is sent). **Self-hosted models**: the SDK reads the network itself. Per token it also records the layer where the model settled on it and which prompt words the last layer attended to: ``` model = AutoModelForCausalLM.from_pretrained(name, attn_implementation="eager", dtype=torch.float32) out = model.generate(**inputs, max_new_tokens=256, return_dict_in_generate=True, output_logits=True, output_hidden_states=True, output_attentions=True) w.explain_transformers(model, tokenizer, inputs, out, model_name="llama-3.1-8b", prompt=prompt) ``` Available on Team and above. ## OpenTelemetry Send traces from any OpenTelemetry-instrumented framework (LangChain, LlamaIndex, CrewAI, the OpenAI Agents SDK, Haystack and others, through their OpenTelemetry, OpenLLMetry or OpenInference instrumentation). Protobuf and JSON both work: ``` OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:8080/wallras/otlp/v1/traces OTEL_EXPORTER_OTLP_HEADERS=x-wallras-token= ``` Model calls that go through Wallras join the same trace automatically, and a call seen both ways counts once. When your HTTP client sends the `traceparent` header, that decides it. Some clients don’t (recent OpenAI Python SDKs use a client OpenTelemetry can’t instrument), so Wallras also links each model-call span to the call through the proxy with the same model that started inside that span. Tested with LangChain (`openinference-instrumentation-langchain`) and the OpenAI Agents SDK (`openinference-instrumentation-openai-agents`). With the Agents SDK, keep its tracing on and replace its processors with OpenInference’s, so spans come to Wallras and none go to OpenAI: ``` OpenAIAgentsInstrumentor().instrument(tracer_provider=provider) from agents.tracing import get_trace_provider, set_trace_processors set_trace_processors([p for p in get_trace_provider()._multi_processor._processors if "openinference" in type(p).__module__]) ``` Spans are sorted into agent, model call, tool, retrieval, database and API steps. Only names, timing, status, model names and token counts are kept; prompts, completions and tool arguments are dropped on arrival. ## Hidden tests Questions with known answers, sent hourly with the same system prompt and settings as your real traffic. They go straight to your provider and never appear in your records. ``` curl -X POST localhost:8080/wallras/canaries -H "X-Wallras-Token: …" -H "Content-Type: application/json" \ -d '{"question": "What is our refund window?", "expect_type": "contains", "expect_value": "30 days", "topic": "billing"}' ``` `expect_type` is `contains`, `exact`, `regex` or `number`. ## Model swap diff Before switching models, replay real past prompts on the new one: ``` curl -X POST localhost:8080/wallras/swap-diff -H "X-Wallras-Token: …" -H "Authorization: Bearer $PROVIDER_KEY" \ -H "Content-Type: application/json" -d '{"candidate_model": "gpt-5-mini", "current_model": "gpt-5", "sample_size": 30}' ``` ## Inside-out mode with the SDK For self-hosted models, the SDK sends what the model computed (how sure it was of each word) so Wallras can catch answers that sound certain while the model was unsure. It works for models behind the proxy and for models running inside your app. [Download wallras_sdk.py](assets/wallras_sdk.py) and put it next to your code. It’s one file with no dependencies, and it never slows or breaks your app. ``` from wallras_sdk import Wallras w = Wallras("http://localhost:8080", token=WALLRAS_APP_TOKEN) # Hugging Face transformers out = model.generate(**inputs, max_new_tokens=256, output_scores=True, return_dict_in_generate=True) w.report_transformers(out, inputs["input_ids"].shape[1], tokenizer, model="llama-3.1-8b", prompt=prompt) # vLLM, llama.cpp, Ollama: anything returning OpenAI-style logprobs w.report(model="llama-3.1-8b", answer=text, tokens=Wallras.tokens_from_openai(logprobs_content)) # your own probe on hidden states, 0 to 1 w.report(model="llama-3.1-8b", answer=text, probe_score=0.31) ``` Results: `GET /wallras/inside-out`. API models that return logprobs show up there too. ## Tamper check and receipts Every record is chained to the one before it. `GET /wallras/integrity` checks them all. To prove nothing was rewritten even by someone with full server access, keep receipts elsewhere: `GET /wallras/integrity/receipt`, or set `WALLRAS_ANCHOR_URL` to a Slack webhook or your log system and Wallras sends one every hour. Check later with `POST /wallras/integrity/verify`. ## Data deletion requests ``` curl -X POST localhost:8080/wallras/redact -H "X-Wallras-Token: …" -H "Content-Type: application/json" \ -d '{"containing": "priya@example.com", "reason": "deletion request", "preview": true}' ``` Removal can’t be undone, so start with `"preview": true`, which only counts the matching records. Send it again without `preview` to remove them. The search matches your text exactly as typed, and a search that matches more than 200 records also needs `"confirm_many": true`. Content is removed, the tamper check keeps passing, and the deletion is logged. Available on every plan. ## Compliance reports `GET /wallras/report` returns a print-ready page (save as PDF), or add `?format=md` or `?format=json`. It covers drift, verification, accuracy, blind spots, token-probability certainty, the oversight log and a receipt. ## Access, audit and backups Give people and systems their own tokens with a role: **admin** (everything), **viewer** (read only) or **app** (feedback, SDK reports and agent steps only). Tokens must be at least 24 characters. ``` WALLRAS_TOKENS='{"alice": {"token": "…", "role": "admin"}, "grafana": {"token": "…", "role": "viewer"}}' ``` **Single sign-on.** Instead of shared tokens, people can use the access token your identity provider issues (Okta, Microsoft Entra ID, Google, Auth0, Keycloak or any OpenID Connect provider). Send it in `X-Wallras-Token`, never in `Authorization`, which some endpoints pass on to your model provider. Wallras checks the signature against your provider’s published keys, the issuer, the audience and the expiry, and takes the role from a claim: ``` WALLRAS_OIDC_ISSUER=https://yourcompany.okta.com/oauth2/default WALLRAS_OIDC_AUDIENCE=wallras WALLRAS_OIDC_ROLE_CLAIM=groups WALLRAS_OIDC_ROLE_MAP='{"wallras-admins": "admin", "wallras-readers": "viewer"}' ``` Someone in several groups gets the highest role. A valid sign-in with no mapped group is refused. For Keycloak use `WALLRAS_OIDC_ROLE_CLAIM=realm_access.roles`. The engine won’t start with an issuer but no audience. Every change made with an admin token or an SSO account is logged with its name (`GET /wallras/audit`). `POST /wallras/backup` writes a consistent copy of the database while Wallras keeps running, into `WALLRAS_BACKUP_DIR`, with a receipt to check a restore against. `GET /wallras/ready` tells load balancers and Kubernetes the database can be written; `/wallras/health` only says the process is up. ## The browser workspace Open `http://your-wallras:8080/wallras/ui` and sign in with a token (admin to edit, viewer to look, or a single sign-on token). It runs inside your Wallras, so nothing is hosted elsewhere, and the token stays in that browser tab only. - **Overview:** what needs attention, with the next step for each. - **Prompts:** edit a prompt and save it as a new version, compare any two versions line by line, release a version to production or staging through a regression gate (with test values for its `{{variables}}`), roll back, and see live results per version and who released what, when and why. - **Working together:** everyone signs in with their own token or SSO account, so every version and release is attributed. If someone saves a version while you are editing, your save is stopped, your draft is kept, and you see their change beside yours before saving yours as the next version. Nothing is overwritten. - **Review:** a labelling screen for the review queue. Read the question, the answer and the accepted or re-asked answer, then mark it correct or wrong, or accept it as the new expected answer. Keys: j and k to move, c, w and a to decide. - **Live quality:** judge scores per model and per prompt version, and the recent failures. The page loads only its own script, with a strict security policy (no inline code, no framing), and shows everything as plain text, so a prompt or answer can’t run code in your browser. Not included: two people typing into the same draft at the same moment, as in a shared document. ## Live judge scoring **Question:** are live answers actually good, not just stable? Set `WALLRAS_JUDGE_MODEL` and a judge model scores a sample of real answers against your rubric, after each answer has gone back to your app. Available on every plan; the judge runs on your provider, with your key. - `WALLRAS_JUDGE_RATE`: the share of answers scored (default 0.05, so about 5 judge calls per 100 requests). `WALLRAS_JUDGE_RUBRIC`: what a good answer is (default: correct, answers the question, follows the system prompt, safe). - `WALLRAS_JUDGE_UPSTREAM` and `WALLRAS_JUDGE_KEY` put the judge on a different OpenAI-compatible provider; otherwise it uses your main one and the key the request came with (memory only). - Each score is 1 to 5 with pass or fail and a short reason, kept with the request in the tamper-evident ledger. `GET /wallras/judge` shows the pass rate per model and per prompt version with its 95% range. Failures go to the review queue, so a person checks the judge too, and the overview flags a model the judge keeps failing. - Use a different, stronger model than the one being judged. The judge is told not to follow instructions inside the conversation it grades. Judges can still be wrong, so treat a score as a reason to look. - A judge on another provider (`WALLRAS_JUDGE_UPSTREAM`) must have its own `WALLRAS_JUDGE_KEY`: Wallras won’t start otherwise, so your requests’ keys never go there. ## Active-active: many containers, one Wallras Run several Wallras containers behind your load balancer, on different machines if you like, all on one Postgres database. Any container answers any request, and together they behave as one Wallras: one record, one hash chain, one admin token, one set of limits. Lose a container and the others carry on with every record. Available on every plan, and off unless you set a database address. ``` # the same on every container WALLRAS_DATABASE_URL=postgresql://wallras:…@db.internal:5432/wallras?sslmode=require WALLRAS_ADMIN_TOKEN=… WALLRAS_TRUSTED_PROXY_HOPS=1 # optional, per container: the name shown in the fleet view WALLRAS_INSTANCE_NAME=wallras-eu-1 ``` - **One chain, many writers:** before writing, every container takes the same database lock, so records join the hash chain strictly one at a time, in order. There are no forks to merge and no gaps, and `POST /wallras/integrity/verify` on any container checks the whole fleet’s history. Writes are recorded after each answer is sent, so waiting for the lock never delays an answer. - **Any container, any request:** feedback, replays, traces, the review queue, prompts and regression sets all live in the shared database, so they work from whichever container the load balancer picks. - **Shared across the fleet:** per-minute, token and spend limits, the wrong-token lockout, jobs, and background checks (snapshots, hidden tests, alerts, hub reports, each run by one container at a time). The response cache stays per container. - **The fleet view:** `GET /wallras/fleet` lists every container and worker writing to the database, its version, when it last checked in and what it served. The request log says which one answered each request (`served_by`). Different versions running at once are flagged: fine during a rolling upgrade, which works because new columns are added without stopping the others. - **Try it:** [docker-compose.active.yml](assets/docker-compose.active.yml) starts Postgres, three containers and a load balancer: `docker compose -f docker-compose.active.yml up -d`, then send requests to `localhost:8080/v1`. - **Backups:** with Postgres, use `pg_dump` or your provider’s backups; `POST /wallras/backup` is for the SQLite file. Keep receipts from `GET /wallras/integrity` to check a restore against. - **Hub or active-active?** Active-active is one Wallras spread over many containers, for availability and load. A [hub](#hub) joins separate Wallras installations (each app, team or region with its own database) into one view. They combine: an active-active fleet can report to a hub as one node. ## Hub and nodes: many containers, one view Run a Wallras next to each app, team or region, as usual, and make one more the hub. Each node keeps answering, recording and checking on its own; every minute it also sends the hub a summary. The hub shows every node in one overview, one combined report and one set of alerts. Available on every plan, and off unless you set it up. ``` # on the hub: one token per node, with the "node" role (node tokens can only send summaries) WALLRAS_TOKENS='{"eu-checkout": {"token": "…", "role": "node"}, "us-support": {"token": "…", "role": "node"}}' # on each node WALLRAS_HUB_URL=https://wallras-hub.internal:8443 WALLRAS_HUB_TOKEN=… WALLRAS_NODE_NAME="Checkout EU" ``` - **What a node sends:** drift status and flags per model and app, request counts, errors, latency, tokens and cost, live judge and consistency rates, the review backlog, its own next actions, its engine version and its ledger receipt. Never prompts, answers, traces or keys. `GET /wallras/hub/preview` on a node shows the exact payload. - **What the hub adds:** a model drifting on several nodes at once is flagged as a likely provider change rather than one app’s problem; a node that stops reporting is flagged after three missed reports; every node’s own problems appear in the hub’s overview and alerts. `GET /wallras/hub/overview`, `GET /wallras/hub/nodes`, `GET /wallras/hub/report`, and the Nodes tab in the workspace. - **An outside copy of every node’s history:** the hub keeps each ledger receipt a node sends. `GET /wallras/hub/receipts?node=…` gives them back to check with `POST /wallras/integrity/verify` on the node, and a node whose ledger goes backwards, or that comes back with a different database, is flagged at once. - **Network problems:** the hub is never in the request path, so a node works normally while the hub is down. A summary replaces the previous one, so a missed summary loses nothing the next one doesn’t carry; receipts that couldn’t be delivered are queued on the node (up to 500) and sent when the hub is back. Each summary carries a number that only goes up, so retries and late deliveries are ignored instead of looking like tampering, whatever the clocks say. - **Hash chains:** each node keeps its own. Chains are never merged across machines, so there is no cross-machine ordering to get wrong. - **Try it in 30 seconds:** [docker-compose.hub.yml](assets/docker-compose.hub.yml) starts a hub and two nodes on your machine: `docker compose -f docker-compose.hub.yml up -d`, send requests to `localhost:8081/v1` or `:8082`, then open the Nodes tab at `localhost:8080/wallras/ui`. Node tokens must be at least 24 characters; a shorter one is ignored, with a warning in the hub’s log. - One node is named by its token, not by what it sends, so a node can’t report as another. Use HTTPS between nodes and the hub (see [Running it reliably](#resilience)). ## Alerts Set `WALLRAS_ALERT_URL` to a Slack incoming webhook, or any webhook, and `WALLRAS_PAGERDUTY_KEY` to a PagerDuty Events v2 routing key. Every five minutes Wallras checks the same list as the overview: drift, silent model swaps, a model the judge keeps failing, failing hidden tests, broken regression sets and the review backlog. Each new problem is sent once, again if it’s still there a day later, and a resolved message follows when it clears. A blocked gated release alerts at once. Available on every plan. - Webhook bodies are Slack-compatible (`text`) and carry the structured alert too. With `WALLRAS_ALERT_SECRET`, each is signed: `X-Wallras-Signature: sha256=`. - PagerDuty gets high-severity alerts as incidents, and resolves them when the problem clears. - `WALLRAS_ALERT_MIN` sets the lowest severity sent (default `medium`), and `WALLRAS_ALERT_INTERVAL` the seconds between checks (default 300). `POST /wallras/alerts/test` sends a test alert; `POST /wallras/alerts/check` checks now. - Destinations come from settings only, never from a request. Every alert sent is in the audit log. ## For AI agents Assistants and agents can read this site and use Wallras directly. - **Reading the site:** [llms.txt](llms.txt) says what Wallras is and where everything is; [llms-full.txt](llms-full.txt) is every page as Markdown in one file, and each page has its own Markdown copy (this one is [docs.md](docs.md)). [openapi.json](openapi.json) describes every engine endpoint, generated from the engine’s code. All of them are rebuilt from the pages and the engine, and the site’s tests fail if any is out of date. - **Using Wallras: the MCP server.** [wallras_mcp.py](assets/wallras_mcp.py) lets an MCP client such as Claude Desktop, Claude Code or Cursor work with your Wallras: what needs attention, drift, agent runs, the request log, the review queue, prompts and their versions, regression sets and experiments. One Python file, no dependencies. It talks only to your Wallras with your token, so nothing leaves your machine. The MCP server and the Python SDK are open source (MIT) at [github.com/wallras/wallras-python](https://github.com/wallras/wallras-python), where you can open issues, and install with `pip install git+https://github.com/wallras/wallras-python` or run with `uvx --from git+https://github.com/wallras/wallras-python wallras-mcp`. - **Read-only by default.** The tools that change something (feedback, review decisions, saving and promoting prompts, regression runs that call your provider, optionally graded by a `judge_model`) appear only with `--enable-writes`, and still need an admin token. A viewer token stays read-only either way. Promotions keep their regression gate: a version that fails it isn’t released. ``` {"mcpServers": {"wallras": { "command": "uvx", "args": ["--from", "git+https://github.com/wallras/wallras-python", "wallras-mcp"], # or "python", ["/path/to/wallras_mcp.py"] "env": {"WALLRAS_URL": "http://localhost:8080", "WALLRAS_TOKEN": "a viewer token"}}}} # to let the agent run regression sets and promote prompts too (admin token, your provider key) "args": ["--from", "git+https://github.com/wallras/wallras-python", "wallras-mcp", "--enable-writes"], "env": {"WALLRAS_URL": "…", "WALLRAS_TOKEN": "…", "WALLRAS_PROVIDER_KEY": "…"} ``` Tools: `wallras_overview`, `wallras_drift`, `wallras_traces`, `wallras_requests`, `wallras_review_queue`, then `wallras_judge` (live judge scores), `wallras_fleet` (every container in an active-active setup), status, single requests and traces, usage, consistency, blind spots, regression sets, experiments, prompts, prompt diffs and per-version results. With writes on: `wallras_feedback`, `wallras_decide_review`, `wallras_run_dataset`, `wallras_save_prompt` and `wallras_promote_prompt`. Answers and prompts can hold customer data, and the agent sees whatever its token can see, so give it the narrowest token that does the job. ## Configuration | Variable | What it does | |---|---| | `WALLRAS_UPSTREAM` | Your OpenAI-compatible provider’s API address | | `WALLRAS_ANTHROPIC_KEY`, `WALLRAS_GEMINI_KEY` | Optional keys for hidden tests and watched regression sets on that traffic (otherwise the last key seen, kept in memory only) | | `WALLRAS_ANTHROPIC_UPSTREAM`, `WALLRAS_GEMINI_UPSTREAM` | Where `/anthropic` and `/gemini` requests go (defaults: Anthropic’s and Google’s public APIs). See [Anthropic and Gemini](#native) | | `WALLRAS_ADMIN_TOKEN` | Full access to /wallras. Generated and printed in the logs if unset | | `WALLRAS_APP_TOKEN` | Feedback and SDK reports only | | `WALLRAS_LICENSE` | Your license key | | `WALLRAS_CONSISTENCY_RATE` | Share of questions re-asked in other words to look for contradictions. Off by default (0): set 0.02 to re-ask 2%, about 2 to 4 extra calls per 100 requests on your key. Needs Growth or the Verification Pack | | `WALLRAS_LICENSE_SERVER` | Where paid plans renew and report usage (set by default; change only if told to) | | `WALLRAS_RETRIES`, `WALLRAS_FALLBACK_*`, `WALLRAS_CACHE_TTL`, `WALLRAS_LIMITS` | Gateway controls, all off by default (see [Gateway controls](#gateway)) | | `WALLRAS_TOKENS` | Named tokens with roles (admin, viewer, app, and node for a hub) | | `WALLRAS_OIDC_ISSUER`, `WALLRAS_OIDC_AUDIENCE` | Turn on single sign-on with your identity provider | | `WALLRAS_OIDC_ROLE_CLAIM`, `WALLRAS_OIDC_ROLE_MAP` | Which claim holds groups (default `roles`) and how they map to admin, viewer and app | | `WALLRAS_CAPTURE_LOGPROBS` | Set to 1 to record decision records for hosted models | | `WALLRAS_MAX_REQUEST_MB` | Largest request body accepted on any route, in MB (default 32). Larger ones get 413 before they’re read | | `WALLRAS_BACKUP_DIR` | Where backups are written (default: backups next to the database) | | `WALLRAS_WATCH_INTERVAL` | Seconds between checks of watched regression sets (default 600) | | `WALLRAS_PRICES` | Model prices for trace costs, e.g. `{"gpt-5-mini": [0.25, 2]}` (USD per million input and output tokens) | | `WALLRAS_ALERT_URL`, `WALLRAS_ALERT_SECRET`, `WALLRAS_PAGERDUTY_KEY`, `WALLRAS_ALERT_MIN` | Alerts to a webhook (Slack-compatible) and PagerDuty (see [Alerts](#alerts)) | | `WALLRAS_DATABASE_URL` | A Postgres database shared by several containers (see [Active-active](#active-active)). Unset: a SQLite file at `WALLRAS_DB` | | `WALLRAS_DATABASE_SCHEMA`, `WALLRAS_INSTANCE_NAME` | The Postgres schema to use (default `wallras`), and this container’s name in the fleet view | | `WALLRAS_HUB_URL`, `WALLRAS_HUB_TOKEN`, `WALLRAS_NODE_NAME`, `WALLRAS_HUB_INTERVAL` | Make this Wallras a node that reports to a hub (see [Hub and nodes](#hub)) | | `WALLRAS_TLS_CERT`, `WALLRAS_TLS_KEY`, `WALLRAS_PORT`, `WALLRAS_WORKERS` | HTTPS, the port, and worker processes (see [Running it reliably](#resilience)) | | `WALLRAS_JUDGE_MODEL`, `WALLRAS_JUDGE_RATE`, `WALLRAS_JUDGE_RUBRIC` | Live judge scoring of a sample of answers (see [Live judge scoring](#judge)); `WALLRAS_JUDGE_UPSTREAM` and `WALLRAS_JUDGE_KEY` for a separate judge provider | | `WALLRAS_EMBED_MODEL` | Compare answers by meaning instead of wording | | `WALLRAS_ANCHOR_URL` | Where hourly receipts go; `WALLRAS_ANCHOR_SECRET` signs them | | `WALLRAS_UPSTREAM_KEY` | Key for hidden tests (otherwise the last key seen, kept in memory only) | | `WALLRAS_BASELINE_SIZE` | Answers per model that define “normal” before drift checks start (default 200) | ## All endpoints All need the `X-Wallras-Token` header, except health. | `GET /wallras/status` | Mode, provider, models, silent swaps, plan | |---|---| | `GET /wallras/plan` | Plan, usage this month, recording state | | `POST /wallras/license` | Install a license key | | `GET /wallras/drift` | Drift radar, per model | | `POST /wallras/baseline/reset` | Accept current behaviour as normal: every model, one `model`, or one `app`’s use of a model | | `POST /wallras/swap-diff` | Model swap diff | | `POST /wallras/feedback` | Ground truth (app token works) | | `GET /wallras/confidence` | Confidence check | | `GET /wallras/events/{id}` | One answer: stated vs real confidence, internals | | `GET /wallras/blind-spots` | Failures by topic and task | | `GET /wallras/usage` | Requests, errors, latency, tokens and cost by model, app or day | | `GET /wallras/requests` | The request log: filter and search | | `GET /wallras/metrics` | Prometheus metrics (bearer token works) | | `POST /wallras/events/{event_id}/replay` | Send a recorded request again with changes | | `GET /wallras/traces` | Agent runs: steps, tools, timing, problems | | `GET /wallras/traces/{trace_id}` | One run, step by step | | `GET /wallras/traces/compare` | Two runs side by side | | `POST /wallras/traces/{trace_id}/steps` | Report a step your app ran (app token works) | | `POST /wallras/datasets` | Create a regression test set (GET lists them) | | `POST /wallras/datasets/{name}/items` | Add answers from a trace, ids or feedback | | `POST /wallras/datasets/{name}/run` | Run the set on a model, with a pass/fail verdict | | `POST /wallras/datasets/{name}/watch` | Re-run automatically on swaps, drift and a schedule | | `POST /wallras/datasets/{name}/compare` | Several models, system prompts or saved prompt versions side by side; `save_as` keeps it as an experiment | | `GET /wallras/experiments` | Saved experiments; `/wallras/experiments/{id}` for one | | `POST /wallras/prompts` | Save a new prompt version | | `GET /wallras/prompts/{name}` | Versions, labels and label history | | `GET /wallras/prompts/{name}/resolve` | The messages to send for a label or version, with variables (app tokens allowed) | | `GET /wallras/prompts/{name}/diff` | What changed between two versions | | `POST /wallras/prompts/{name}/promote` | Point a label at a version, optionally gated by a regression set | | `POST /wallras/prompts/{name}/rollback` | Put a label back on its previous version | | `GET /wallras/prompts/{name}/usage` | Live results per version | | `GET /wallras/judge` | Live judge scores per model and prompt version | | `GET /wallras/jobs/{id}` | A background job, e.g. a gated release started with `"wait": false` | | `POST /wallras/alerts/test` | Send a test alert | | `GET /wallras/fleet` | Every container and worker writing to this database: version, last check-in, requests served | | `GET /wallras/hub/overview` | On a hub: every node in one view | | `GET /wallras/hub/nodes` | On a hub: each node’s status and summary | | `GET /wallras/hub/report` | On a hub: the combined report | | `GET /wallras/hub/receipts` | On a hub: a node’s ledger receipts, to verify on the node | | `GET /wallras/hub/preview` | On a node: exactly what it sends to its hub | | `POST /wallras/hub/ship` | On a node: send the summary now | | `POST /wallras/alerts/check` | Check for alerts now | | `GET /wallras/me` | Which token or user this is, and its role | | `GET /wallras/review` | The review queue | | `POST /wallras/review/{id}` | Decide one item | | `POST /wallras/datasets/{name}/import` | Add your own examples (JSON Lines) | | `GET /wallras/datasets/{name}/export` | The set as JSON Lines | | `GET /wallras/overview` | Everything that needs attention, with next actions | | `GET /wallras/events/{event_id}/explain` | How the model reached this answer | | `GET /wallras/decisions` | Answers with decision records, shakiest first | | `POST /wallras/otlp/v1/traces` | OpenTelemetry trace export (app token works) | | `GET /wallras/audit` | Who did what | | `POST /wallras/backup` | Consistent backup while running | | `GET /wallras/ready` | Readiness check, no token | | `/wallras/canaries` | Hidden tests: add, list, remove, run | | `GET /wallras/consistency` | Consistency checks | | `POST /wallras/internals` | SDK reports (app token works) | | `GET /wallras/inside-out` | Stated vs token-probability certainty | | `/wallras/integrity` | Tamper check, receipts, verify | | `POST /wallras/redact` | Data deletion requests | | `GET /wallras/report` | Compliance report | | `GET /wallras/health` | Health check, no token | Features outside your plan answer `402` with what to add or upgrade. ## Running it reliably Wallras sits in the path of your AI calls, so plan for it the way you would for any proxy. Recording happens after each answer is sent, so slow analysis or a locked database never delays an answer. What has to keep running is the container itself. - **Restart automatically:** `docker run --restart unless-stopped`, or a Kubernetes Deployment with one replica and a persistent volume for `/data`; or several replicas on one Postgres database (see [Active-active](#active-active)). - **Health checks:** `GET /wallras/health` says the process is up (use it as the liveness probe). `GET /wallras/ready` also writes to the database and answers 503 if it can’t (use it as the readiness probe, or behind a load balancer). Neither needs a token. - **Your app falls back to the provider** if Wallras can’t be reached, so a Wallras outage never becomes an AI outage. Answers in that window aren’t recorded: ``` try: r = wallras_client.chat.completions.create(**req) except openai.APIConnectionError: # Wallras unreachable: go straight to the provider r = direct_client.chat.completions.create(**req) ``` - **Backups and a warm standby:** `POST /wallras/backup` writes a consistent copy while Wallras keeps running, with a receipt. Restore the latest copy on a standby machine and point traffic there if the main one is lost; the tamper check on the restored copy is checked against the receipt. - **Several apps or regions:** run one Wallras per app or region. Each learns its own models’ normal behaviour from its own database. - **Several worker processes on one machine:** `WALLRAS_WORKERS=4`. They share the database and the tamper-evident ledger safely. With more than one: - Per-app limits count from the database, so a limit holds for the whole engine, not once per worker. Requests a minute are counted exactly, per clock minute. - The wrong-token lockout is shared: 20 wrong tokens a minute from one address, across all workers. - Background work runs on one worker at a time (alerts, hidden tests, drift snapshots, watched regression sets, receipts), so nothing is sent or run twice. - Background jobs are in the database, so any worker can answer `GET /wallras/jobs/{id}`. - The response cache stays per worker, so the same answer may be fetched once by each worker before it’s cached everywhere. - **HTTPS:** mount a certificate and set `WALLRAS_TLS_CERT` and `WALLRAS_TLS_KEY` (PEM files). Wallras then serves HTTPS only, with TLS 1.2 or newer and forward-secret ciphers. Without them it serves plain HTTP, for use behind your own TLS proxy or service mesh. `WALLRAS_PORT` and `WALLRAS_HOST` set where it listens. ``` docker run -d -p 8443:8443 -v wallras-data:/data -v /etc/wallras/certs:/certs:ro \ -e WALLRAS_TLS_CERT=/certs/fullchain.pem -e WALLRAS_TLS_KEY=/certs/privkey.pem -e WALLRAS_PORT=8443 \ -e WALLRAS_UPSTREAM=… -e WALLRAS_ADMIN_TOKEN=… ghcr.io/wallras/wallras-engine:0.19 ``` - **Encryption at rest:** the database file isn’t encrypted by Wallras itself. Put the `/data` volume on encrypted storage: an encrypted cloud disk (AWS EBS, Google Persistent Disk and Azure Managed Disks encrypt by default), LUKS on Linux, or BitLocker. Backups go to `WALLRAS_BACKUP_DIR`, so keep that on encrypted storage too. Deleted content is removed from the database, and the fingerprints left behind can’t be used to confirm a guess of what it was. For no single point of failure, run several containers behind a load balancer on one Postgres database ([active-active](#active-active)): any of them answers, and losing one loses nothing. For many apps, teams or regions, run a Wallras for each and connect them to a [hub](#hub) for one view. With the default SQLite file, high availability means automatic restart, a fallback in your app, and a restored standby. ## What leaves your server On Free: nothing. On a trial or paid plan: your license key, a random id for the installation, and this month’s count of successful requests, once an hour. Like any connection over the internet, our server also sees the time of each check and your server’s IP address. Never prompts, answers or logs. Consistency checks call your own provider with your own key, about 2 to 4 extra calls per 100 requests, and each hidden test you add makes one call an hour. --- # How Wallras measures this | Wallras Source: methodology.html # How Wallras measures this Every number Wallras shows, how it is calculated, how much data it needs, and what it can’t tell you. No black boxes. ## What runs where Wallras is one container on your server, between your app and your AI provider. Prompts, answers and every record stay on your server. On a trial or paid plan, the only thing sent to Wallras is license metadata: your license key, a random installation id and a monthly count of successful requests. Like any connection over the internet, our server also sees the time of each check and your server’s IP address. Free installations never contact the license server. Your account, sign-in and payment details are handled separately by the website, Supabase and the payment provider; see the [privacy policy](privacy). ## Drift radar **Question:** is this model behaving like it used to? Measured separately for every model name, so a change in your traffic mix never looks like drift. - **Baseline:** the first 200 answers per model (`WALLRAS_BASELINE_SIZE`). When several apps use the same model, each app (by `X-Wallras-App`) gets its own baseline, because different apps ask different things: a new app starting on a model is never mistaken for that model drifting. Without `X-Wallras-App`, all traffic to a model counts as one app. Checks start after 30 more. Until then the status is “collecting” and shows how many requests are left. `POST /wallras/baseline/reset` accepts current behaviour as the new normal. - **Answer length and wording:** how long answers are and the words they use, compared with the model’s own normal, both in their spread and on average, so a sudden change and a slow stretch are both caught. - **Measured against the model’s own variation:** a change has to stand out from how much this model naturally varies, so models that are naturally varied aren’t flagged for being themselves. - **Refusals:** a rise in “I can’t help with that” answers, recognised in 13 languages. - **Served model:** a model version the provider has never served you before. - **Token certainty:** where the provider returns token probabilities, a drop in how sure the model was of its own words. - **Speed:** answers getting markedly slower. - **Three levels:** watch, medium and high, so you choose what deserves an alert. All of this is deterministic code: no AI model judges your AI. Tested on 10 Nov 2026 with NVIDIA’s Nemotron 3.5 Bolt: after the app’s instructions were changed, Wallras reported refusals up from 0% to 33% and flagged length, wording and speed. ## Drift accuracy benchmark How often does drift detection catch a real change, and how often does it cry wolf? `tools/drift_benchmark.py`, shipped in the engine image, simulates answer streams from four kinds of model (terse, chatty, highly variable, mixed tasks), each with natural randomness, then applies real drifts of different sizes, plus controls with no change at all. Every case runs 40 times per model with different random seeds: 1,280 cases in all, 200 baseline and 60 recent answers each. Wallras is compared with the rules generic monitoring uses, on exactly the same data. | Case (160 runs each) | Wallras | Length ±20% | Length t-test | Error rate | |---|---|---|---|---| | No change: false alarms | 1.9% | 3.1% | 1.3% | 0% | | Answers 30% longer | **41%** | 27% | 31% | 0% | | Answers 60% longer | 81% | **83%** | 69% | 0% | | Refusals up 8 points | **35%** | 4% | 4% | 0% | | Refusals up 15 points | **78%** | 6% | 12% | 0% | | New writing style | **99%** | 69% | 71% | 0% | | Subtle mix (15% longer, hedging) | **91%** | 70% | 67% | 0% | | Silent swap, new version name served | **100%** | 3% | 2% | 0% | | Silent swap, name hidden (a different model behind the same name) | **100%** | 100% | 100% | 0% | | **Average over all drifts** | **78%** | 45% | 44% | 0% | Wallras catches far more drifts than the best generic rule (78% vs 45%) while raising false alarms about as rarely (1.9% vs 1.3% to 3.1%), and it’s the only one that sees refusals. Two kinds of silent swap are tested. When the provider reports a new version name, Wallras catches it from that name alone, which generic rules miss. When the name stays the same, only the answers can give it away: here the swapped-in model writes very differently, so length rules catch it too. A swap to a model that writes almost the same way would be harder for every method; that is what the judge and regression sets are for. Error-rate monitoring sees nothing, because drifted answers are still successful responses. Limits: the data is simulated (see [real models](#real-models) below), small changes over only 60 answers are hard for any method (refusals up 8 points: 35%), and real traffic will differ. Run it yourself with `docker run --rm --entrypoint python ghcr.io/wallras/wallras-engine tools/drift_benchmark.py`; the numbers are deterministic. ### On real models The benchmark above uses simulated answers. To check it against real model output, we asked three models hosted on NVIDIA (Nemotron 3.5 Bolt, Ising Calibration 1.5 31B and Nemotron 3 Ultra) the same 30 customer-support questions three times each at temperature 0.7, plus once with a changed system prompt (“answer thoroughly”), on 11 November 2026. Nemotron 3 Ultra spent most of its token budget reasoning and returned text for only 18 of 120 questions, so it was left out. Each case was scored on 20 random draws of 60 baseline and 30 recent answers, with the served model name hidden so only the answers could reveal a swap. | Case (real answers) | Wallras | Length +20% | Length t-test | Error rate | |---|---|---|---|---| | No change (false alarms) | 10% (5% high severity) | 10% | 2.5% | 0% | | Silent swap to another real model | **100%** | 100% | 100% | 0% | | System prompt changed | **97.5%** | 100% | 50% | 0% | On real models, Wallras caught every silent swap and 97.5% of prompt changes. These real changes were large, so simple length rules caught them too; the simulated benchmark above is where Wallras’s wider coverage shows (refusals, subtle wording, swaps behind the same name). With nothing changed, most alerts were at the lower “watch” level. Error-rate monitoring alone saw none of these changes. The answers are saved, so anyone can re-score them: `docker run --rm --entrypoint python ghcr.io/wallras/wallras-engine tools/real_model_benchmark.py --offline`. This is real model output with staged changes, not production traffic. ## How it decided A decision record shows how the model arrived at one answer. What can be seen depends on where the model runs: - **Hosted models** (OpenAI, NVIDIA and others): the provider doesn’t expose the network. Wallras records each token’s probability and the top alternatives when the API returns them (`WALLRAS_CAPTURE_LOGPROBS` asks on your app’s behalf), and the reasoning text that reasoning models send. - **Self-hosted models** (Hugging Face transformers, through the SDK): Wallras reads the model’s internals directly, so probabilities aren’t distorted by sampling settings, and it can see how early in the network each word was settled and which parts of the prompt the model relied on. The record highlights **forks** (close calls between two words), **unlikely picks**, **guessed phrases**, words **decided late and unsure**, what the model **leaned on**, numbers in the answer that the **reasoning never reached**, and reasoning that **used up the token budget** before answering. **Built to be trusted:** probabilities are exact; layer and attention signals show where the model looked and how settled it was; and reasoning text is checked against the answer rather than taken at face value. Tested on 11 Nov 2026 with Pythia-14M and tiny-GPT2 (GPT-NeoX and GPT-2 architectures) and NVIDIA Nemotron 3.5 Bolt, 3.5 Lightning and 3 Ultra. ## Confidence check **Question:** when the AI sounds sure, is it right? It needs your feedback (`POST /wallras/feedback`) on which answers were correct. - **Stated confidence** is read from the answer: explicit percentages first, then wording such as “might”, “I think” or “definitely”. Wallras weighs it against the model’s real track record from your feedback, and prefers token probabilities and real outcomes where they exist. Politeness like “I’m sure you’ll love it” doesn’t count. - For each level of confidence: how often those answers were actually right, with a range that shows how sure that number is. - The AI is called over- or under-confident only once there’s enough feedback to say so reliably. - Where the provider exposes token probabilities, the same check runs on those, which are harder to fake than wording. ## Blind spot map **Question:** where shouldn’t we trust it? Answers are grouped by the `X-Wallras-Topic` header you send or, without it, by topics Wallras finds itself, on your server with no outside services. A topic is flagged as a blind spot only once there’s enough data to be sure, when: - accuracy is clearly below the rest, shown with its range; - refusals are clearly above the rest; - consistency checks often contradict each other; or - hidden tests often fail. ## Agent traces **Question:** where did the agent go wrong? Built only from records already on your server. - **Grouping:** requests with the same `X-Wallras-Trace` header are one run. Without it, Wallras joins the steps of each run automatically by following the conversation, keeping only fingerprints of the messages, and two conversations that start with the same prompt stay separate. If your app rewrites earlier messages between steps, send `X-Wallras-Trace` instead. One-off chats aren’t shown as runs. - **Per step:** model asked for and model served, status, time taken, input and output tokens, tool calls with arguments, and the tool results sent back. - **Flagged:** repeated identical tool calls (a likely loop), failed steps, tool results reporting an error, unusually slow steps, unusually long runs, runs that end on a tool request, and answers your feedback marked wrong. - **Cost** is shown only when you set `WALLRAS_PRICES`, from the provider’s own token counts. - **Steps outside Wallras:** traces show every model call through Wallras. Tools your app runs, database lookups and API calls appear when your app reports them (`POST /wallras/traces/{id}/steps`, or one line with the SDK): name, kind, timing and outcome, never inputs or outputs. They’re shown in order with the model calls, and failed ones are flagged. ## Regression tests **Question:** did our change break anything that used to work? Wallras uses the right tool for each kind of answer: rules wherever an answer can be checked by rule, similarity to screen, people for the close calls, and an AI judge where quality is a matter of judgement. Similar wording doesn’t make an answer true and different wording doesn’t make it wrong, so no single score decides alone. | Method | What it checks | In Wallras | |---|---|---| | Exact-match tests | Deterministic answers stay the same | `exact` | | Structured-output validation | JSON, schemas and required fields | `json` (required keys), `schema` (JSON Schema: types, required, enums, ranges, patterns, no extra fields) | | Rule-based assertions | Explicit business rules | `rules`: must and must-not mention, patterns, how it starts, word and length limits; `contains`, `regex` | | Ground-truth comparisons | Factual answers match verified reference data | The accepted answer per item, with its numbers, names, dates, yes/no and negations checked by rule; hidden tests with known answers | | Tool-call validation | The right tool and arguments | `tool`, with required arguments or exact values | | Numerical tolerances | Calculations within an acceptable error | `number`: exact, ± an amount, or ± a percentage | | Citation validation | References exist and support the claims | `citations`: every [1], [doc-id] or link must be a source the model was given, and each cited sentence must be supported by that source; `grounded` for answers without references | | Metamorphic testing | Equivalent inputs give consistent answers | Consistency checks: real questions re-asked in other words, and asked again as they were, with contradictions reported | | Statistical regression | Refusal rate, latency, length and other measures change significantly | Drift detection per model: length, wording, refusals, latency, certainty and the served model, each against its own natural variation | - Each item is a request with the answer you accept: usually a real one that went through Wallras, or your own examples imported as JSON Lines. Real items come from a trace, chosen answers, or answers your feedback marked correct (and never marked wrong). - **Default check (similar):** the new answer must be close to the accepted one and keep its facts: the same numbers, names and dates, the same yes or no, no negation flipped (“you can” to “you cannot”) and no new refusal (or the reverse). Closeness is by **meaning** when `WALLRAS_EMBED_MODEL` is set, otherwise by wording in any word order. Tool-call steps compare the tool and arguments. - **Close calls go to a person:** similarity screens answers; it doesn’t prove them. When no rule decided and the score is close to the bar either way, the run lists the item under `needs_review` with both answers, so someone checks it instead of trusting the number. - **Calibrated:** with `calibrate`, the original model is asked each question again, and the bar for that item is set from how close it gets to its own accepted answer. Questions with many correct phrasings stop failing on wording alone. Costs one extra call per item. - **Judge:** for quality that similarity can’t capture, a separate judge model grades each answer against your written rubric, with temperature 0, and returns PASS or FAIL with a reason. Use a different, stronger model than the one being tested; a run with judge checks refuses to start without one. Judges can be wrong too, so check a sample of their reasons. - **With a judge model:** pass `judge_model` to a run and a separate model decides whether each answer says the same thing as the accepted one (similar checks) or whether every claim is supported by the context (grounded checks), so correct paraphrases pass and fluent answers with a different fact fail. Changed numbers and refusals are still caught by rule. Without a judge or an embedding model, runs compare wording and say so in a warning. - **JSON:** for structured outputs, the answer must be valid JSON with the keys you list. Exact, contains, regex and number checks are also available. - **Each run** is stored with which items passed (not the answers), so the next run reports what newly broke and what was fixed. The pass rate comes with a 95% range. - **Comparisons:** the same set run on several models or system prompts at once, each with its pass rate and 95% range and the items they disagree on. Overlapping ranges mean the difference may be chance. Comparisons aren’t added to the set’s run history. - **CI gate:** with `fail_below`, the verdict is fail if the pass rate is below it or anything newly broke. - Runs call your provider once per item, on your key, and aren’t billed by Wallras. ## Hidden tests Questions you write, with known answers, checked as `exact`, `contains`, `regex` or `number`. They run roughly hourly at random times (±50%) with the same system prompt and settings as your recent real traffic, so they test the AI as your users get it. They go straight to your provider with your key, aren’t stored as customer traffic, and aren’t billed. Only the questions you add are tested. ## Consistency checks When you turn it on, a small share of real questions (`WALLRAS_CONSISTENCY_RATE`, off by default; 0.02 is typical) is asked again in different words, after the original answer has been sent. If the answers disagree, it’s asked once more to separate real contradictions from randomness. Changed numbers, or a refusal on one side, count as contradictions. ## Model swap diff **Question:** what breaks if we switch? Real past prompts (you choose how many) are replayed on the candidate. The current model is also re-asked a sample of them (12 by default) to measure its own natural randomness, and the candidate is judged only on differences beyond that noise: - similarity of each answer pair, by wording, or by meaning if `WALLRAS_EMBED_MODEL` is set; - the share of answers that differ more than the current model differs from itself; - refusal flips beyond the current model’s own flip rate, and the answer length ratio. Results include the worst examples side by side and a confidence level, which is low for small samples. Replays cost provider calls on your key, and the key is only ever sent to your own provider. ## Tamper evidence Wallras records are **tamper-evident**: any change to past records shows. - **Detected:** every record (answers, feedback, deletions, the oversight log) carries a hash of the one before it. Editing, inserting or deleting any record breaks the chain, and `GET /wallras/integrity` shows where. - **Receipts, for the strongest proof:** someone with full server access could try to rebuild the whole history, and receipts catch that: a receipt is the latest record number and its fingerprint. Keep it where the server can’t change it (a Slack channel, your log system, a report you sent). A rebuilt history won’t match an older receipt (`POST /wallras/integrity/verify`). - **Automatic receipts:** set `WALLRAS_ANCHOR_URL` and Wallras sends one every hour (paid feature), signed with `WALLRAS_ANCHOR_SECRET` if you set one. - **Deletion requests** remove the content of matching records but keep their place in the chain, so the check still passes, and the deletion itself is logged. ## Speed and failure Measured with `tools/bench.py`, shipped in the engine image, which anyone can rerun (`docker run --rm --entrypoint python ghcr.io/wallras/wallras-engine tools/bench.py`). Setup: a 4-core Windows laptop, Python 3.14, the test suite’s fake provider on the same machine, 400 requests each, 10 Nov 2026. | | Direct | Through Wallras | Added | |---|---|---|---| | One at a time, median | 4.6 ms | 10.0 ms | +5.4 ms | | One at a time, 99th percentile | 6.2 ms | 13.5 ms | +7.3 ms | | Streaming, first chunk, median | 4.4 ms | 11.7 ms | +7.3 ms | Expect about 5 to 7 ms per request on modest hardware, next to AI answers that usually take 0.5 to 5 seconds. With 10 requests at once, the median added about 20 ms, because the provider, Wallras and the test shared the same 4 cores; on its own server it’s lower. - **Recording never blocks answering:** with Wallras’s database locked by another process, 20 of 20 requests were still answered. - **Provider down:** Wallras returned a clear 502 error in about 3.5 seconds instead of hanging. - **Always available:** run several containers on Postgres ([active-active](docs#active-active)) so losing one loses nothing, or use automatic restarts with a fallback to your provider (see [Running it reliably](docs#resilience)). ## Tested providers Wallras speaks the OpenAI chat completions API, Anthropic’s Messages API and Google’s Gemini API. “Tested with real traffic” means we sent real requests to the provider through Wallras. “Tested against a test server” means it runs on every release against test servers we built from the provider’s published formats; it hasn’t yet been run against that provider with live keys. The rest are expected to work, but we haven’t verified them ourselves. Each real-traffic row names the engine version it was run on; every release since runs the full test suite, including the test servers. | Provider | Status | What was checked | |---|---|---| | NVIDIA NIM: Nemotron 3.5 Bolt, 3.5 Lightning, 3 Ultra, Ising Calibration 1.5 | Tested with real traffic, 10 to 12 Nov 2026 (engines 0.14 and 0.16) | As a cross-format backup (engine 0.16, 7 of 7): Anthropic requests, streamed and with a tool call, and Gemini requests, plain and streamed, answered by Nemotron 3.5 Bolt in their own formats while the main providers were down; and live judge scoring with Nemotron as the judge. Earlier: Chat, streaming with multi-byte characters, tool calls, token probabilities, reasoning text, provider errors passed through, drift on real answers, swap diff, hidden tests, consistency checks | | OpenRouter, with OpenAI gpt-4o-mini and Anthropic Claude Haiku 5.5 | Tested with real traffic, 12 Nov 2026 (engine 0.18), 19 of 19 checks | Through `/v1`, both models: answers, streaming, tool calls, and token probabilities from the OpenAI model. Through `/anthropic`, Claude in Anthropic’s own Messages format: answers, streaming, a tool call and its result joined into one agent run, a replay and a regression run sent back in that format. The same test set compared on Claude and GPT, and GPT as the live judge scoring real answers. The key never stored | | OpenRouter, 28 models from 24 companies: OpenAI, Anthropic, Google, Meta, Mistral, DeepSeek, Qwen, Amazon, Cohere, IBM, NVIDIA, xAI, Liquid, Z.ai, MiniMax, Moonshot, Tencent, ByteDance, Xiaomi, StepFun, Upstage, Reka, Microsoft, Perplexity | Tested with real traffic, 12 Nov 2026 (engine 0.18): 25 of 28 models passed every check; the other 3 couldn’t be fully tested, for reasons on the provider’s side | Per model through `/v1`: an answer, a streamed answer and a tool call, each compared with what Wallras recorded; Claude models also in Anthropic’s own Messages format through `/anthropic`. Not fully tested: Microsoft Phi-4 and Perplexity Sonar don’t support tool calls (their answers and streams were recorded correctly), and AI21 Jamba Mini 1.7 is no longer served on OpenRouter. Results in `tools/real_models_matrix_results.json`, script in `tests/real_models_matrix.py` | | Anthropic Messages API | Tested with real Claude answers through OpenRouter’s Messages-compatible address (12 Nov 2026), with llama.cpp’s (11 Nov 2026), and against a test server on every release | Answers and streaming (split mid-character), tool use joined into one agent run, thinking, cache tokens, 529 “overloaded” retried, errors passed through, the key reaching only the provider and never stored | | Google Gemini API: Gemini 3.5 Flash, Gemini Flash Lite | Tested with real traffic, 11 Nov 2026 (engine 0.14): 15 of 15 checks on 3.5 Flash; on Flash Lite with engine 0.15, 21 of 21, including a replay and a regression run sent back to Gemini, and a prompt version released through a regression gate that ran on Gemini, then filed with the live requests that used it; and on every release against a test server | generateContent, streaming with and without `alt=sse` (Hindi text intact), a function call and its result joined into one agent run, a key in the header or the URL never stored, Google’s “high demand” errors retried, and Gemini’s OpenAI-compatible address through `/v1`. Token probabilities and Vertex AI paths are checked against the test server only | | OpenAI-style test provider | Tested on every release | Gzip, streaming with split multi-byte characters, tool calls, token probabilities, rate limits, server errors, embeddings, silent model swaps | | OpenTelemetry Python SDK with the OTLP/HTTP exporter | Tested on every release | Protobuf and JSON export, GenAI, OpenInference and OpenLLMetry span types, traceparent joining with proxied calls, prompts never stored | | LangChain with OpenInference | Tested, 11 Nov 2026 | A tool-using agent on NVIDIA Nemotron: 3 model calls and 2 tool calls in one trace, model calls joined without traceparent, nothing counted twice | | OpenAI Agents SDK with OpenInference | Tested, 11 Nov 2026 | The same agent: agent, turn and tool steps from the SDK joined with the 3 proxied model calls, no spans sent to OpenAI | | LlamaIndex, CrewAI, Haystack | Compatible through OpenTelemetry | Through their OpenTelemetry, OpenLLMetry or OpenInference instrumentation | | OpenAI directly, Azure OpenAI (its `/openai/v1` address), AWS Bedrock’s OpenAI-compatible address, Groq, Together, DeepSeek, Fireworks, xAI | Compatible | Same OpenAI-compatible API as the providers above | | llama.cpp (llama-server), with Qwen 2.5 0.5B Instruct | Tested with real traffic, 11 Nov 2026 (engine 0.14), 14 of 14 checks | Through both `/v1` and `/anthropic`: answers, streaming, token probabilities, tool calls, a tool result joined into one agent run, a replay and a regression run sent back in Anthropic’s format, the key never stored | | Ollama, vLLM, LM Studio | Compatible | Same OpenAI-compatible API as the providers above | Not supported in their own formats: other native APIs, such as Cohere’s, or Bedrock’s Converse API. Use them through an OpenAI-compatible address or gateway. Anthropic and Gemini traffic gets every feature, including those that ask the model again (replays, regression runs, swap diff, hidden tests, consistency checks): each request goes back in its own format to its own provider. ## What you’re billed for - **One request** is one successful model call through Wallras: `/v1/chat/completions`, `/anthropic/v1/messages`, or a Gemini `generateContent` or `streamGenerateContent`. A streamed answer counts once. An agent that makes five model calls for one task makes five requests. - **Not counted:** provider errors and rate limits, failed retries, answers served from the response cache, hidden tests, consistency re-asks, swap-diff replays, step replays, regression test runs and comparisons, and SDK reports about answers that already went through Wallras. - **SDK reports for models running inside your app** (not through Wallras) count as one request each. - **Going over:** extra requests cost your plan’s per-request rate, billed after the month ends, and a month never costs more than the next plan up. On Free, recording pauses at 10,000 requests; your AI keeps working. - **Yearly billing:** pay for 11 months, get 12. - **After cancelling:** the plan runs to the end of the paid period, then Wallras steps down to Free. Your records stay on your server. - **Trial:** one 7-day trial per email inbox, no card. ## Extra provider calls Wallras bills only your own requests, but two features call your provider on your key, and your provider charges for those: - **Consistency checks:** off until you set a rate. At 2% of questions re-asked, plus a follow-up when answers disagree: about 2 to 4 extra calls per 100 requests. - **Hidden tests:** one call per test you add, about once an hour each. - **Retries** (only if you set `WALLRAS_RETRIES`): a failed call is tried again, up to the number you set. A response cache, if you switch one on, does the opposite and saves calls. **Example:** with consistency checks at 2%, 100,000 requests a month adds roughly 2,000 to 4,000 provider calls, plus about 720 a month per hidden test. **By default neither runs**, trials included, so Wallras makes no calls of its own until you set `WALLRAS_CONSISTENCY_RATE` or add a hidden test. The overview suggests turning consistency checks on when your plan includes them. Swap diff runs only when you ask. ## Where Wallras fits Wallras covers monitoring, agent debugging, a gateway, evaluation and prompt releases in one container on your own servers. Specialist tools in each category go deeper in places. So you can decide, here is what’s included and what isn’t: | Area | Included | Not included (yet) | |---|---|---| | Prompts | Versions in a tamper-evident log, production and staging labels, releases gated by a regression set, rollback, diffs, live results per version, SDK fetch with fallback, and a browser workspace where a team edits and releases them, with every change attributed and simultaneous saves caught instead of overwritten | Two people typing into one draft at the same moment, as in a shared document | | Experiments and review | Regression sets from real traffic or your own examples, side-by-side comparisons saved as experiments, rule-based checks, judges, a human review queue with a labelling screen in the workspace, and live judge scoring of a sample of answers | Judging every single live answer (it’s a sample you set, to keep provider cost in check); labelling by an outside workforce | | Gateway | Retries, fallback, cache and per-app limits on OpenAI-compatible, Anthropic and Gemini traffic, including a fallback that answers Anthropic or Gemini requests with an OpenAI-compatible model in the app’s format | Token-by-token streaming during a cross-format fallback (the backup’s answer arrives whole, as a stream) | | Integrations | See [tested providers](#compat): each is marked tested with real traffic, tested against a test server, or expected to work | Real-traffic tests of the “expected to work” list | | Running it | Active-active: several containers behind a load balancer on one Postgres database, writing one hash chain, with shared limits and a fleet view; one Wallras per app, team or region, connected to a hub for one view, one report and one set of alerts; within each, automatic restart, several worker processes, built-in HTTPS, health and readiness checks, Prometheus metrics, alerts to Slack, PagerDuty or any webhook, live backups and a standby | Writes from every container go through one database lock, so write throughput is that of one Postgres writer (ample for monitoring, not unlimited); built-in encryption of the database file | ## Getting the most from Wallras - **Accuracy comes from evidence.** Drift shows when behaviour changes; your feedback, hidden tests and regression sets prove correctness for the questions that matter to you. Rule checks catch changed numbers, names, dates, yes/no answers, negations and refusals; add meaning comparison or a judge model for open-ended quality, and close calls are listed for a person to check. - **Scale as you grow.** The default SQLite file serves one container with several workers. For several containers and no single point of failure, use Postgres ([active-active](docs#active-active)): every container writes to one chain, in order. - **Complete agent traces.** Every model call through Wallras is traced; add your app’s own steps (tools, lookups, API calls) directly or through OpenTelemetry for the full picture. - **Decision records** show how sure the model was and where it looked; together with your feedback they show where to trust it. - **Confidence sharpens with feedback.** Every number comes with a range, so you always see how sure it is. - **Languages:** refusals are recognised in 13 languages and hedging in 6, and every language is measured for length, wording and speed. - **Audit-ready evidence.** Reports document monitoring and human oversight for your reviews and auditors; certification itself is your auditor’s call. --- # Security | Wallras Source: security.html # Security What protects your data in Wallras today, what every release is checked for, and what hasn’t been done yet. Facts only, no badges. ## Where we stand Below is our security assessment: standard tools and attack tests, with every finding fixed, and the same checks on every release. ## Security testing, 11 November 2026 An internal security assessment of the engine, the license server and this website, using standard open tools and attack tests run against working copies of both servers. Every finding was fixed and re-tested; the checks below now pass, and they run again on every release. | Check | Tool | Result | |---|---|---| | Code scan for the OWASP Top 10, Python, JavaScript and secrets rules (332 rules) | Semgrep | 0 findings | | Python security scan | bandit | 0 high severity; every medium finding reviewed (all were queries with bound parameters) | | Known vulnerabilities in every dependency of both servers | pip-audit | 0 | | Container image | Trivy | 0 critical with a fix available | | Keys, tokens and passwords in the code and in the entire git history | detect-secrets, gitleaks, pattern scan of every commit | 0 secrets ever committed (the only key in the code is the public sign-in key browsers need, checked to be the public one) | | Every engine endpoint without a token, with a wrong, truncated or read-only token, and with the app token | Attack test suite | All refused; read-only and app tokens can only do what their role allows | | Token guessing | Attack test suite | Address locked out after 20 wrong tokens a minute, others unaffected | | One customer reading, paying or changing another’s bills and licenses; every admin action without the admin token | Attack test suite | All refused | | SQL injection, script injection and malformed input in every field a stranger can reach | Attack test suite | Stored and returned as plain data; no server errors | | Forged, replayed and stale payment webhooks | Attack test suite | All refused | | Requests from other websites using a visitor’s session | Attack test suite | Refused (only this site is allowed) | | Oversized and parser-crashing requests | Attack test suite | Refused before parsing; servers stay up | | Disaster recovery: restore a live backup on a new engine | Recovery test | Same history, tamper check intact, keeps recording | | Website security headers | Mozilla HTTP Observatory | A+ (score 130), 12 of 12 tests passed | **Found and fixed in this assessment:** a request nested thousands of levels deep could crash the license server or stop the engine’s feedback endpoint (now refused before parsing); the license server’s size limit could be bypassed by sending a body in chunks (now refused); the website allowed inline scripts in its security policy (all scripts moved to files, inline scripts now forbidden); payment redirects now only follow secure addresses on the payment provider’s domain; the SDK now refuses non-web addresses; one endpoint checked a request’s contents before its token (now the token first). This is our own testing, and it’s repeatable: the attack tests are part of the automated suite every release must pass. ## Your data - **Runs on your infrastructure.** Prompts, answers, traces, test sets and every record are stored in a local database on your server. Wallras the company never receives them. See the [data-flow diagram](methodology#dataflow). - **What leaves:** on a trial or paid plan, your license key, a random installation id and a monthly count of successful requests, about once an hour. Like any connection over the internet, our server also sees the time of each check and your server’s IP address. On Free, nothing. - **Encryption in transit:** calls from Wallras to your AI provider use the provider’s HTTPS. Wallras serves HTTPS itself when given a certificate (`WALLRAS_TLS_CERT`, `WALLRAS_TLS_KEY`; TLS 1.2 or newer, forward-secret ciphers), or plain HTTP behind your own TLS proxy or service mesh. - **Encryption at rest:** Wallras doesn’t encrypt its database file itself. Put it on encrypted storage (most cloud disks are by default; LUKS or BitLocker otherwise). [How](docs#resilience). - **Deleted means deleted:** a deletion request removes the content and the random value mixed into its fingerprint, so the fingerprint that keeps the ledger checkable can’t be used to confirm a guess of what was said. - **Active-active:** containers sharing a Postgres database write one hash chain, in order, under one database lock, so the tamper check covers the whole fleet. Use TLS to the database (`sslmode=require`) and a database user that only Wallras uses: anyone who can write to the database directly can attempt what anyone with the SQLite file could, and is caught the same way, by your external receipts. - **Hub and nodes:** a node sends its hub statuses, counts and ledger receipts only, never prompts or answers, using a token that can do nothing else. The hub keeps every receipt, so each node’s history is checked against a copy held on another machine. - **Request limits:** bodies over 32 MB are refused on every route before they’re read (`WALLRAS_MAX_REQUEST_MB`), and keys passed in a URL, such as Gemini’s `?key=`, are removed from logs. - **Keys stay with their provider:** fallbacks and a judge on another provider must have their own keys, or Wallras won’t start. Text graded by a judge is fenced off as material, and the judge is told to ignore instructions inside it. - **Never in the way:** recording and analysis happen after each answer is sent. If storage fails, answers still go through ([tested](methodology#performance)). ## Access control - **Named tokens with roles:** admin (everything), viewer (read only) and app (can only send feedback, SDK reports, agent steps and OpenTelemetry traces, and can’t read anything). Tokens shorter than 24 characters are refused. - **Single sign-on** with any OpenID Connect provider (Okta, Entra ID, Google, Auth0, Keycloak). Signatures are checked against the provider’s published keys; only public-key algorithms are accepted (no unsigned or shared-secret tokens); issuer, audience and expiry are checked; groups map to roles; every change is logged under the person’s name. - **Audit trail:** every change made with an admin token or an SSO account is logged in the tamper-evident ledger with the token’s or person’s name, what was done and whether it worked. - If you don’t set an admin token, Wallras generates a random one at start-up and prints it in the logs. There’s no default password. - Tokens are compared in constant time, so they can’t be guessed by timing. After 20 wrong tokens in a minute, that address is refused for the rest of the minute. - No trust based on IP address: behind a reverse proxy every caller can look local, so Wallras never relies on it. - Proxied traffic uses your provider key exactly as your app sends it; Wallras doesn’t add its own authentication to the AI endpoint. ## Keys and secrets - **Your provider key** passes through to your provider and is never written to disk. The most recent one is kept in memory only, so hidden tests can run; set `WALLRAS_UPSTREAM_KEY` to use a dedicated key instead. - **Model swap diff** sends your current key only to your own provider, never to a different address. - **A fallback provider** (optional) gets its own key. Your provider key and every `X-Wallras-*` header are removed first, and Wallras refuses to start with a fallback address but no key of its own. - **The response cache** (optional, off by default) is held in memory only, separately for each provider key, and emptied by every deletion request. - **Licenses** are signed with a private key that exists only on the Wallras license server. The engine ships with the public key, which can check a license but can’t create one; edited or forged licenses are refused. - The `X-Wallras-*` headers and admin token are stripped before requests reach your provider. ## Records and deletion - Every record is **tamper-evident**: chained to the one before it, so editing, adding or deleting any record shows up in `GET /wallras/integrity`. Keep receipts off the server to catch a full rewrite; the [threat model](methodology#integrity) explains what this does and doesn’t prove. - **Deletion requests** remove matching content from answers, consistency checks, decision records and regression test sets, empty the response cache, keep the chain intact, and log the deletion. - Agent steps your app reports carry only a name, kind, timing, outcome and error type, never inputs or outputs. Regression runs store which items passed, not the answers. ## Every release An engine image is only published if all of these pass: - The full automated test suite: about 240 engine tests, most of them end to end against a running engine, including attacks on the tamper evidence, licenses and sign-in tokens, plus about 65 for the license server and website. - **Dependency audit** (`pip-audit`) against known vulnerabilities. - **Static security scans** (`Semgrep` with OWASP Top 10, Python, JavaScript and secrets rules, and `bandit`): any Semgrep finding or high-severity bandit finding fails the build. - **Secrets scan** (`gitleaks`) of the whole git history. - **Attack tests** against running copies of both servers (above), on SQLite and on Postgres. - **Container scan** (`trivy`): any critical vulnerability with a fix available fails the build; high ones are reported. The container runs as an unprivileged user on a minimal Python base image. Dependencies are pinned to exact versions. Every image carries its version and source, and versions are kept so you can pin or roll back. ## Website and account - The website loads no third-party scripts, fonts or trackers. It sends a strict Content Security Policy that forbids inline scripts, HSTS, no-framing, no-sniffing and cross-origin isolation headers. There are no cookies; sign-in sessions stay in the browser. - Sign-in is handled by Supabase. Payments are handled by the payment provider; Wallras never sees card details. - The license server stores your account email, plan, license records and usage counts. See the [privacy policy](privacy). ## Deployment notes - **High availability:** run several containers on Postgres ([active-active](docs#active-active)), or one container with automatic restarts, a readiness check (`/wallras/ready`), live backups and a fallback to your provider (see [Running it reliably](docs#resilience)). - **Encryption at rest:** keep Wallras’s data on encrypted storage, as most platforms do by default ([how](docs#resilience)). - **Deletion requests** clear the live database immediately. Backups you made earlier follow your own rotation, and compacting the database (VACUUM) clears freed space. ## Report a vulnerability Email [security@wallras.com](mailto:security@wallras.com) with what you found and how to reproduce it. We aim to acknowledge within 3 working days and to keep you updated until it’s fixed. Please give us a reasonable time to fix it before telling others. We won’t take legal action against good-faith research that avoids privacy violations, data destruction and service disruption. --- # Live demo | Wallras Source: demo.html # Watch Wallras catch ten problems AI teams really hit These are staged, like a crash test. The AI is a test model we built to misbehave in ten ways, and the questions and feedback are simulated. What you see from Wallras is real: the actual engine’s output, unedited. Loading results… ## Try it yourself Choose what goes wrong and see what Wallras reports. Every combination was run through the real engine ahead of time, so nothing runs in your browser and no engine or code is on this page. ### 1. Change how the AI behaves A support AI answers 100 questions normally, then 100 more after your change. ### 2. Test a new model before switching Three real answers became a test set. Pick a candidate model and how answers are compared. ## Now try it on your own AI The Free plan runs in about ten minutes, on your servers, with no account. Or try any paid plan free for 7 days, no card needed. [Start a free 7-day trial](checkout?tier=growth)[Set up the Free plan](docs#start) ``` docker run -d -p 8080:8080 \ -e WALLRAS_UPSTREAM=https://api.openai.com/v1 \ ghcr.io/wallras/wallras-engine:0.19 # then point your AI client at it base_url="http://localhost:8080/v1" ``` Want to see what reaches your auditors? [Open a sample compliance report](sample-report), made by the same engine from staged traffic. --- # Privacy policy | Wallras Source: privacy.html # Privacy policy Effective 13 November 2026 This policy explains what personal data Wallras (“we”, “us”) collects through wallras.com, your account and the Wallras software, why we collect it, who we share it with, and the rights you have. Wallras is operated from India. For the personal data described here, we are the data fiduciary (under India’s Digital Personal Data Protection Act, 2023) and the data controller (under the GDPR, where it applies). ## 1. The short version - **Your AI data never reaches us.** The Wallras software runs on your own servers. Your prompts, answers, traces and logs stay there, under your control. - **Licensed installations send us three things:** the license key, a random installation id and a monthly count of successful requests, so we can renew your license and bill correctly. On the Free plan the software sends us nothing. - **For your account** we keep your email address, how you signed in, and your plan and billing history. - **No advertising, no tracking cookies, no data sales.** We never sell or rent personal data, and we don’t use it to train AI models. ## 2. What we collect - **Account data:** your email address; your name and profile picture if your sign-in provider (Google, GitHub or Microsoft) shares them; and an account id from that provider. If you sign in with email, we never see your password in plain text: it’s stored only as a secure hash by our sign-in provider. - **License and usage data:** license key, installation id, monthly count of successful requests, the time of each license check and the IP address it came from. - **Billing data:** your plan, invoices, billing name, country and tax details. Card, UPI and other payments are processed by our payment provider, Dodo Payments, as merchant of record; we receive payment confirmations, never your full card number or bank credentials. - **Communications:** messages you send us, and our replies. - **Security logs:** standard server logs (IP address, time, page or API endpoint requested, and errors), used to keep the service secure and working. - **Trial records:** a normalised form of your email address, so each person receives one free trial. We don’t collect sensitive personal data (such as health, financial account or biometric data) about you, and we ask that you don’t send it to us. ## 3. Why we use it, and on what basis - **To provide the service** you signed up for: your account, licenses, trials, usage limits and support (performance of our contract with you). - **To bill and keep records** required by tax and accounting law (legal obligation). - **To keep the service secure**: detecting abuse, fraud, repeated trials and license misuse (our legitimate interests, and yours, in a secure service). - **To tell you about your account**: usage warnings, bills, security notices and important changes (performance of contract). - **Product news** only if you ask for it (consent, which you can withdraw at any time). ## 4. Cookies and local storage We use no advertising or analytics cookies, and this website loads no scripts, fonts or trackers from other companies. When you sign in, your browser stores a session token in local storage so you stay signed in; signing out removes it. That storage is strictly necessary for the service and is never used to track you. ## 5. Who we share it with We share personal data only with service providers that help us run Wallras, each bound to use it only to provide their service to us: - **Supabase:** sign-in and account authentication. - **Render:** hosting for this website and our license server. - **Dodo Payments:** payments, invoicing and tax, as merchant of record. - **Google, GitHub and Microsoft:** only if you choose to sign in with them. - **Our email provider:** to send and receive email. We may also disclose data when required by law or a valid legal request, to protect our rights or the safety of others, or as part of a merger or sale of the business, in which case this policy continues to protect it. We never sell personal data. ## 6. International transfers Some of our providers process data outside India, including in the United States and the European Union. Where the law requires, transfers are covered by appropriate safeguards such as standard contractual clauses. ## 7. How long we keep it - Account data: while your account is open, and up to 90 days after you close it. - License and usage records: 24 months, so bills can be checked. - Billing records: as long as tax and accounting law requires (generally up to 8 years in India). - Security logs: up to 30 days, longer only while investigating an incident. - Trial records: for as long as needed to keep trials to one per person. ## 8. How we protect it Connections are encrypted (HTTPS), access to production systems is limited and protected by two-factor authentication, license keys are cryptographically signed, and our software and server are tested for security on every release. See our [security page](security). If a personal data breach affects you, we’ll notify you and the relevant authority as the law requires. ## 9. Your rights Depending on where you live, including under the DPDP Act and the GDPR, you can: - access the personal data we hold about you, and get a copy; - correct or complete it; - have it erased, unless we must keep it by law (for example, billing records); - object to or restrict certain uses, and withdraw consent at any time; - nominate someone to exercise your rights if you’re unable to; - complain to the Data Protection Board of India or your local data protection authority. Email [privacy@wallras.com](mailto:privacy@wallras.com). We reply within 30 days and may need to confirm your identity first. California residents: we don’t sell or share personal information for cross-context advertising. ## 10. Grievance officer For any concern about your personal data, contact our grievance officer at [privacy@wallras.com](mailto:privacy@wallras.com). We acknowledge grievances within 72 hours and resolve them within 30 days. ## 11. Data inside your Wallras installation Data that passes through Wallras on your servers (your users’ prompts and answers, for example) is controlled by you, not us. You decide what is recorded, for how long, and you can delete it at any time with the built-in deletion tools. You are responsible for having a lawful basis and giving your own users any notices the law requires. ## 12. Children Wallras is a business service and isn’t meant for anyone under 18. We don’t knowingly collect data from children; if you believe a child has given us data, contact us and we’ll delete it. ## 13. Changes If we change this policy in a way that matters, we’ll update the date above and email account holders before the change takes effect. ## 14. Contact Privacy questions: [privacy@wallras.com](mailto:privacy@wallras.com). Everything else: [hello@wallras.com](mailto:hello@wallras.com). --- # Terms | Wallras Source: terms.html # Terms of service Effective 13 November 2026 These terms are an agreement between you and Wallras (“we”, “us”), operated from India. They cover wallras.com, your account and the Wallras software (together, the “Service”). By creating an account, ticking the box to agree, starting a trial, buying a plan or using the software, you accept these terms and our [privacy policy](privacy). If you use Wallras for a company or other organisation, you confirm you have authority to accept these terms for it, and “you” means that organisation. ## 1. Who can use Wallras You must be at least 18 and able to form a binding contract. Wallras is a business service, provided for your trade, business or profession. You may not use it if you are barred from receiving it under the laws of India or any other applicable country, including sanctions and export laws. ## 2. Your account Give accurate information and keep it up to date. Keep your sign-in details, admin tokens and license keys secret: you’re responsible for everything done with them. Tell us straight away at [security@wallras.com](mailto:security@wallras.com) if you think they’ve been misused. One free trial is allowed per person; creating extra accounts to get more trials is not allowed. ## 3. What Wallras is Software you run on your own servers. It watches the traffic between your apps and your AI provider and reports on how your AI behaves. It doesn’t change your prompts or your models, and your AI’s answers reach your apps unchanged unless you switch on optional gateway features (retries, fallback providers, caching, limits), which then work as you configure them. ## 4. Your data Everything Wallras records stays on your servers, and you own it. We don’t receive your prompts, answers or logs. You are responsible for that data, including having the right to process it, giving your own users any notices the law requires, and complying with your AI provider’s terms. Licensed installations send us only a license key, an installation id and a monthly request count (and our server sees the time and IP address of each check), as described in our privacy policy. Don’t alter, block or falsify these. ## 5. Plans, trials and usage - **Free:** core features and 10,000 requests a month. At the limit, recording pauses until next month; your AI keeps working. - **Trial:** any paid plan, free for 7 days, once per person. A trial without payment details ends on its own and is never charged. A trial started at checkout with payment details becomes a paid subscription when it ends unless you cancel before then. - **Paid plans:** priced as shown on our pricing page when you subscribe. Paid plans never stop at their limit: extra successful requests are billed after each month at your plan’s rate, capped so a month never costs more than the next plan up. Failed requests don’t count. ## 6. Payments, renewal and cancellation Payments are processed by Dodo Payments as merchant of record, or by invoice. Plans are billed in advance and **renew automatically** each month or year until cancelled; you authorise each renewal charge. Prices exclude taxes, which are added where required. We may change prices for future periods with at least 30 days’ notice; the change applies from your next renewal. You can cancel any time; your plan then runs to the end of the period you paid for and doesn’t renew. If a payment fails or an extra-usage bill stays unpaid for 14 days, your license isn’t renewed and, after a 7-day grace period, Wallras steps down to the Free plan. **Refunds:** payments are non-refundable, including for partly used periods, except where the law requires otherwise or where you were charged in error (for example, twice for the same period). Don’t open a chargeback without contacting us first; we resolve billing errors promptly. ## 7. Your license to the software While your account is in good standing, we grant you a limited, non-exclusive, non-transferable, revocable license to install and use the Wallras software for your own internal business purposes, within your plan’s limits. You may not: - sell, resell, rent, sublicense or offer Wallras as a service to others; - share your license key outside your organisation or use one key beyond your plan; - reverse engineer, decompile or disassemble the software, or try to derive its source code, methods or algorithms, except where the law expressly allows this despite this restriction; - remove, disable or work around license checks, usage reporting or any security feature; - copy, modify or create derivative works of the software, or use it or anything learned from it to build a competing product or service; - use Wallras to break the law, infringe others’ rights, or violate your AI provider’s terms; - probe, overload or attack our website, servers or other customers, or test their security without our written permission (please report vulnerabilities as described on our security page). The open-source clients (the MCP server and Python SDK) are licensed under their own MIT license, which applies to them instead of this section. ## 8. Our intellectual property Wallras, including the software, its methods, documentation, website, name and logo, belongs to us and our licensors and is protected by intellectual property laws. Apart from the license above, no rights are granted to you. If you send us suggestions or feedback, we may use them freely without obligation to you. ## 9. Confidentiality Non-public information either of us shares with the other (such as pricing offered only to you, unreleased features, or security details) must be kept confidential and used only for our relationship, except where it becomes public through no fault of the receiver or must be disclosed by law. ## 10. Third-party services Wallras connects to services we don’t control, such as your AI providers and the sign-in and payment providers. Their terms govern your use of them, and we aren’t responsible for their availability, accuracy or conduct. ## 11. Disclaimers Wallras helps you spot problems in AI behaviour. Its findings are signals for you to review, not guarantees that your AI is correct, safe or compliant; it can’t detect every problem. Reports document monitoring evidence and don’t certify compliance with any law or standard. You remain responsible for your AI systems and the decisions you make about them. To the fullest extent the law allows, the Service is provided “as is” and “as available”, without warranties of any kind, express or implied, including merchantability, fitness for a particular purpose, non-infringement and uninterrupted or error-free operation. ## 12. Limitation of liability To the fullest extent the law allows: we are not liable for any indirect, incidental, special, consequential or punitive damages, or for lost profits, revenue, data or goodwill; and our total liability for all claims relating to the Service is limited to the amount you paid us in the 12 months before the event giving rise to the claim, or US$100 if you haven’t paid anything. Nothing in these terms limits liability that cannot be limited by law, such as for fraud. ## 13. Indemnity You will defend and compensate us against claims, losses and costs (including reasonable legal fees) arising from your misuse of the Service, your breach of these terms, or the data you process with Wallras. ## 14. Suspension and termination You can stop using Wallras and close your account at any time. We may suspend or end your access, with notice where reasonable, for non-payment, breach of these terms, misuse, or if required by law. When access ends, your license ends and the software steps down to the Free plan; sections that by their nature should survive (including 7 to 13 and 16) continue to apply. ## 15. Force majeure Neither of us is liable for delays or failures caused by events beyond reasonable control, such as natural disasters, outages of internet or cloud providers, war, government action or labour disputes. ## 16. Governing law and disputes These terms are governed by the laws of India. We’ll first try to resolve any dispute informally: contact us and give us 30 days to respond. If it isn’t resolved, the courts of India have exclusive jurisdiction, unless the law where you live gives you rights that cannot be waived. ## 17. General These terms and our privacy policy are the entire agreement between us about the Service. If any part is found unenforceable, the rest stays in effect. Not enforcing a right isn’t a waiver of it. You may not transfer these terms without our consent; we may transfer them as part of a merger, acquisition or sale of the business. We send notices to your account email; you can send them to [hello@wallras.com](mailto:hello@wallras.com). ## 18. Changes If we change these terms in a way that matters, we’ll email account holders at least 30 days before the change takes effect. Continuing to use Wallras after that means you accept the new terms; if you don’t, you can cancel before they take effect. ## 19. Contact Questions about these terms: [hello@wallras.com](mailto:hello@wallras.com).