# Wallras: catch AI regressions before your customers do Source: index.html Free 7-day trial, no card needed # Catch AI regressions before your customers do. Wallras watches your AI in production and tells you when it starts drifting, when it’s confidently wrong, and what will break before you switch models. It runs on your servers: prompts, answers and records never leave them. [Start free 7-day trial](checkout?tier=growth) [Watch it catch problems](demo) Example: **support-assistant** on llama-3.1-70b Baseline This week Works with **Anthropic’s and Google Gemini’s own APIs** and any OpenAI-compatible API, such as **OpenAI, Azure OpenAI, NVIDIA NIM, Groq, Together, OpenRouter, DeepSeek, Ollama and vLLM**. [See what we’ve tested, and how](methodology#compat). ## Your AI changes. Nobody tells you. The model you tested is not the model answering your customers today. Most teams find out from a complaint. ### Providers update quietly The same model name can serve a different model next month. Answers shift in length, tone and what the AI is willing to do. ### Confidence isn't accuracy AI sounds just as sure when it's wrong. Without a track record, you can't tell a good answer from a convincing one. ### Switching models is a gamble A cheaper or faster model might handle 90% of your traffic perfectly and quietly break the other 10%. You only find out after. ## Running in about ten minutes Wallras sits between your app and your AI provider as a transparent proxy. Requests pass through untouched. By default Wallras only watches. 1. ### Run it on your servers One Docker container on your own infrastructure. Logs and analysis stay there. 1. ### Change one line Point your AI client at Wallras instead of the provider. Streaming, tool calls and rate limits keep working exactly as before. 1. ### Let it learn normal Wallras builds a baseline for each model from real traffic, then flags anything that moves away from it. ``` # before client = OpenAI(base_url="https://integrate.api.nvidia.com/v1") # after: one line, nothing else changes client = OpenAI(base_url="http://wallras.internal:8080/v1") # ask what changed GET /wallras/drift → status: drifting → "AI is refusing more requests than its baseline" ``` ## Eight ways to check your AI Each one answers a question your team can’t answer today. [How each one is measured](methodology), with thresholds and limits. ### Drift radar **Is my AI still behaving like it did last month?** Tracks answer length, wording, refusals, certainty and the model actually being served, per model, against its own baseline.Live ### Model swap diff **What breaks if we switch models?** Replays your real past prompts on the new model and reports only the differences beyond normal run-to-run randomness.Live ### Confidence check **When it sounds sure, is it right?** Compares how certain your AI sounds with how often it has actually been correct, using feedback from your app.Live ### Blind spot map **Where should we not trust it?** Groups failures by topic and task so you can see exactly where your AI keeps getting things wrong.Live ### Agent traces **Where did the agent go wrong?** Every step of an agent run, with tools called, results, timing, tokens and cost. Flags loops, failing tools and runaway runs, with no code changes.Live ### Regression tests **Did our change break anything?** Turns real answers you trust into a test set, re-runs it on any model or prompt, and shows what newly broke. Pass or fail, ready for CI.Live ### How it decided **Why did it say that?** For each answer: how sure it was of every word, where it nearly said something else, which phrases it guessed, and whether its reasoning matches its answer. With self-hosted models, which layer of the network made each decision.Live ### Compliance reports **Can we prove our AI is monitored?** Turns your local records into evidence reports for audits, clients and your own board. Evidence for reviews, not a certificate of compliance. [See a sample](sample-report).Live ## One container. Four jobs. Teams usually buy a separate tool for each of these. Wallras does all four from the same records, on your own servers. ### General-purpose AI observability **Is our AI healthy today?** - Drift per model against its own baseline: length, wording, refusals, certainty - Catches a different model being served under the same name - Confidence compared with real accuracy, and blind spots by topic - Errors, latency, tokens and cost by model or app, with a Prometheus feed for Grafana - Alerts to Slack, PagerDuty or any webhook when something changes - Many containers, one view: nodes report to a hub, and prompts never leave each node - Active-active: many containers on one database, writing one tamper-evident chain [The eight features](#features) ### Agent development and debugging **Where did the agent go wrong?** - Every step of a run with tools, results, timing, tokens and cost, with no code changes - OpenTelemetry from LangChain, the OpenAI Agents SDK and others, joined to the model calls - Flags loops, failing tools and runaway runs - Replay any step with another model or prompt, and see how each answer was decided [Connect your framework](docs#otel) ### AI gateway and request monitoring **What is going through, and what does it cost?** - One-line switch for Anthropic, Gemini or any OpenAI-compatible provider, self-hosted models included - A searchable log of every request, with usage by model or app - Optional retries, a fallback provider, a response cache and per-app limits, all off by default - Every request recorded in a tamper-evident log - Built-in HTTPS, and nothing leaves your servers [Gateway controls](docs#gateway) ### Evaluation **Is it still right, and will the change break it?** - Regression sets built from real answers you trust, run in CI - Rule-based checks first: JSON Schema, business rules, tool calls, citations, grounding, exact text, numbers with tolerances; meaning or an AI judge where rules can’t decide - A review queue and labelling screen: a person decides close calls, and can accept the better answer - A judge model scores a sample of live answers against your rubric - Works the same on Anthropic, Gemini and OpenAI-compatible traffic - Hidden known-answer tests and consistency checks on live traffic - Compare models or prompts side by side; import and export sets as JSON Lines [How it’s measured](methodology#regression) ### Prompt management **Which prompt is live, and is the new one safe to ship?** - Every prompt version saved, attributed and kept in the tamper-evident log - Production and staging labels your apps read, with no redeploy - A new version goes live only if it passes your regression set; one-call rollback - A browser workspace where your team edits, compares and releases them - Live results per version: errors, latency, cost and accuracy [Prompt management](docs#prompts) ## Wallras doesn't believe your AI. It verifies it. An AI's description of itself is the easiest evidence to get wrong. So every signal is weighted by how hard it is to fake. Real outcomesWas the answer actually right? Hidden known-answer testsSent with the same settings as real traffic Consistency checksSame question, asked differently Token probabilitiesWhere the provider exposes them The AI's stated confidenceDiscounted by its track record The AI's own explanationOften written after the fact If your AI says it's 95% sure but has been right 60% of the time on that topic, Wallras treats it as 60%. ## Sees as deep as your setup allows Wallras works out how much of your AI it can observe, and adapts what it checks. ### Outside-in Hosted APIs Learns from behaviour over time: what goes in, what comes out, and how that changes. ### Hybrid APIs with token probabilities, or self-hosted models Adds the model’s own word-by-word probabilities and checks them against how often the model is actually right. ### Inside-out Self-hosted models, with the Wallras SDK Reads the network itself through the SDK (probabilities from raw logits, the layer where each word was settled, attention) to catch confident-sounding answers the model was actually unsure of. Hosted APIs don’t expose this. ## Many containers. One chain. Put several Wallras containers behind your load balancer, on one Postgres database. Any of them answers any request, and together they keep a single tamper-evident record, as if they were one. ### Any container, any request Feedback, replays and lookups work from whichever container your load balancer picks. Lose one, and the others carry on with every record. ### One write at a time Every container takes the same database lock before it writes, so records join the chain strictly in order. No forks to merge, no gaps to explain. ### Shared, not copied Rate and spend limits, the wrong-token lockout, background checks and jobs hold for the whole fleet, not per container. ### See the fleet The fleet view lists every container and worker, its version and what it served. Mixed versions during a rolling upgrade are flagged. ## Leave any time. Nothing breaks. Most tools hold your data or your workflow hostage. Wallras was built so that walking away costs you nothing. - **Your data stays on your servers**Prompts, answers and reports never leave your infrastructure. - **Only a license key and a count go out**Paid plans send their key and a monthly count of requests, for billing. No prompts, no answers, no logs. Ever. Free sends nothing. - **Stop paying and it steps down**Wallras moves to the Free plan. Your AI keeps running exactly as it did, and your records stay yours. - **Watching never blocks answering**Recording and checks happen after each answer is sent, so a slow or failed analysis never touches your traffic. Run Wallras with automatic restarts, or add a short fallback straight to your provider. **Example weekly report**Status drifting - High New model version served under the same namellama-3.1-70b now resolves to a different build - High AI is refusing more requests than its baseline4% of requests last month, 11% this week - Medium Answer length is shiftingAnswers are 1.6x longer on billing questions - Medium Wording style is driftingMore hedging phrases in support replies [See a full sample report](sample-report), made by the real engine from staged traffic. ## Priced per request. Nothing random. Every price is your plan’s included requests times a per-request rate, and the rate never rises as you grow. Go over and nothing stops: extra requests cost your plan’s rate, and a month never costs more than the next plan up. Every plan starts with a free 7-day trial, no card needed. ### Free $0 forever **10,000** requests a month, free forever - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue [Start free](#how) ### Team $50 / month **25,000** requests × **$0.002** each. Extra requests at the same rate. - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue - Confidence check - Agent traces [Start 7-day trial](checkout?tier=team) ### Starter $200 / month **100,000** requests × **$0.002** each. Extra requests at the same rate. - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue - Confidence check - Agent traces - Model swap diff - Blind spot map - Regression test sets - Add-on: Verification pack - Add-on: Compliance reports - Add-on: Automatic signed receipts [Start 7-day trial](checkout?tier=starter) ### Growth $1,000 / month **1,000,000** requests × **$0.001** each. Extra requests at the same rate. - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue - Confidence check - Agent traces - Model swap diff - Blind spot map - Regression test sets - Hidden tests - Consistency checks - Compliance reports - Add-on: Automatic signed receipts [Start 7-day trial](checkout?tier=growth) ### Scale $4,000 / month **10,000,000** requests × **$0.0004** each. Extra requests at the same rate. - Drift radar - Feedback - Tamper check and receipts - Data deletion requests - Request log, usage and metrics - Gateway: retries, fallback, cache, limits - Single sign-on - Prompt versions, labels and rollback - Human review queue - Confidence check - Agent traces - Model swap diff - Blind spot map - Regression test sets - Hidden tests - Consistency checks - Compliance reports - Automatic signed receipts [Start 7-day trial](checkout?tier=scale) ### How the bill works One request is one successful AI answer through Wallras; a streamed answer counts once, and an agent making five model calls makes five. Errors, rate limits, hidden tests and consistency checks are never billed. You’ll get a message at 80% and 100% of your included requests, so nothing is a surprise. Consistency checks and hidden tests call your AI provider on your own key (about 2 to 4 extra calls per 100 requests), and you can switch them off. [Exactly what’s counted](methodology#billing). Extra usage is billed after the month ends. If it would cost more than moving up a plan, you pay the next plan’s price instead, and we’ll suggest the upgrade. On Free, recording pauses when you reach 10,000 requests and resumes next month. Your AI keeps working either way. Need more than 10 million requests? [Talk to us](mailto:hello@wallras.com?subject=Wallras%20Enterprise). ### Add a feature without changing plan | Add-on | On | Price | |---|---|---| | Verification packHidden tests, Consistency checks | Starter | $50 / month | | Compliance reports | Starter | $75 / month | | Automatic signed receipts | Starter | $25 / month | | Automatic signed receipts | Growth | $250 / month | Add-ons are priced per request too, on your plan’s included volume. Prices exclude taxes, which are added at checkout where they apply. ## Try it free for 7 days Start any plan’s trial in a minute, no card needed. You get the full plan, and direct access to the person building it. [Start free trial](checkout?tier=growth)[Watch the live demo](demo) - **7 days free.** No card needed, and it ends on its own. - **About 10 minutes** to set up. - **Pricing is public,** so the trial ends with a clear choice. - **Walk away any time** and your AI runs exactly as before. ## Questions ### Does Wallras slow down my AI? Wallras passes each request straight through and records it after the answer has gone back, so it adds one short hop inside your own infrastructure: about 5 to 7 milliseconds per request in our benchmark on a 4-core laptop, next to AI answers that take 0.5 to 5 seconds. If Wallras’s storage fails, answers still go through. [How we measured](methodology#performance). ### Can Wallras see my data? No. Wallras runs on your servers, and prompts, answers and records stay there. On a trial or paid plan it sends us only your license key, a random installation id and a monthly count of requests, so we can renew your license and bill extra usage. Our server also sees the time of each check and your server’s IP address, as with any connection. On Free, it sends nothing at all. Your account and payment details are handled separately, as the [privacy policy](privacy) explains. [See the data flow](methodology#dataflow). ### Which AI providers does it work with? Anthropic’s Messages API and Google’s Gemini API in their own formats, and any OpenAI-compatible API: OpenAI, Azure OpenAI, NVIDIA NIM, Groq, Together, OpenRouter, DeepSeek, Fireworks, and self-hosted models on Ollama, vLLM, LM Studio or llama.cpp. We’ve run real traffic through NVIDIA NIM, Google Gemini, OpenRouter (28 models from 24 companies, including OpenAI and Anthropic Claude, and Claude in Anthropic’s own Messages format) and a self-hosted model on llama.cpp. [Tested providers](methodology#compat). ### Will Wallras add to my AI provider bill? Only if you turn it on. Consistency checks are off until you set `WALLRAS_CONSISTENCY_RATE`: at 0.02 they re-ask 2% of questions (roughly 2 to 4 extra calls per 100 requests). Each hidden test you add runs about once an hour. Both use your own key. Out of the box, trials included, Wallras makes no calls of its own. [Worked example](methodology#extra). ### How does Wallras decide something has drifted? It learns each model’s normal from its first 200 answers, then compares recent answers on length, wording, refusals, the model actually served, certainty and speed. A change is flagged only if it’s both large and bigger than this model’s own random variation, so naturally varied models don’t cry wolf. Every threshold is published. [The method](methodology#drift). ### Does Wallras change how my AI answers? Not unless you ask it to. Out of the box Wallras only observes: it doesn't rewrite prompts, filter answers or modify your model, and removing it changes nothing about your AI's behaviour. If you switch on the optional gateway controls, an identical request can be answered from a short-lived cache and a backup provider can answer when yours is down; responses say when that happened. ### What happens when I stop paying? Your plan stays active until the end of the period you paid for, plus a 7-day grace period if a payment fails. Then Wallras steps down to the Free plan: core features keep working, paid features switch off, and traffic always flows. Your past records stay on your servers. ### Does Wallras make extra calls to my AI provider? A few, and only on your key. To check your AI without trusting it, Wallras can re-ask a small share of real questions in different words (off until you set it; 2% is typical), ask once more when answers disagree, and run hidden tests with known answers. At 2% that's about 2 to 4 extra calls per 100 requests, plus one call an hour for each hidden test you add. Nothing runs until you turn it on. ### Can records be edited after the fact? Not without it showing. Every record, from answers to feedback to the oversight log, is chained to the one before it, so editing, adding or deleting any of them is detected. Someone with full server access could try to rebuild the whole chain, which is why Wallras issues receipts: small fingerprints you keep elsewhere, like a Slack channel or your compliance reports. A rebuilt history won't match them. ### What if someone asks us to delete their data? Search by any detail, like an email address, and Wallras removes that content from its records. The tamper check keeps working, and the deletion is logged so you can show you handled the request. ### How is this different from logging tools? Logging tools show you what your AI said. Wallras tells you whether to believe it: whether its behaviour is drifting, whether its confidence matches its track record, and what will change before you switch models.