wallrasCatch AI regressions before your customers do.
Wallras watches your AI in production and tells you when it starts drifting, when it’s confidently wrong, and what will break before you switch models. It runs on your servers: prompts, answers and records never leave them.
Works with Anthropic’s and Google Gemini’s own APIs and any OpenAI-compatible API, such as OpenAI, Azure OpenAI, NVIDIA NIM, Groq, Together, OpenRouter, DeepSeek, Ollama and vLLM. See what we’ve tested, and how.
Your AI changes. Nobody tells you.
The model you tested is not the model answering your customers today. Most teams find out from a complaint.
Providers update quietly
The same model name can serve a different model next month. Answers shift in length, tone and what the AI is willing to do.
Confidence isn't accuracy
AI sounds just as sure when it's wrong. Without a track record, you can't tell a good answer from a convincing one.
Switching models is a gamble
A cheaper or faster model might handle 90% of your traffic perfectly and quietly break the other 10%. You only find out after.
Running in about ten minutes
Wallras sits between your app and your AI provider as a transparent proxy. Requests pass through untouched. By default Wallras only watches.
Run it on your servers
One Docker container on your own infrastructure. Logs and analysis stay there.
Change one line
Point your AI client at Wallras instead of the provider. Streaming, tool calls and rate limits keep working exactly as before.
Let it learn normal
Wallras builds a baseline for each model from real traffic, then flags anything that moves away from it.
# before
client = OpenAI(base_url="https://integrate.api.nvidia.com/v1")
# after: one line, nothing else changes
client = OpenAI(base_url="http://wallras.internal:8080/v1")
# ask what changed
GET /wallras/drift
→ status: drifting
→ "AI is refusing more requests than its baseline"Eight ways to check your AI
Each one answers a question your team can’t answer today. How each one is measured, with thresholds and limits.
Drift radar
Is my AI still behaving like it did last month? Tracks answer length, wording, refusals, certainty and the model actually being served, per model, against its own baseline.
LiveModel swap diff
What breaks if we switch models? Replays your real past prompts on the new model and reports only the differences beyond normal run-to-run randomness.
LiveConfidence check
When it sounds sure, is it right? Compares how certain your AI sounds with how often it has actually been correct, using feedback from your app.
LiveBlind spot map
Where should we not trust it? Groups failures by topic and task so you can see exactly where your AI keeps getting things wrong.
LiveAgent traces
Where did the agent go wrong? Every step of an agent run, with tools called, results, timing, tokens and cost. Flags loops, failing tools and runaway runs, with no code changes.
LiveRegression tests
Did our change break anything? Turns real answers you trust into a test set, re-runs it on any model or prompt, and shows what newly broke. Pass or fail, ready for CI.
LiveHow it decided
Why did it say that? For each answer: how sure it was of every word, where it nearly said something else, which phrases it guessed, and whether its reasoning matches its answer. With self-hosted models, which layer of the network made each decision.
LiveCompliance reports
Can we prove our AI is monitored? Turns your local records into evidence reports for audits, clients and your own board. Evidence for reviews, not a certificate of compliance. See a sample.
LiveOne container. Four jobs.
Teams usually buy a separate tool for each of these. Wallras does all four from the same records, on your own servers.
General-purpose AI observability
Is our AI healthy today?
- Drift per model against its own baseline: length, wording, refusals, certainty
- Catches a different model being served under the same name
- Confidence compared with real accuracy, and blind spots by topic
- Errors, latency, tokens and cost by model or app, with a Prometheus feed for Grafana
- Alerts to Slack, PagerDuty or any webhook when something changes
- Many containers, one view: nodes report to a hub, and prompts never leave each node
- Active-active: many containers on one database, writing one tamper-evident chain
Agent development and debugging
Where did the agent go wrong?
- Every step of a run with tools, results, timing, tokens and cost, with no code changes
- OpenTelemetry from LangChain, the OpenAI Agents SDK and others, joined to the model calls
- Flags loops, failing tools and runaway runs
- Replay any step with another model or prompt, and see how each answer was decided
AI gateway and request monitoring
What is going through, and what does it cost?
- One-line switch for Anthropic, Gemini or any OpenAI-compatible provider, self-hosted models included
- A searchable log of every request, with usage by model or app
- Optional retries, a fallback provider, a response cache and per-app limits, all off by default
- Every request recorded in a tamper-evident log
- Built-in HTTPS, and nothing leaves your servers
Evaluation
Is it still right, and will the change break it?
- Regression sets built from real answers you trust, run in CI
- Rule-based checks first: JSON Schema, business rules, tool calls, citations, grounding, exact text, numbers with tolerances; meaning or an AI judge where rules can’t decide
- A review queue and labelling screen: a person decides close calls, and can accept the better answer
- A judge model scores a sample of live answers against your rubric
- Works the same on Anthropic, Gemini and OpenAI-compatible traffic
- Hidden known-answer tests and consistency checks on live traffic
- Compare models or prompts side by side; import and export sets as JSON Lines
Prompt management
Which prompt is live, and is the new one safe to ship?
- Every prompt version saved, attributed and kept in the tamper-evident log
- Production and staging labels your apps read, with no redeploy
- A new version goes live only if it passes your regression set; one-call rollback
- A browser workspace where your team edits, compares and releases them
- Live results per version: errors, latency, cost and accuracy
Wallras doesn't believe your AI. It verifies it.
An AI's description of itself is the easiest evidence to get wrong. So every signal is weighted by how hard it is to fake.
If your AI says it's 95% sure but has been right 60% of the time on that topic, Wallras treats it as 60%.
Sees as deep as your setup allows
Wallras works out how much of your AI it can observe, and adapts what it checks.
Outside-in
Learns from behaviour over time: what goes in, what comes out, and how that changes.
Hybrid
Adds the model’s own word-by-word probabilities and checks them against how often the model is actually right.
Inside-out
Reads the network itself through the SDK (probabilities from raw logits, the layer where each word was settled, attention) to catch confident-sounding answers the model was actually unsure of. Hosted APIs don’t expose this.
Many containers. One chain.
Put several Wallras containers behind your load balancer, on one Postgres database. Any of them answers any request, and together they keep a single tamper-evident record, as if they were one.
Any container, any request
Feedback, replays and lookups work from whichever container your load balancer picks. Lose one, and the others carry on with every record.
One write at a time
Every container takes the same database lock before it writes, so records join the chain strictly in order. No forks to merge, no gaps to explain.
Shared, not copied
Rate and spend limits, the wrong-token lockout, background checks and jobs hold for the whole fleet, not per container.
See the fleet
The fleet view lists every container and worker, its version and what it served. Mixed versions during a rolling upgrade are flagged.
Leave any time. Nothing breaks.
Most tools hold your data or your workflow hostage. Wallras was built so that walking away costs you nothing.
- Your data stays on your serversPrompts, answers and reports never leave your infrastructure.
- Only a license key and a count go outPaid plans send their key and a monthly count of requests, for billing. No prompts, no answers, no logs. Ever. Free sends nothing.
- Stop paying and it steps downWallras moves to the Free plan. Your AI keeps running exactly as it did, and your records stay yours.
- Watching never blocks answeringRecording and checks happen after each answer is sent, so a slow or failed analysis never touches your traffic. Run Wallras with automatic restarts, or add a short fallback straight to your provider.
- HighNew model version served under the same namellama-3.1-70b now resolves to a different build
- HighAI is refusing more requests than its baseline4% of requests last month, 11% this week
- MediumAnswer length is shiftingAnswers are 1.6x longer on billing questions
- MediumWording style is driftingMore hedging phrases in support replies
See a full sample report, made by the real engine from staged traffic.
Priced per request. Nothing random.
Every price is your plan’s included requests times a per-request rate, and the rate never rises as you grow. Go over and nothing stops: extra requests cost your plan’s rate, and a month never costs more than the next plan up. Every plan starts with a free 7-day trial, no card needed.
Free
10,000 requests a month, free forever
- Drift radar
- Feedback
- Tamper check and receipts
- Data deletion requests
- Request log, usage and metrics
- Gateway: retries, fallback, cache, limits
- Single sign-on
- Prompt versions, labels and rollback
- Human review queue
Team
25,000 requests × $0.002 each. Extra requests at the same rate.
- Drift radar
- Feedback
- Tamper check and receipts
- Data deletion requests
- Request log, usage and metrics
- Gateway: retries, fallback, cache, limits
- Single sign-on
- Prompt versions, labels and rollback
- Human review queue
- Confidence check
- Agent traces
Starter
100,000 requests × $0.002 each. Extra requests at the same rate.
- Drift radar
- Feedback
- Tamper check and receipts
- Data deletion requests
- Request log, usage and metrics
- Gateway: retries, fallback, cache, limits
- Single sign-on
- Prompt versions, labels and rollback
- Human review queue
- Confidence check
- Agent traces
- Model swap diff
- Blind spot map
- Regression test sets
- Add-on: Verification pack
- Add-on: Compliance reports
- Add-on: Automatic signed receipts
Growth
1,000,000 requests × $0.001 each. Extra requests at the same rate.
- Drift radar
- Feedback
- Tamper check and receipts
- Data deletion requests
- Request log, usage and metrics
- Gateway: retries, fallback, cache, limits
- Single sign-on
- Prompt versions, labels and rollback
- Human review queue
- Confidence check
- Agent traces
- Model swap diff
- Blind spot map
- Regression test sets
- Hidden tests
- Consistency checks
- Compliance reports
- Add-on: Automatic signed receipts
Scale
10,000,000 requests × $0.0004 each. Extra requests at the same rate.
- Drift radar
- Feedback
- Tamper check and receipts
- Data deletion requests
- Request log, usage and metrics
- Gateway: retries, fallback, cache, limits
- Single sign-on
- Prompt versions, labels and rollback
- Human review queue
- Confidence check
- Agent traces
- Model swap diff
- Blind spot map
- Regression test sets
- Hidden tests
- Consistency checks
- Compliance reports
- Automatic signed receipts
How the bill works
One request is one successful AI answer through Wallras; a streamed answer counts once, and an agent making five model calls makes five. Errors, rate limits, hidden tests and consistency checks are never billed. You’ll get a message at 80% and 100% of your included requests, so nothing is a surprise.
Consistency checks and hidden tests call your AI provider on your own key (about 2 to 4 extra calls per 100 requests), and you can switch them off. Exactly what’s counted.
Extra usage is billed after the month ends. If it would cost more than moving up a plan, you pay the next plan’s price instead, and we’ll suggest the upgrade.
On Free, recording pauses when you reach 10,000 requests and resumes next month. Your AI keeps working either way. Need more than 10 million requests? Talk to us.
Add a feature without changing plan
| Add-on | On | Price |
|---|---|---|
| Verification packHidden tests, Consistency checks | Starter | $50 / month |
| Compliance reports | Starter | $75 / month |
| Automatic signed receipts | Starter | $25 / month |
| Automatic signed receipts | Growth | $250 / month |
Add-ons are priced per request too, on your plan’s included volume. Prices exclude taxes, which are added at checkout where they apply.
Try it free for 7 days
Start any plan’s trial in a minute, no card needed. You get the full plan, and direct access to the person building it.
- 7 days free. No card needed, and it ends on its own.
- About 10 minutes to set up.
- Pricing is public, so the trial ends with a clear choice.
- Walk away any time and your AI runs exactly as before.
Questions
Does Wallras slow down my AI?
Wallras passes each request straight through and records it after the answer has gone back, so it adds one short hop inside your own infrastructure: about 5 to 7 milliseconds per request in our benchmark on a 4-core laptop, next to AI answers that take 0.5 to 5 seconds. If Wallras’s storage fails, answers still go through. How we measured.
Can Wallras see my data?
No. Wallras runs on your servers, and prompts, answers and records stay there. On a trial or paid plan it sends us only your license key, a random installation id and a monthly count of requests, so we can renew your license and bill extra usage. Our server also sees the time of each check and your server’s IP address, as with any connection. On Free, it sends nothing at all. Your account and payment details are handled separately, as the privacy policy explains. See the data flow.
Which AI providers does it work with?
Anthropic’s Messages API and Google’s Gemini API in their own formats, and any OpenAI-compatible API: OpenAI, Azure OpenAI, NVIDIA NIM, Groq, Together, OpenRouter, DeepSeek, Fireworks, and self-hosted models on Ollama, vLLM, LM Studio or llama.cpp. We’ve run real traffic through NVIDIA NIM, Google Gemini, OpenRouter (28 models from 24 companies, including OpenAI and Anthropic Claude, and Claude in Anthropic’s own Messages format) and a self-hosted model on llama.cpp. Tested providers.
Will Wallras add to my AI provider bill?
Only if you turn it on. Consistency checks are off until you set WALLRAS_CONSISTENCY_RATE: at 0.02 they re-ask 2% of questions (roughly 2 to 4 extra calls per 100 requests). Each hidden test you add runs about once an hour. Both use your own key. Out of the box, trials included, Wallras makes no calls of its own. Worked example.
How does Wallras decide something has drifted?
It learns each model’s normal from its first 200 answers, then compares recent answers on length, wording, refusals, the model actually served, certainty and speed. A change is flagged only if it’s both large and bigger than this model’s own random variation, so naturally varied models don’t cry wolf. Every threshold is published. The method.
Does Wallras change how my AI answers?
Not unless you ask it to. Out of the box Wallras only observes: it doesn't rewrite prompts, filter answers or modify your model, and removing it changes nothing about your AI's behaviour. If you switch on the optional gateway controls, an identical request can be answered from a short-lived cache and a backup provider can answer when yours is down; responses say when that happened.
What happens when I stop paying?
Your plan stays active until the end of the period you paid for, plus a 7-day grace period if a payment fails. Then Wallras steps down to the Free plan: core features keep working, paid features switch off, and traffic always flows. Your past records stay on your servers.
Does Wallras make extra calls to my AI provider?
A few, and only on your key. To check your AI without trusting it, Wallras can re-ask a small share of real questions in different words (off until you set it; 2% is typical), ask once more when answers disagree, and run hidden tests with known answers. At 2% that's about 2 to 4 extra calls per 100 requests, plus one call an hour for each hidden test you add. Nothing runs until you turn it on.
Can records be edited after the fact?
Not without it showing. Every record, from answers to feedback to the oversight log, is chained to the one before it, so editing, adding or deleting any of them is detected. Someone with full server access could try to rebuild the whole chain, which is why Wallras issues receipts: small fingerprints you keep elsewhere, like a Slack channel or your compliance reports. A rebuilt history won't match them.
What if someone asks us to delete their data?
Search by any detail, like an email address, and Wallras removes that content from its records. The tamper check keeps working, and the deletion is logged so you can show you handled the request.
How is this different from logging tools?
Logging tools show you what your AI said. Wallras tells you whether to believe it: whether its behaviour is drifting, whether its confidence matches its track record, and what will change before you switch models.