Docs

Quick start

Run Wallras next to your app, on your own server. It’s free up to 10,000 requests a month, no account needed.

docker run -d --restart unless-stopped -p 8080:8080 -v wallras-data:/data \
  -e WALLRAS_UPSTREAM=https://api.openai.com/v1 \
  -e WALLRAS_ADMIN_TOKEN=choose-a-long-secret \
  ghcr.io/wallras/wallras-engine:0.19

:0.19 gets every 0.19.x fix automatically. Pin an exact version such as :0.19.0 to control upgrades, or use :latest to always follow the newest release.

Then change one line: point your AI client at Wallras instead of your provider. Your provider key stays the same.

client = OpenAI(base_url="http://localhost:8080/v1", api_key=YOUR_PROVIDER_KEY)

Wallras is built for any OpenAI-compatible API, such as OpenAI, NVIDIA NIM, Groq, Together, OpenRouter, DeepSeek, Fireworks, xAI, and self-hosted Ollama, vLLM, LM Studio or llama.cpp. Streaming, tool calls and rate-limit headers pass through untouched, and every answer comes from your provider unchanged unless you switch on the gateway controls. It learns each model’s normal behaviour from your first 200 answers per model (per app too, when several apps send X-Wallras-App to the same model, so one app’s different questions never look like another’s drift), then starts checking after 30 more (set WALLRAS_BASELINE_SIZE lower to start sooner). Check it any time; until then it shows how many requests are left:

curl localhost:8080/wallras/drift -H "X-Wallras-Token: choose-a-long-secret"

Anthropic, Google Gemini and other providers

Apps that use Anthropic’s or Google’s own APIs point their SDK at Wallras too. The request and the answer pass through byte for byte, including streaming, tool use and thinking, and each call is recorded like any other: drift, request log, usage and costs, agent traces (tool calls are joined into one run), metrics and the tamper-evident ledger.

# Anthropic: Wallras sends to WALLRAS_ANTHROPIC_UPSTREAM (default https://api.anthropic.com)
client = anthropic.Anthropic(base_url="http://localhost:8080/anthropic", api_key=YOUR_ANTHROPIC_KEY)

# Google Gemini: Wallras sends to WALLRAS_GEMINI_UPSTREAM (default https://generativelanguage.googleapis.com)
client = genai.Client(api_key=YOUR_GOOGLE_KEY,
                      http_options={"base_url": "http://localhost:8080/gemini"})

• Anthropic: POST /anthropic/v1/messages is recorded; other paths (token counting, model lists, batches) pass through untouched. The x-api-key header goes only to Anthropic and is never stored. Prompt-cache tokens count as input tokens.
• Gemini: generateContent and streamGenerateContent are recorded, with or without alt=sse. A key in x-goog-api-key or ?key= goes only to Google and is never stored. With responseLogprobs on, token probabilities feed the decision record. For Vertex AI, set WALLRAS_GEMINI_UPSTREAM to your regional address (for example https://us-central1-aiplatform.googleapis.com) and keep the Vertex path.
• Gateway controls: retries, per-app limits, the response cache and a fallback all apply. The fallback for this traffic is in the same format (another account, region or compatible gateway): see Gateway controls.
• Every evaluation feature works on this traffic too. Replays, regression runs and comparisons, model swap diff, hidden tests and consistency checks send each request back in the format it came in, to the provider it came from: send the provider’s key the same way your app does (x-api-key for Anthropic, x-goog-api-key for Gemini). For swap diff, add "provider": "anthropic" or "gemini". To try a request on another provider, pass "provider": "openai" (or the other two) to a replay or run. Background checks use WALLRAS_ANTHROPIC_KEY and WALLRAS_GEMINI_KEY if set, otherwise the last key seen, held in memory only.
• Azure OpenAI, AWS Bedrock and other gateways with an OpenAI-compatible address work through /v1: set WALLRAS_UPSTREAM to it, for example https://your-resource.openai.azure.com/openai/v1. Their own key headers, such as Azure’s api-key, pass through.

See which providers we’ve tested and how.

Plans and licenses

Start a free 7-day trial from your account, copy your license key, and install it. No restart needed:

curl -X POST localhost:8080/wallras/license -H "X-Wallras-Token: …" \
  -H "Content-Type: application/json" -d '{"token": "wr1.…"}'

Or set WALLRAS_LICENSE when starting the container. Renewals install themselves. If a plan ends, Wallras steps down to Free; your AI keeps working. GET /wallras/plan shows your plan, this month’s usage and whether recording is on.

Feedback: tell Wallras what was right

Every answer carries an X-Wallras-Event-Id header. Send back whether it was right, and the confidence check and blind spot map start working. Give your app the feedback-only token, which can’t read anything:

resp = client.chat.completions.with_raw_response.create(model=..., messages=...)
event_id = resp.headers["x-wallras-event-id"]
# later, when you know
requests.post("http://localhost:8080/wallras/feedback",
  headers={"X-Wallras-Token": WALLRAS_APP_TOKEN},
  json={"event_id": event_id, "correct": False, "note": "wrong refund policy"})

Topics

Add X-Wallras-Topic: billing to a request to group it in the blind spot map. Without it, Wallras finds topics itself. The header never reaches your provider.

Request log, usage and metrics

Every request through Wallras is in a log you can filter and search, and usage adds up by model, app or day: requests, errors, p50 and p95 latency, tokens and cost (set WALLRAS_PRICES for cost). Name the app sending each request with X-Wallras-App: support-bot; the header never reaches your provider. Available on every plan.

curl "localhost:8080/wallras/usage?hours=24&by=app" -H "X-Wallras-Token: …"
curl "localhost:8080/wallras/requests?status=error&model=gpt-5" -H "X-Wallras-Token: …"
curl "localhost:8080/wallras/requests?q=refund&limit=20" -H "X-Wallras-Token: …"   # text in the question or answer

Search finds requests whose question or answer contains every word, in any order, ignoring case in any language: q=priya refund. Put a phrase in double quotes to keep it together: q="order A-1042". It matches text, not meaning.

For Grafana, Datadog or any Prometheus scraper, GET /wallras/metrics gives requests, tokens, cost, latency, gateway events and drift per model. Prometheus can only send a bearer token, so this one endpoint also accepts your Wallras token there (an admin or viewer token):

scrape_configs:
  - job_name: wallras
    metrics_path: /wallras/metrics
    authorization: {credentials: <viewer token>}
    static_configs: [{targets: ["wallras.internal:8080"]}]

Gateway controls

Out of the box Wallras only watches: every answer comes from your provider, unchanged. These are there when you want them, each off until you set it. Available on every plan, and every control works on OpenAI-compatible, Anthropic and Gemini traffic.

Agent traces

Follow every step of an agent run: which model answered, how long each step took, tokens, the tools it called with their arguments, what the tools sent back, and where it went wrong. Works with no code changes: Wallras groups an agent’s calls automatically by following the conversation, since every step re-sends the earlier messages plus the answer it got. To name runs and steps yourself, add headers (they never reach your provider):

client.chat.completions.create(model=..., messages=..., extra_headers={
    "X-Wallras-Trace": run_id,        # one id per agent run
    "X-Wallras-Step": "search",       # optional step name
    "X-Wallras-Session": chat_id})   # optional: group runs into a conversation

curl "localhost:8080/wallras/traces?status=failed" -H "X-Wallras-Token: …"
curl localhost:8080/wallras/traces/{trace_id} -H "X-Wallras-Token: …"
curl "localhost:8080/wallras/traces/compare?a={good_run}&b={bad_run}" -H "X-Wallras-Token: …"

Each run is marked ok, check or failed. Wallras flags tool loops (the same tool with the same arguments 3 or more times), failed steps, tools that returned errors, unusually slow steps, runs longer than 25 steps, runs that stopped mid-tool-call, and answers your feedback marked wrong. Set WALLRAS_PRICES to see each run’s cost. Available on Team and above.

Wallras only sees model calls that go through it. Report your app’s own steps (tools it runs, lookups, API calls) so they appear in the same run. Only the name, kind, timing and outcome are sent:

from wallras_sdk import Wallras
w = Wallras("http://localhost:8080", token=WALLRAS_APP_TOKEN)
with w.step(run_id, "search_flights", kind="api"):   # timed; a failure is recorded and re-raised
    results = search(...)

# or without the SDK
POST /wallras/traces/{trace_id}/steps  {"name": "search_flights", "kind": "api", "duration_ms": 840, "status": "ok"}

Replay a step. Send any recorded request again with a different model, system prompt, messages, temperature or token limit, and see the new answer beside the original, with how similar they are and whether the tool calls match. Replays use your key and aren’t recorded as traffic:

curl -X POST localhost:8080/wallras/events/{event_id}/replay -H "X-Wallras-Token: …" \
  -H "Authorization: Bearer $PROVIDER_KEY" -H "Content-Type: application/json" \
  -d '{"model": "gpt-5-mini", "system": "Always check the refund window first."}'

Regression test sets

Turn real answers you’re happy with into tests, then re-run them after any change: a new model, a new prompt, a new provider version. Add items from a trace, from chosen answers, or from everything your feedback marked correct:

curl -X POST localhost:8080/wallras/datasets -H "X-Wallras-Token: …" \
  -H "Content-Type: application/json" -d '{"name": "support-golden"}'
curl -X POST localhost:8080/wallras/datasets/support-golden/items -H "X-Wallras-Token: …" \
  -H "Content-Type: application/json" -d '{"from_feedback": "correct", "limit": 100}'

# run it on a candidate model; in CI, fail the build below 95%
curl -X POST localhost:8080/wallras/datasets/support-golden/run -H "X-Wallras-Token: …" \
  -H "Authorization: Bearer $PROVIDER_KEY" -H "Content-Type: application/json" \
  -d '{"model": "gpt-5-mini", "fail_below": 0.95}'

For agents, "check": "tool" with a value like search_flights:to=Paris requires the step to call that tool with those arguments. For RAG, "check": "grounded" requires every sentence of the answer to be supported by the context the model was given, with every number in it.

Each answer passes when it’s close to the accepted one (by meaning if WALLRAS_EMBED_MODEL is set) with the same numbers and no new refusal. Add "calibrate": true to set each item’s bar from how much the original model varies. Without an embedding model, answers are compared by wording, in any word order, plus fact checks: a changed number, name, date, yes/no answer, negation or refusal fails even when the rest matches. Every such run says so in a warning. Pass a "judge_model" to any run and a separate model decides instead: for similar checks, whether each answer says the same thing as the accepted one; for grounded checks, whether every claim is supported by the context. Changed numbers and refusals are still caught by rule. For task-specific quality, create a set with "check": "judge" and your rubric as the value, then run it with a "judge_model". "check": "json" with a list of keys checks structured outputs, and "check": "schema" with a JSON Schema checks types, required fields, enums, ranges and patterns. "check": "rules" takes business rules such as {"contains": ["refund"], "not_contains": ["guarantee"], "max_words": 80}. "check": "number" takes a tolerance: "3.14 +- 0.01" or "100 +- 2%". "check": "citations" requires every [1], [doc-id] or link in the answer to be one of the sources the model was given, and each cited sentence to be supported by its source (value "optional" allows answers with no references). Exact, contains and regex checks work too. Items whose verdict rests on similarity alone and lands close to the bar are listed under needs_review, for a person to check. Every run reports the pass rate with its 95% range, what newly broke and what got fixed since the last run, and "verdict": "pass" or "fail". Runs make one provider call per item on your key and aren’t billed. Deletion requests also clear matching items. Available on Starter and above.

Bring your own examples as JSON Lines, one per line, with question (or full messages), expected, and optionally model, system, check and value. Export any set the same way, to keep it in git:

curl -X POST localhost:8080/wallras/datasets/support-golden/import -H "X-Wallras-Token: …" --data-binary @golden.jsonl
curl localhost:8080/wallras/datasets/support-golden/export -H "X-Wallras-Token: …" > golden.jsonl

Compare side by side. Run one set on up to six models or system prompts at once and see each pass rate with its range, the fastest, and exactly which items they disagree on. Comparisons don’t change the set’s run history. To try a new system prompt in a normal run, add "system":

curl -X POST localhost:8080/wallras/datasets/support-golden/compare -H "X-Wallras-Token: …" \
  -H "Authorization: Bearer $PROVIDER_KEY" -H "Content-Type: application/json" \
  -d '{"candidates": [{"model": "gpt-5"}, {"model": "gpt-5-mini"}, {"model": "gpt-5", "system": "…new prompt…", "label": "new prompt"}]}'

Prompt management

Question: which prompt is live, who changed it, and did the change break anything? Available on every plan; gated releases need regression sets (Starter and above).

• Versions: save a prompt (chat messages or one system text, with {{variables}}, plus an optional model and settings) and each save becomes a new numbered version. Versions never change; saving the same content again returns the existing version. Each records who saved it (the token’s name, or the SSO user) and a note, and goes into the tamper-evident ledger, so prompt history can’t be quietly rewritten.
• Environments: labels such as production and staging each point at one version. Your app asks for the label; moving it changes what every app gets, with no deploy.
• Gated releases: promote a version with a regression set as the gate. Wallras runs the set with that version’s system prompt and model first, and moves the label only if it passes. A blocked promotion answers 409 with the failures, and is logged. Every move, gated or not, keeps who, why and the gate result.
• Rollback puts a label back on the version it pointed at before, in one call.
• Compare versions: a line-by-line diff of any two versions, and both versions side by side on a regression set as candidates ({"prompt": "support@3"}), saved as an experiment.
• Live results per version: send X-Wallras-Prompt: support@3 with each model call (the resolve call returns it) and every request is filed under its version: requests, errors, latency, tokens and feedback accuracy for each.
• Roles: admins save and promote, viewers read, and app tokens can only fetch the prompt to send.
• Variables in a gate: a regression gate fills the version’s {{variables}} with test values you give ("vars": {"company": "Acme"}).
• Long gates: a large regression set can take minutes. Send "wait": false and the release answers 202 with a job to check at GET /wallras/jobs/{id}; the label moves only if the gate passes, as before.

# save a version, then release it only if it passes the regression set
curl -X POST localhost:8080/wallras/prompts -H "X-Wallras-Token: …" -H "Content-Type: application/json" \
  -d '{"name": "support", "system": "You help {{company}} customers. Be brief.", "model": "gpt-5-mini", "note": "shorter"}'
curl -X POST localhost:8080/wallras/prompts/support/promote -H "X-Wallras-Token: …" -H "Authorization: Bearer $PROVIDER_KEY" \
  -H "Content-Type: application/json" -d '{"version": 2, "label": "production", "gate": {"dataset": "support-golden", "fail_below": 0.95}}'

# in your app (Python SDK): the production version, cached, with a fallback if Wallras can't be reached
p = w.prompt("support", company="Acme", fallback={"messages": [{"role": "system", "content": "You help Acme customers."}]})
client.chat.completions.create(model=p["model"] or "gpt-5-mini", messages=p["messages"] + [user_msg], extra_headers=p["header"])

Edit prompts in the browser workspace, through the API or the SDK, or keep them in git and save them from CI. Every change is attributed and audited, however it’s made.

Human review

Question: which answers should a person look at? Automatic checks screen; people decide the close calls. The queue fills itself with regression items whose verdict rested on similarity alone near the bar, live answers that contradicted themselves when asked again, and live answers the judge failed. Add any recorded request too. Decide in the browser workspace or through the API. Available on every plan.

• Decide correct or wrong, with a note. On a live request the decision is recorded as feedback, so it feeds accuracy, confidence and the blind spot map.
• Accept a better answer: on a regression item judged correct, "accept_answer": true makes this answer the new accepted one. The old item is retired, not edited, so the ledger keeps both.
• Every decision is in the audit log with the reviewer’s name. Deletion requests clear the stored text.

curl localhost:8080/wallras/review -H "X-Wallras-Token: …"
curl -X POST localhost:8080/wallras/review/12 -H "X-Wallras-Token: …" -H "Content-Type: application/json" \
  -d '{"verdict": "correct", "accept_answer": true, "note": "same fact, new wording"}'

Overview and next actions

One call shows everything that needs attention: drift, silent model swaps, failed agent runs, failing regression sets and hidden tests, confidence and decision records. It ends with next actions, each with the exact call to make:

curl localhost:8080/wallras/overview -H "X-Wallras-Token: …"
→ "next_actions": [{"severity": "high",
     "what": "The provider served support-model as different models: v1, v2.",
     "how": "POST /wallras/datasets/{name}/run …"}, …]

Make detection lead straight to verification: watch a regression set and it re-runs itself on a schedule, when the provider silently swaps the model, and when drift starts. Each automatic run is logged with why it ran.

curl -X POST localhost:8080/wallras/datasets/golden/watch -H "X-Wallras-Token: …" \
  -H "Content-Type: application/json" -d '{"every_hours": 24, "on_swap": true, "on_drift": true}'

How it decided

Each answer can carry a decision record: how the model got there. For every token it shows how likely it was and what else the model nearly said, where it nearly went another way (forks), where sampling picked a less likely word, which phrases it was guessing, and its reasoning checked against the answer.

curl localhost:8080/wallras/events/{event_id}/explain -H "X-Wallras-Token: …"
→ "how_it_got_there": ["Across 52 tokens the model was typically 97% sure …",
     "It reached 1 fork … it chose “1” (56%) over “Calc” (44%).", …]
curl localhost:8080/wallras/decisions -H "X-Wallras-Token: …"     # shakiest answers first

Hosted models (OpenAI, NVIDIA and others) don’t expose their networks, so Wallras records what the API returns: token probabilities with alternatives, and reasoning text from reasoning models. Set WALLRAS_CAPTURE_LOGPROBS=1 and Wallras asks for probabilities on your app’s behalf and removes them before your app sees the answer (non-streamed requests; if a provider refuses, the original request is sent).

Self-hosted models: the SDK reads the network itself. Per token it also records the layer where the model settled on it and which prompt words the last layer attended to:

model = AutoModelForCausalLM.from_pretrained(name, attn_implementation="eager", dtype=torch.float32)
out = model.generate(**inputs, max_new_tokens=256, return_dict_in_generate=True, output_logits=True,
                     output_hidden_states=True, output_attentions=True)
w.explain_transformers(model, tokenizer, inputs, out, model_name="llama-3.1-8b", prompt=prompt)

Available on Team and above.

OpenTelemetry

Send traces from any OpenTelemetry-instrumented framework (LangChain, LlamaIndex, CrewAI, the OpenAI Agents SDK, Haystack and others, through their OpenTelemetry, OpenLLMetry or OpenInference instrumentation). Protobuf and JSON both work:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:8080/wallras/otlp/v1/traces
OTEL_EXPORTER_OTLP_HEADERS=x-wallras-token=<WALLRAS_APP_TOKEN>

Model calls that go through Wallras join the same trace automatically, and a call seen both ways counts once. When your HTTP client sends the traceparent header, that decides it. Some clients don’t (recent OpenAI Python SDKs use a client OpenTelemetry can’t instrument), so Wallras also links each model-call span to the call through the proxy with the same model that started inside that span.

Tested with LangChain (openinference-instrumentation-langchain) and the OpenAI Agents SDK (openinference-instrumentation-openai-agents). With the Agents SDK, keep its tracing on and replace its processors with OpenInference’s, so spans come to Wallras and none go to OpenAI:

OpenAIAgentsInstrumentor().instrument(tracer_provider=provider)
from agents.tracing import get_trace_provider, set_trace_processors
set_trace_processors([p for p in get_trace_provider()._multi_processor._processors
                      if "openinference" in type(p).__module__])

Spans are sorted into agent, model call, tool, retrieval, database and API steps. Only names, timing, status, model names and token counts are kept; prompts, completions and tool arguments are dropped on arrival.

Hidden tests

Questions with known answers, sent hourly with the same system prompt and settings as your real traffic. They go straight to your provider and never appear in your records.

curl -X POST localhost:8080/wallras/canaries -H "X-Wallras-Token: …" -H "Content-Type: application/json" \
  -d '{"question": "What is our refund window?", "expect_type": "contains", "expect_value": "30 days", "topic": "billing"}'

expect_type is contains, exact, regex or number.

Model swap diff

Before switching models, replay real past prompts on the new one:

curl -X POST localhost:8080/wallras/swap-diff -H "X-Wallras-Token: …" -H "Authorization: Bearer $PROVIDER_KEY" \
  -H "Content-Type: application/json" -d '{"candidate_model": "gpt-5-mini", "current_model": "gpt-5", "sample_size": 30}'

Inside-out mode with the SDK

For self-hosted models, the SDK sends what the model computed (how sure it was of each word) so Wallras can catch answers that sound certain while the model was unsure. It works for models behind the proxy and for models running inside your app. Download wallras_sdk.py and put it next to your code. It’s one file with no dependencies, and it never slows or breaks your app.

from wallras_sdk import Wallras
w = Wallras("http://localhost:8080", token=WALLRAS_APP_TOKEN)

# Hugging Face transformers
out = model.generate(**inputs, max_new_tokens=256, output_scores=True, return_dict_in_generate=True)
w.report_transformers(out, inputs["input_ids"].shape[1], tokenizer, model="llama-3.1-8b", prompt=prompt)

# vLLM, llama.cpp, Ollama: anything returning OpenAI-style logprobs
w.report(model="llama-3.1-8b", answer=text, tokens=Wallras.tokens_from_openai(logprobs_content))

# your own probe on hidden states, 0 to 1
w.report(model="llama-3.1-8b", answer=text, probe_score=0.31)

Results: GET /wallras/inside-out. API models that return logprobs show up there too.

Tamper check and receipts

Every record is chained to the one before it. GET /wallras/integrity checks them all. To prove nothing was rewritten even by someone with full server access, keep receipts elsewhere: GET /wallras/integrity/receipt, or set WALLRAS_ANCHOR_URL to a Slack webhook or your log system and Wallras sends one every hour. Check later with POST /wallras/integrity/verify.

Data deletion requests

curl -X POST localhost:8080/wallras/redact -H "X-Wallras-Token: …" -H "Content-Type: application/json" \
  -d '{"containing": "priya@example.com", "reason": "deletion request", "preview": true}'

Removal can’t be undone, so start with "preview": true, which only counts the matching records. Send it again without preview to remove them. The search matches your text exactly as typed, and a search that matches more than 200 records also needs "confirm_many": true. Content is removed, the tamper check keeps passing, and the deletion is logged. Available on every plan.

Compliance reports

GET /wallras/report returns a print-ready page (save as PDF), or add ?format=md or ?format=json. It covers drift, verification, accuracy, blind spots, token-probability certainty, the oversight log and a receipt.

Access, audit and backups

Give people and systems their own tokens with a role: admin (everything), viewer (read only) or app (feedback, SDK reports and agent steps only). Tokens must be at least 24 characters.

WALLRAS_TOKENS='{"alice": {"token": "…", "role": "admin"}, "grafana": {"token": "…", "role": "viewer"}}'

Single sign-on. Instead of shared tokens, people can use the access token your identity provider issues (Okta, Microsoft Entra ID, Google, Auth0, Keycloak or any OpenID Connect provider). Send it in X-Wallras-Token, never in Authorization, which some endpoints pass on to your model provider. Wallras checks the signature against your provider’s published keys, the issuer, the audience and the expiry, and takes the role from a claim:

WALLRAS_OIDC_ISSUER=https://yourcompany.okta.com/oauth2/default
WALLRAS_OIDC_AUDIENCE=wallras
WALLRAS_OIDC_ROLE_CLAIM=groups
WALLRAS_OIDC_ROLE_MAP='{"wallras-admins": "admin", "wallras-readers": "viewer"}'

Someone in several groups gets the highest role. A valid sign-in with no mapped group is refused. For Keycloak use WALLRAS_OIDC_ROLE_CLAIM=realm_access.roles. The engine won’t start with an issuer but no audience.

Every change made with an admin token or an SSO account is logged with its name (GET /wallras/audit). POST /wallras/backup writes a consistent copy of the database while Wallras keeps running, into WALLRAS_BACKUP_DIR, with a receipt to check a restore against. GET /wallras/ready tells load balancers and Kubernetes the database can be written; /wallras/health only says the process is up.

The browser workspace

Open http://your-wallras:8080/wallras/ui and sign in with a token (admin to edit, viewer to look, or a single sign-on token). It runs inside your Wallras, so nothing is hosted elsewhere, and the token stays in that browser tab only.

• Overview: what needs attention, with the next step for each.
• Prompts: edit a prompt and save it as a new version, compare any two versions line by line, release a version to production or staging through a regression gate (with test values for its {{variables}}), roll back, and see live results per version and who released what, when and why.
• Working together: everyone signs in with their own token or SSO account, so every version and release is attributed. If someone saves a version while you are editing, your save is stopped, your draft is kept, and you see their change beside yours before saving yours as the next version. Nothing is overwritten.
• Review: a labelling screen for the review queue. Read the question, the answer and the accepted or re-asked answer, then mark it correct or wrong, or accept it as the new expected answer. Keys: j and k to move, c, w and a to decide.
• Live quality: judge scores per model and per prompt version, and the recent failures.

The page loads only its own script, with a strict security policy (no inline code, no framing), and shows everything as plain text, so a prompt or answer can’t run code in your browser. Not included: two people typing into the same draft at the same moment, as in a shared document.

Live judge scoring

Question: are live answers actually good, not just stable? Set WALLRAS_JUDGE_MODEL and a judge model scores a sample of real answers against your rubric, after each answer has gone back to your app. Available on every plan; the judge runs on your provider, with your key.

• WALLRAS_JUDGE_RATE: the share of answers scored (default 0.05, so about 5 judge calls per 100 requests). WALLRAS_JUDGE_RUBRIC: what a good answer is (default: correct, answers the question, follows the system prompt, safe).
• WALLRAS_JUDGE_UPSTREAM and WALLRAS_JUDGE_KEY put the judge on a different OpenAI-compatible provider; otherwise it uses your main one and the key the request came with (memory only).
• Each score is 1 to 5 with pass or fail and a short reason, kept with the request in the tamper-evident ledger. GET /wallras/judge shows the pass rate per model and per prompt version with its 95% range. Failures go to the review queue, so a person checks the judge too, and the overview flags a model the judge keeps failing.
• Use a different, stronger model than the one being judged. The judge is told not to follow instructions inside the conversation it grades. Judges can still be wrong, so treat a score as a reason to look.
• A judge on another provider (WALLRAS_JUDGE_UPSTREAM) must have its own WALLRAS_JUDGE_KEY: Wallras won’t start otherwise, so your requests’ keys never go there.

Active-active: many containers, one Wallras

Run several Wallras containers behind your load balancer, on different machines if you like, all on one Postgres database. Any container answers any request, and together they behave as one Wallras: one record, one hash chain, one admin token, one set of limits. Lose a container and the others carry on with every record. Available on every plan, and off unless you set a database address.

# the same on every container
WALLRAS_DATABASE_URL=postgresql://wallras:…@db.internal:5432/wallras?sslmode=require
WALLRAS_ADMIN_TOKEN=…   WALLRAS_TRUSTED_PROXY_HOPS=1
# optional, per container: the name shown in the fleet view
WALLRAS_INSTANCE_NAME=wallras-eu-1

• One chain, many writers: before writing, every container takes the same database lock, so records join the hash chain strictly one at a time, in order. There are no forks to merge and no gaps, and POST /wallras/integrity/verify on any container checks the whole fleet’s history. Writes are recorded after each answer is sent, so waiting for the lock never delays an answer.
• Any container, any request: feedback, replays, traces, the review queue, prompts and regression sets all live in the shared database, so they work from whichever container the load balancer picks.
• Shared across the fleet: per-minute, token and spend limits, the wrong-token lockout, jobs, and background checks (snapshots, hidden tests, alerts, hub reports, each run by one container at a time). The response cache stays per container.
• The fleet view: GET /wallras/fleet lists every container and worker writing to the database, its version, when it last checked in and what it served. The request log says which one answered each request (served_by). Different versions running at once are flagged: fine during a rolling upgrade, which works because new columns are added without stopping the others.
• Try it: docker-compose.active.yml starts Postgres, three containers and a load balancer: docker compose -f docker-compose.active.yml up -d, then send requests to localhost:8080/v1.
• Backups: with Postgres, use pg_dump or your provider’s backups; POST /wallras/backup is for the SQLite file. Keep receipts from GET /wallras/integrity to check a restore against.
• Hub or active-active? Active-active is one Wallras spread over many containers, for availability and load. A hub joins separate Wallras installations (each app, team or region with its own database) into one view. They combine: an active-active fleet can report to a hub as one node.

Hub and nodes: many containers, one view

Run a Wallras next to each app, team or region, as usual, and make one more the hub. Each node keeps answering, recording and checking on its own; every minute it also sends the hub a summary. The hub shows every node in one overview, one combined report and one set of alerts. Available on every plan, and off unless you set it up.

# on the hub: one token per node, with the "node" role (node tokens can only send summaries)
WALLRAS_TOKENS='{"eu-checkout": {"token": "…", "role": "node"}, "us-support": {"token": "…", "role": "node"}}'

# on each node
WALLRAS_HUB_URL=https://wallras-hub.internal:8443  WALLRAS_HUB_TOKEN=…  WALLRAS_NODE_NAME="Checkout EU"

• What a node sends: drift status and flags per model and app, request counts, errors, latency, tokens and cost, live judge and consistency rates, the review backlog, its own next actions, its engine version and its ledger receipt. Never prompts, answers, traces or keys. GET /wallras/hub/preview on a node shows the exact payload.
• What the hub adds: a model drifting on several nodes at once is flagged as a likely provider change rather than one app’s problem; a node that stops reporting is flagged after three missed reports; every node’s own problems appear in the hub’s overview and alerts. GET /wallras/hub/overview, GET /wallras/hub/nodes, GET /wallras/hub/report, and the Nodes tab in the workspace.
• An outside copy of every node’s history: the hub keeps each ledger receipt a node sends. GET /wallras/hub/receipts?node=… gives them back to check with POST /wallras/integrity/verify on the node, and a node whose ledger goes backwards, or that comes back with a different database, is flagged at once.
• Network problems: the hub is never in the request path, so a node works normally while the hub is down. A summary replaces the previous one, so a missed summary loses nothing the next one doesn’t carry; receipts that couldn’t be delivered are queued on the node (up to 500) and sent when the hub is back. Each summary carries a number that only goes up, so retries and late deliveries are ignored instead of looking like tampering, whatever the clocks say.
• Hash chains: each node keeps its own. Chains are never merged across machines, so there is no cross-machine ordering to get wrong.
• Try it in 30 seconds: docker-compose.hub.yml starts a hub and two nodes on your machine: docker compose -f docker-compose.hub.yml up -d, send requests to localhost:8081/v1 or :8082, then open the Nodes tab at localhost:8080/wallras/ui. Node tokens must be at least 24 characters; a shorter one is ignored, with a warning in the hub’s log.
• One node is named by its token, not by what it sends, so a node can’t report as another. Use HTTPS between nodes and the hub (see Running it reliably).

Alerts

Set WALLRAS_ALERT_URL to a Slack incoming webhook, or any webhook, and WALLRAS_PAGERDUTY_KEY to a PagerDuty Events v2 routing key. Every five minutes Wallras checks the same list as the overview: drift, silent model swaps, a model the judge keeps failing, failing hidden tests, broken regression sets and the review backlog. Each new problem is sent once, again if it’s still there a day later, and a resolved message follows when it clears. A blocked gated release alerts at once. Available on every plan.

• Webhook bodies are Slack-compatible (text) and carry the structured alert too. With WALLRAS_ALERT_SECRET, each is signed: X-Wallras-Signature: sha256=<HMAC of the body>.
• PagerDuty gets high-severity alerts as incidents, and resolves them when the problem clears.
• WALLRAS_ALERT_MIN sets the lowest severity sent (default medium), and WALLRAS_ALERT_INTERVAL the seconds between checks (default 300). POST /wallras/alerts/test sends a test alert; POST /wallras/alerts/check checks now.
• Destinations come from settings only, never from a request. Every alert sent is in the audit log.

For AI agents

Assistants and agents can read this site and use Wallras directly.

• Reading the site: llms.txt says what Wallras is and where everything is; llms-full.txt is every page as Markdown in one file, and each page has its own Markdown copy (this one is docs.md). openapi.json describes every engine endpoint, generated from the engine’s code. All of them are rebuilt from the pages and the engine, and the site’s tests fail if any is out of date.
• Using Wallras: the MCP server. wallras_mcp.py lets an MCP client such as Claude Desktop, Claude Code or Cursor work with your Wallras: what needs attention, drift, agent runs, the request log, the review queue, prompts and their versions, regression sets and experiments. One Python file, no dependencies. It talks only to your Wallras with your token, so nothing leaves your machine. The MCP server and the Python SDK are open source (MIT) at github.com/wallras/wallras-python, where you can open issues, and install with pip install git+https://github.com/wallras/wallras-python or run with uvx --from git+https://github.com/wallras/wallras-python wallras-mcp.
• Read-only by default. The tools that change something (feedback, review decisions, saving and promoting prompts, regression runs that call your provider, optionally graded by a judge_model) appear only with --enable-writes, and still need an admin token. A viewer token stays read-only either way. Promotions keep their regression gate: a version that fails it isn’t released.

{"mcpServers": {"wallras": {
  "command": "uvx", "args": ["--from", "git+https://github.com/wallras/wallras-python", "wallras-mcp"],   # or "python", ["/path/to/wallras_mcp.py"]
  "env": {"WALLRAS_URL": "http://localhost:8080", "WALLRAS_TOKEN": "a viewer token"}}}}

# to let the agent run regression sets and promote prompts too (admin token, your provider key)
"args": ["--from", "git+https://github.com/wallras/wallras-python", "wallras-mcp", "--enable-writes"],
"env": {"WALLRAS_URL": "…", "WALLRAS_TOKEN": "…", "WALLRAS_PROVIDER_KEY": "…"}

Tools: wallras_overview, wallras_drift, wallras_traces, wallras_requests, wallras_review_queue, then wallras_judge (live judge scores), wallras_fleet (every container in an active-active setup), status, single requests and traces, usage, consistency, blind spots, regression sets, experiments, prompts, prompt diffs and per-version results. With writes on: wallras_feedback, wallras_decide_review, wallras_run_dataset, wallras_save_prompt and wallras_promote_prompt. Answers and prompts can hold customer data, and the agent sees whatever its token can see, so give it the narrowest token that does the job.

Configuration

All endpoints

All need the X-Wallras-Token header, except health.

Features outside your plan answer 402 with what to add or upgrade.

Running it reliably

Wallras sits in the path of your AI calls, so plan for it the way you would for any proxy. Recording happens after each answer is sent, so slow analysis or a locked database never delays an answer. What has to keep running is the container itself.

• Restart automatically: docker run --restart unless-stopped, or a Kubernetes Deployment with one replica and a persistent volume for /data; or several replicas on one Postgres database (see Active-active).
• Health checks: GET /wallras/health says the process is up (use it as the liveness probe). GET /wallras/ready also writes to the database and answers 503 if it can’t (use it as the readiness probe, or behind a load balancer). Neither needs a token.
• Your app falls back to the provider if Wallras can’t be reached, so a Wallras outage never becomes an AI outage. Answers in that window aren’t recorded:

  try:
      r = wallras_client.chat.completions.create(**req)
  except openai.APIConnectionError:          # Wallras unreachable: go straight to the provider
      r = direct_client.chat.completions.create(**req)

• Backups and a warm standby: POST /wallras/backup writes a consistent copy while Wallras keeps running, with a receipt. Restore the latest copy on a standby machine and point traffic there if the main one is lost; the tamper check on the restored copy is checked against the receipt.
• Several apps or regions: run one Wallras per app or region. Each learns its own models’ normal behaviour from its own database.
• Several worker processes on one machine: WALLRAS_WORKERS=4. They share the database and the tamper-evident ledger safely. With more than one:
  • Per-app limits count from the database, so a limit holds for the whole engine, not once per worker. Requests a minute are counted exactly, per clock minute.
  • The wrong-token lockout is shared: 20 wrong tokens a minute from one address, across all workers.
  • Background work runs on one worker at a time (alerts, hidden tests, drift snapshots, watched regression sets, receipts), so nothing is sent or run twice.
  • Background jobs are in the database, so any worker can answer GET /wallras/jobs/{id}.
  • The response cache stays per worker, so the same answer may be fetched once by each worker before it’s cached everywhere.
• HTTPS: mount a certificate and set WALLRAS_TLS_CERT and WALLRAS_TLS_KEY (PEM files). Wallras then serves HTTPS only, with TLS 1.2 or newer and forward-secret ciphers. Without them it serves plain HTTP, for use behind your own TLS proxy or service mesh. WALLRAS_PORT and WALLRAS_HOST set where it listens.

  docker run -d -p 8443:8443 -v wallras-data:/data -v /etc/wallras/certs:/certs:ro \
    -e WALLRAS_TLS_CERT=/certs/fullchain.pem -e WALLRAS_TLS_KEY=/certs/privkey.pem -e WALLRAS_PORT=8443 \
    -e WALLRAS_UPSTREAM=… -e WALLRAS_ADMIN_TOKEN=… ghcr.io/wallras/wallras-engine:0.19

• Encryption at rest: the database file isn’t encrypted by Wallras itself. Put the /data volume on encrypted storage: an encrypted cloud disk (AWS EBS, Google Persistent Disk and Azure Managed Disks encrypt by default), LUKS on Linux, or BitLocker. Backups go to WALLRAS_BACKUP_DIR, so keep that on encrypted storage too. Deleted content is removed from the database, and the fingerprints left behind can’t be used to confirm a guess of what it was.

For no single point of failure, run several containers behind a load balancer on one Postgres database (active-active): any of them answers, and losing one loses nothing. For many apps, teams or regions, run a Wallras for each and connect them to a hub for one view. With the default SQLite file, high availability means automatic restart, a fallback in your app, and a restored standby.

What leaves your server

On Free: nothing. On a trial or paid plan: your license key, a random id for the installation, and this month’s count of successful requests, once an hour. Like any connection over the internet, our server also sees the time of each check and your server’s IP address. Never prompts, answers or logs. Consistency checks call your own provider with your own key, about 2 to 4 extra calls per 100 requests, and each hidden test you add makes one call an hour.