# Explicit Decisions in RAG: A Calibrated Decision Substrate Source: https://hlab.in/blog/jev-rag-decision-substrate [Blog](https://hlab.in/blog) Explicit Decisions in RAG: A Calibrated Decision Substrate with Jev # Explicit Decisions in RAG: A Calibrated Decision Substrate with Jev reading time 3 min published 9 Nov 2026 industry AI agents RAG pipelines are full of hardcoded thresholds nobody measures. A design that makes each mid-flight decision explicit, typed, and calibrated, with a swappable backend. Industry [AI agents](https://hlab.in/industries/ai-agentic-development) Topics [AI](https://hlab.in/blog/tag/ai) Published 9 Nov 2026 Reading time 3 min [Let’s talk growth](https://h.hlab.in/contact) On this page - [The problem: decisions hidden in glue code](#the-problem-decisions-hidden-in-glue-code) - [The design: one interface, swappable backends](#the-design-one-interface-swappable-backends) - [Backends](#backends) - [What calibration measurement adds](#what-calibration-measurement-adds) - [What the results show](#what-the-results-show) - [Takeaways](#takeaways) Open almost any production RAG pipeline and you will find the same quiet pattern: a similarity cutoff of 0.75, a chunk size of 512, “retrieve three times, then stop”, a cache that serves anything above 0.9. These numbers steer the whole system, and almost none of them were ever measured. They are guesses that hardened into config. JevRAG (v0.3.0) takes a different approach. It treats every mid-flight decision in a RAG pipeline as an explicit call with a typed answer and a calibrated confidence, and it lets the calling code decide what to do with that confidence. ## The problem: decisions hidden in glue code A RAG pipeline makes judgment calls constantly: where to split a document, whether the retrieved evidence is enough, which passages deserve space in the prompt, whether to trust the answer. Usually each is an `if` statement against a threshold. That has two costs: - **No signal about reliability.** A threshold returns yes or no. It never says how sure it is, so you cannot trade coverage for risk. - **No way to compare approaches.** If the decision is buried in glue code, swapping a heuristic for a model means rewriting the pipeline, and nothing tells you whether the swap helped. ## The design: one interface, swappable backends Given some state, a decision backend returns a typed answer plus a calibrated confidence. The caller owns the policy: act, retry, escalate, or abstain. The backend sits behind a single interface and can be replaced without touching the pipeline. Five decisions are wired in: Decision Stage Question Chunk boundary Ingestion Split here, or keep the text merged? Evidence sufficiency Retrieval Is the evidence enough, or should we run another retrieval round? Context selection Assembly Does this passage earn a slot in the generator’s context? Answer / abstain Generation Is the answer grounded in the evidence, or should it be suppressed? Cache trust Serving Is this semantic-cache hit safe to serve? Cache trust is deliberately narrow: it decides using structural state only, such as similarity, age, and scope, without rereading content. ``` flowchart LR D["Documents"] --> C{"Chunk
boundary"} C --> I["Index"] Q["Query"] --> K{"Cache
trust"} K -->|"serve"| A["Answer"] K -->|"miss"| R["Retrieve"] I --> R R --> S{"Evidence
sufficient?"} S -->|"no: another round"| R S -->|"yes"| X{"Context
selection"} X --> G["Generate"] G --> B{"Answer /
abstain"} B -->|"grounded"| A B -->|"not grounded"| N["Suppress"] ``` _Each diamond is a decision call: state in, typed answer and confidence out._ ## Backends Jev is the first backend wired up. To prove the interface is not shaped around a single implementation, there is also a deliberately trivial second backend that uses token log-probabilities and costs nothing extra. That a zero-cost backend slots in unchanged is the evidence that the abstraction generalizes. ## What calibration measurement adds A confidence score is only useful if it means something: when the backend says 0.8, it should be right about 80% of the time. The evaluation harness is calibration-first and checks exactly that, using: - **Risk-coverage curves and AURC**: how error falls as you answer fewer questions, and the area under that curve. - **ECE and MCE**: average and worst-case gap between stated confidence and observed accuracy. - **Brier score and Brier skill score**: a proper scoring rule, and its improvement over a naive baseline. Evaluation reads JSONL records and runs from the CLI (`jevrag eval ...`). Setup is Python 3.11+, installed with pip or uv, with API keys in a `.env` file. The harness also includes a **fixed-iteration baseline**: the same stack with the sufficiency gate disabled, so the retrieval loop runs a set number of rounds regardless. Comparing it to the gated run shows whether deciding adaptively actually saves cost without losing accuracy. ## What the results show The sufficiency decision was evaluated on a 700-question test set and a 300-question validation set. On answer-abstain, the Jev backend ties a free alternative. That is a useful result rather than a disappointment: it shows where a paid decision call earns its cost and where a zero-cost signal is enough, which is only knowable because the decisions are measured on the same footing. ## Takeaways - Make mid-pipeline decisions explicit, typed, and confidence-bearing instead of burying thresholds in code. - Keep the policy (what to do with a confidence) in the caller and the judgment in a swappable backend. - Measure calibration before trusting any confidence, and always keep a no-gate baseline to compare cost and accuracy against. ## Explore more articles - AI agents ### AI Agents in Production: Patterns That Survive Contact with Real Users 2 min read 28 Jun 2026 [AI Agents in Production: Patterns That Survive Contact with Real Users](https://hlab.in/blog/ai-agents-production-patterns) - AI agents ### MCP Explained: The Model Context Protocol for Agent Tools 2 min read 5 Nov 2026 [MCP Explained: The Model Context Protocol for Agent Tools](https://hlab.in/blog/mcp-model-context-protocol-explained) - AI agents ### ANP: The Agent Network Protocol and the Open Web of Agents 2 min read 4 Nov 2026 [ANP: The Agent Network Protocol and the Open Web of Agents](https://hlab.in/blog/anp-agent-network-protocol) ## Next stop: a 30-minute call. Tell us what you are building. A senior lead will tell you honestly what it takes to ship it and take it to market. [Let’s talk growth](https://h.hlab.in/contact) [Email us](mailto:business@hlab.in)