Explicit Decisions in RAG: A Calibrated Decision Substrate with Jev

Open almost any production RAG pipeline and you will find the same quiet pattern: a similarity cutoff of 0.75, a chunk size of 512, “retrieve three times, then stop”, a cache that serves anything above 0.9. These numbers steer the whole system, and almost none of them were ever measured. They are guesses that hardened into config.

JevRAG (v0.3.0) takes a different approach. It treats every mid-flight decision in a RAG pipeline as an explicit call with a typed answer and a calibrated confidence, and it lets the calling code decide what to do with that confidence.

The problem: decisions hidden in glue code

A RAG pipeline makes judgment calls constantly: where to split a document, whether the retrieved evidence is enough, which passages deserve space in the prompt, whether to trust the answer. Usually each is an if statement against a threshold. That has two costs:

• No signal about reliability. A threshold returns yes or no. It never says how sure it is, so you cannot trade coverage for risk.
• No way to compare approaches. If the decision is buried in glue code, swapping a heuristic for a model means rewriting the pipeline, and nothing tells you whether the swap helped.

The design: one interface, swappable backends

Given some state, a decision backend returns a typed answer plus a calibrated confidence. The caller owns the policy: act, retry, escalate, or abstain. The backend sits behind a single interface and can be replaced without touching the pipeline.

Five decisions are wired in:

Decision	Stage	Question
Chunk boundary	Ingestion	Split here, or keep the text merged?
Evidence sufficiency	Retrieval	Is the evidence enough, or should we run another retrieval round?
Context selection	Assembly	Does this passage earn a slot in the generator’s context?
Answer / abstain	Generation	Is the answer grounded in the evidence, or should it be suppressed?
Cache trust	Serving	Is this semantic-cache hit safe to serve?

Cache trust is deliberately narrow: it decides using structural state only, such as similarity, age, and scope, without rereading content.

flowchart LR
  D["Documents"] --> C{"Chunk<br/>boundary"}
  C --> I["Index"]
  Q["Query"] --> K{"Cache<br/>trust"}
  K -->|"serve"| A["Answer"]
  K -->|"miss"| R["Retrieve"]
  I --> R
  R --> S{"Evidence<br/>sufficient?"}
  S -->|"no: another round"| R
  S -->|"yes"| X{"Context<br/>selection"}
  X --> G["Generate"]
  G --> B{"Answer /<br/>abstain"}
  B -->|"grounded"| A
  B -->|"not grounded"| N["Suppress"]

Each diamond is a decision call: state in, typed answer and confidence out.

Backends

Jev is the first backend wired up. To prove the interface is not shaped around a single implementation, there is also a deliberately trivial second backend that uses token log-probabilities and costs nothing extra. That a zero-cost backend slots in unchanged is the evidence that the abstraction generalizes.

What calibration measurement adds

A confidence score is only useful if it means something: when the backend says 0.8, it should be right about 80% of the time. The evaluation harness is calibration-first and checks exactly that, using:

• Risk-coverage curves and AURC: how error falls as you answer fewer questions, and the area under that curve.
• ECE and MCE: average and worst-case gap between stated confidence and observed accuracy.
• Brier score and Brier skill score: a proper scoring rule, and its improvement over a naive baseline.

Evaluation reads JSONL records and runs from the CLI (jevrag eval ...). Setup is Python 3.11+, installed with pip or uv, with API keys in a .env file.

The harness also includes a fixed-iteration baseline: the same stack with the sufficiency gate disabled, so the retrieval loop runs a set number of rounds regardless. Comparing it to the gated run shows whether deciding adaptively actually saves cost without losing accuracy.

What the results show

The sufficiency decision was evaluated on a 700-question test set and a 300-question validation set. On answer-abstain, the Jev backend ties a free alternative. That is a useful result rather than a disappointment: it shows where a paid decision call earns its cost and where a zero-cost signal is enough, which is only knowable because the decisions are measured on the same footing.

Takeaways

• Make mid-pipeline decisions explicit, typed, and confidence-bearing instead of burying thresholds in code.
• Keep the policy (what to do with a confidence) in the caller and the judgment in a swappable backend.
• Measure calibration before trusting any confidence, and always keep a no-gate baseline to compare cost and accuracy against.