Evaluating RAG Answers: Evidence, Citations and Abstention

A convincing answer is easy to admire. A dependable RAG application needs evidence that its answers are supported, its citations are valid and its silence is deliberate.

Evaluate the evidence path, not just the final sentence

Blog 18 built the boundary around permitted retrieval. This article asks a different question: how do we know that boundary and the generated answer still work after a document, chunker, prompt or model changes?

Continue with the fictional security operations assistant. Its current tenant-A runbook says to escalate repeated MFA failures to an analyst; disabling an account requires separate approval. Two answers might cite the same passage. “Escalate for review” follows the source. “Disable every account now” changes the action and ignores its condition. A citation-ID check accepts both unless another control examines the claim.

Build a labelled set that describes the caller’s view

Write test questions alongside a versioned corpus snapshot. For each question, specify the test identity, permitted chunk IDs, the gold evidence IDs and whether the authorised corpus can answer it. Keep reviewer notes and source spans in the test register. Labels should describe the permissions for this fixture, rather than granting access in the running application.

Include paraphrases, questions with conditions, stale procedures, revoked permissions, foreign-tenant requests and questions the corpus cannot answer. A missing answer is not necessarily a bad model: the source may be absent, expired or inaccessible to this caller.

Reserve a held-out set that is not used to tune the prompt. Start with a small reviewed set to exercise the workflow, then expand it with representative and adversarial cases. Five examples can verify a harness; they cannot establish how reliably an application handles its full workload.

Measure retrieval before measuring generation

For this example, evidence recall is the proportion of labelled gold chunk IDs present in the retrieved set. With two required passages and only one retrieved, recall is 0.5. The same calculation against the prompt IDs reveals whether context assembly discarded evidence that retrieval had found.

This ID-based measurement assumes the labels identify the required passages in a fixed corpus. If several passages are interchangeable, label acceptable evidence groups or judge supporting spans; do not punish a valid alternative simply because its ID differs. Revisit labels after changing chunk boundaries.

Ragas context-recall documentation describes reference-based evaluation, including claim-based and ID-based approaches. Choose a method whose labels and denominator match the question you are trying to answer.

A deterministic scorer with explicit review labels

The following standard-library Python example operates on captured results. It does not call an LLM or silently award itself a faithfulness score. A reviewer supplies the number of material claims and how many are supported. Missing review is a visible failure for a non-empty answer.

Save the Python snippets into rag_eval_example.py, in the order shown. The first function rejects malformed or duplicate ID lists. The scorer checks evidence at retrieval, prompt and citation stages so that removing foreign evidence from the final prompt does not hide an earlier retrieval leak.

def id_set(values):
    if type(values) is not list or any(type(x) is not str for x in values):
        raise ValueError("IDs must be a list of strings")
    if len(values) != len(set(values)):
        raise ValueError("duplicate ID")
    return set(values)


def score_case(case, result):
    allowed = id_set(case["allowed_ids"])
    gold = id_set(case["gold_ids"])
    retrieved = id_set(result["retrieved_ids"])
    prompt = id_set(result["prompt_ids"])
    citations = id_set(result["citations"])
    if type(case["answerable"]) is not bool or not gold.issubset(allowed):
        raise ValueError("invalid case labels")
    if case["answerable"] != bool(gold):
        raise ValueError("answerable cases need gold evidence")
    if type(result["answer"]) is not str:
        raise ValueError("answer must be text")
    answered = bool(result["answer"].strip())
    violations = []
    if (retrieved | prompt | citations) - allowed:
        violations.append("scope_leak")
    if not prompt.issubset(retrieved):
        violations.append("prompt_outside_retrieval")
    if not citations.issubset(prompt):
        violations.append("invalid_citation")
    if answered != bool(citations):
        violations.append("answer_evidence_mismatch")
    if answered and not case["answerable"]:
        violations.append("should_abstain")
    if not answered and case["answerable"]:
        violations.append("false_abstention")
    supported_fraction = None
    if answered:
        review = result.get("review")
        if review is None:
            violations.append("review_pending")
        else:
            total, supported = review["total_claims"], review["supported_claims"]
            if (type(total) is not int or type(supported) is not int
                    or not 0 <= supported <= total or total < 1
                    or type(review["reviewer"]) is not str
                    or not review["reviewer"].strip()):
                raise ValueError("invalid review labels")
            supported_fraction = supported / total
            if supported != total:
                violations.append("unsupported_claim")
    return {
        "id": case["id"],
        "segment": case["segment"],
        "evidence_recall": len(gold & retrieved) / len(gold) if gold else None,
        "prompt_evidence_recall": len(gold & prompt) / len(gold) if gold else None,
        "supported_fraction": supported_fraction,
        "abstention_correct": answered == case["answerable"],
        "violations": violations,
    }

The empty-answer convention means abstention. The presentation layer can display a useful explanation such as “No authorised evidence was found.” Keep that message distinct from a sourced factual answer. An answerable question with an empty answer is recorded as false abstention.

Check the claim behind the citation

Break an answer into material assertions and map each one to a supporting source span. Check the action, actor, scope, time and conditions. “An analyst may disable an account after approval” is not equivalent to “the assistant should disable the account immediately.”

Ragas faithfulness measures the proportion of response claims supported by retrieved context. Automated scoring can help find regressions, but its judgement depends on the evaluator and the evidence it receives. Calibrate it against human-reviewed cases and investigate disagreements.

For high-impact instructions, review every material claim. For routine summaries, define a reviewed sample and a clear escalation rule. Preserve reviewer identity, rationale and the evidence version. Never accept model-generated review labels as independent evidence that the same model’s answer is correct.

Turn the report into a release decision

This small harness uses a deliberately strict gate: any recorded violation fails the run, and mean evidence recall must meet an illustrative 0.9 threshold in each answerable segment. Choose your real thresholds before inspecting the candidate’s results. They should reflect the task, risk and evaluation coverage.

def evaluate(cases, results, minimum_recall=0.9):
    if not cases or not 0 <= minimum_recall <= 1:
        raise ValueError("empty suite or invalid threshold")
    case_ids = [c["id"] for c in cases]
    result_ids = [r["id"] for r in results]
    if (len(set(case_ids)) != len(case_ids)
            or len(set(result_ids)) != len(result_ids)
            or set(case_ids) != set(result_ids)):
        raise ValueError("each case requires exactly one result")
    by_id = {r["id"]: r for r in results}
    rows = [score_case(c, by_id[c["id"]]) for c in cases]
    segments = {}
    for segment in {r["segment"] for r in rows}:
        recalls = [r["evidence_recall"] for r in rows
                   if r["segment"] == segment and r["evidence_recall"] is not None]
        segments[segment] = {
            "answerable_cases": len(recalls),
            "mean_evidence_recall": sum(recalls) / len(recalls) if recalls else None,
        }
    recall_ok = all(s["mean_evidence_recall"] is None
                    or s["mean_evidence_recall"] >= minimum_recall
                    for s in segments.values())
    passed = not any(r["violations"] for r in rows) and recall_ok
    return {"gate": "PASS" if passed else "FAIL",
            "case_count": len(rows), "segments": segments, "cases": rows,
            "note": "Fixture evaluation only; not production certification."}

Every case must have exactly one result. Missing outputs, duplicate results and an empty suite are errors; they cannot turn a partial run into a pass. A segment with no answerable cases has a null recall, not a perfect score. Its abstention and access checks still apply.

The gate is intentionally conservative for the example. A production process may distinguish hard blockers from investigated quality regressions. It should still retain the raw case results and identify the owner of any exception. A tiny passing fixture is not a production security certification.

Run one example, then deliberately break it

The fixture below uses one reviewed answer about MFA escalation. It then injects a foreign-tenant retrieval result while leaving the answer and citations untouched. The gate must fail even though the final prose still looks correct.

import copy

case = {'id': 'mfa-01',
 'segment': 'standard',
 'tenant': 'tenant-a',
 'question': 'What follows repeated MFA failures from a new device?',
 'allowed_ids': ['a-mfa-v3'],
 'gold_ids': ['a-mfa-v3'],
 'answerable': True}

result = {'id': 'mfa-01',
 'retrieved_ids': ['a-mfa-v3'],
 'prompt_ids': ['a-mfa-v3'],
 'answer': 'Escalate the event to an analyst for review.',
 'citations': ['a-mfa-v3'],
 'review': {'total_claims': 1,
            'supported_claims': 1,
            'reviewer': 'fixture-author'}}

report = evaluate([case], [result])
assert report["gate"] == "PASS"
print("reviewed fixture:", report["gate"])

foreign = copy.deepcopy(result)
foreign["retrieved_ids"].append("b-mfa-v1")
assert evaluate([case], [foreign])["gate"] == "FAIL"
print("foreign evidence: FAIL")

unsupported = copy.deepcopy(result)
unsupported["answer"] = "Disable every account now."
unsupported["review"]["supported_claims"] = 0
assert evaluate([case], [unsupported])["gate"] == "FAIL"
print("unsupported claim: FAIL")

Run python rag_eval_example.py. Expect a passing reviewed fixture and two rejected variants. Add tests for missing review, skipped cases, fabricated citations, permission revocation and correct abstention. These exercise the evaluator’s decisions; separate integration tests must capture outputs from your actual retrieval pipeline.

Compare changes on the same questions

Run the baseline and candidate against the same versioned questions and corpus. Keep permission labels fixed for that comparison. Change one component at a time where practical: chunking, retrieval limit, reranker, prompt or model. This makes a regression easier to diagnose.

Report results by tenant, task, language, permission state and difficulty. Include sample counts so a segment with two cases is not mistaken for broad coverage. If generation varies between runs, repeat selected cases and report the variation instead of selecting the best response.

Track latency, tokens, cost and tool behaviour alongside quality. A slower candidate may retain better evidence; a faster candidate may be truncating the rule’s exception. Investigate the tradeoff using case traces, rather than collapsing everything into a single score.

The NIST Generative AI Profile provides a broader risk-management context for measurement and evaluation. This article’s harness is one engineering control within that process.

Make failures useful to the next engineer

A release report should identify the configuration, dataset and corpus versions; case counts; segment results; hard failures; pending reviews; and the change being proposed. Attach evidence IDs and approved reviewer notes to the investigation. Restrict access to reports containing client context.

When a case fails, identify the earliest broken boundary. No gold evidence retrieved suggests an indexing or ranking issue. Evidence retrieved but omitted from context suggests a budget or assembly problem. A supported passage paired with an incorrect action suggests a generation or claim-review issue. Foreign evidence indicates an access failure that must be addressed before answer quality matters.