Monitoring AI Applications: Traces, Alerts and Troubleshooting

AI SECURITY · BLOG 21

A secure AI application needs more than uptime charts. Trace how each answer was produced, spot risky changes early, and give responders enough evidence to act.

Monitor the path that produced the answer

Blog 20 placed a policy gateway between an agent and its tools. This article follows that request into production. A green availability check can miss the failures that matter: the retriever may return no permitted evidence, a new model release may weaken citation coverage, or a denied tool attempt may suddenly spike.

Give a user request a trace identifier at the trusted application boundary. Carry it through authentication, retrieval, reranking, provider calls, validation, and any tool gateway. Each step should record a start and end time plus a bounded outcome. The trace should help an authorised responder reproduce the path without collecting entire private prompts and responses by default.

Choose signals that lead to a decision

Begin with a small set of measures tied to a response. Track request volume and latency by stage, provider timeout and error rate, retrieval empty-result rate, citation validation failures, abstentions, policy denials, approval delays, and tool execution outcomes. If a product has a fixed, measured budget, record usage and cost at the request level for authorised operations staff.

Keep metric labels low in variety. A unique question, user ID, document ID, or trace ID on every metric creates expensive, hard-to-read time series and can expose information. Put high-cardinality identifiers in access-controlled traces with short retention. Prefer a keyed pseudonymous tenant token only when operators need to compare tenant behaviour; do not use a raw tenant name as a public metric label.

Write down a baseline before choosing alert thresholds. One team’s acceptable latency or abstention rate may be a failure for another. Alert on a sustained change in a measure that has an owner and a runbook, then test that the alert reaches someone who can respond.

Keep telemetry useful without copying private content

For a first pass, standard JSON logs can carry a trace ID, service release, stage name, policy version, status, duration, bounded reason code, retrieval count, and whether a citation check passed. Record no bearer tokens, passwords, whole prompts, retrieved text, or personally identifying details. Hashing a value does not automatically make its collection safe; consider linkability, access, retention, and the secret used to create the hash.

import json
import time
from uuid import uuid4

trace_id = uuid4().hex
started = time.perf_counter()

# Run the protected request here. Never pass prompt text to this event.
event = {
    "trace_id": trace_id,
    "stage": "retrieval",
    "release": "2026.10.1",
    "policy_version": "rag-v4",
    "outcome": "evidence_found",
    "result_count": 3,
    "duration_ms": round((time.perf_counter() - started) * 1000, 2),
}
print(json.dumps(event, sort_keys=True))

This small example is a log shape, not an observability platform. In a deployed service, send structured events through your approved collector, restrict who can inspect traces, set retention, and verify that exception handlers do not attach request bodies. Avoid making the trace identifier a credential: it is a correlation value, not a secret.

Trace model and retrieval changes

Record model and retrieval configuration versions, deployment cohort, prompt-template version, and the IDs of source records that were actually permitted. Keep raw source content in its source of truth. In a sampled, access-controlled debug path, retain only the minimum redacted material needed to investigate a reported failure, with a documented expiry.

Run a fixed evaluation set when changing a prompt, model, embedding model, chunking rule, or ranking strategy. Compare citation validity, answer relevance, refusal and abstention behaviour, cross-tenant isolation, and latency. A production dashboard can compare aggregate results by release; it should not turn private examples into a shared dashboard feed.

OpenTelemetry provides common conventions for traces, metrics, and logs. The generative AI conventions are maintained separately and can change; check their current status and your instrumentation’s version before adopting attribute names. Some message attributes may contain personal or otherwise sensitive data, so keep content collection opt-in and privacy-reviewed. See the OpenTelemetry GenAI conventions and the guidance on sensitive semantic attributes.

Turn alerts into a troubleshooting path

For every alert, record who owns it, what evidence they may inspect, the immediate containment step, and how they verify recovery. The recent NIST report on monitoring deployed AI systems discusses why post-deployment monitoring must extend beyond a single model benchmark. Keep a release ledger so the investigation can link a change to the request outcomes it affected.

Close the loop after an incident

Preserve the smallest relevant audit window; identify the affected release and route; disable a risky capability through the trusted control plane; and verify that tenant boundaries, citations, and human approvals work before reopening traffic. Then turn the finding into an evaluation case and a monitored control. A graph that turns green is not a substitute for replaying the failure path.

The final article in this series will collect the working patterns in a downloadable, offline starter project, setup guide, and series map. It will show the checks directly so readers can run them and then decide what a real deployment still needs.