Protecting Data in AI Workflows
Part 2 of the KIS AI Security Series: keep sensitive information in the right hands as prompts, documents and model outputs move through an AI-enabled service.
Every prompt is a data transfer
When someone pastes text into an assistant, information leaves one system and enters another. The same is true when an AI service retrieves an internal document, sends a request to a model provider or writes an answer to a ticketing system.
That movement can be useful and safe, but only when it is understood. Treat prompts, uploaded files, retrieved passages and generated responses as parts of the same information flow. This article turns the first blog’s “make AI use visible” advice into a practical data-protection review.
If you have not read AI Security Foundations, start there for the series map. Our earlier AI Security with Human Oversight article explains why accountable people remain part of the control.
Start with a simple data-flow map
For one AI use case, write down five points: where the data comes from, what the user submits, which provider or model receives it, where the response is stored, and who can act on the result. Include connected search indexes, plug-ins, monitoring tools and support teams. These “side doors” often hold more information than the visible chat screen.
Four controls that make the flow safer
1. Minimise before you send
Purpose should decide the data, not convenience. A support team looking for themes in complaints may need product, date and issue category—not names, phone numbers or full account history. Remove direct identifiers and unnecessary free text before the prompt is created.
Practical check: ask, “If this field were exposed, would the task still work?” If the answer is yes, leave it out. Remember that combinations of harmless-looking fields can still identify a person.
2. Separate approved and unapproved services
Two tools with similar interfaces can have very different retention, training, regional-processing and administrator-control settings. A public chatbot, an enterprise tenant and an API call are not automatically equivalent.
Practical check: maintain a short approved-services list with owner, permitted data class, retention setting, location and review date. Make the safe path easier than shadow use.
3. Keep access controls outside the model
Retrieval can make an assistant appear helpful while quietly widening access. If a user cannot open a document in the source system, the AI service should not be able to reveal it through a summary.
Practical check: enforce identity and document permissions in the application and retrieval layer. Test with accounts that should see different results; do not rely on a prompt instruction to “only show authorised content.”
4. Protect logs and outputs too
Prompt and response logs are valuable for troubleshooting, but they can contain confidential information, secrets or regulated data. A generated report can be copied into email, a ticket or a customer record and become part of a longer retention chain.
Practical check: define what is logged, who can read it, how long it is kept and how sensitive values are redacted. Apply the same discipline to exports and downstream systems as to the original prompt.
An everyday example: an internal policy assistant
A people team wants an assistant that answers questions about leave and benefits. The first design indexes every HR file, including investigation notes and medical documents, because “the assistant should know everything.”
A safer design separates the use case from the data estate. It indexes the approved policy library, applies the employee’s existing access rights, and labels answers as guidance with links to the source policy. Questions that require a caseworker are routed to a human rather than answered from restricted records.
The team tests an employee account, a manager account and a support account. It also tries questions that combine two policies and asks the assistant to reveal hidden instructions. The pilot records the results, limits logging to what is needed and names the owner who can pause the service.
A practical review before launch
Purpose and data
Write the use case in one sentence. List each input field and classify it using your organisation’s existing information categories. Mark anything that must never enter the service.
Provider and contract
Confirm the service boundary, processing locations, retention, deletion, subprocessors, breach notification and whether customer content is used to improve a shared model. Record the evidence and the person who approved it.
Identity and retrieval
Document how users authenticate, how permissions are checked, which service identity retrieves data and how access is revoked. Test both normal and denied paths.
Output and records
Decide where responses may be copied, who reviews them and what becomes an official business record. Add a warning or citation pattern where people could mistake a draft for a decision.
Questions to ask your provider
Can our prompts and files be excluded from model training? Which region processes and stores them? What controls are available for retention, deletion, encryption and administrator access? How are support requests handled? Can we export audit events without exporting full prompt content? Ask for current contractual and technical answers, then revisit them when the service or use case changes.
Coming next: when content fights back
Part 3 will explore prompt injection and malicious instructions in documents, websites and messages. We will look at why model instructions are not an access-control boundary, how to test untrusted content and where human approval belongs in an AI workflow.
KIS AI Security Series · Part 2. Examples are illustrative and intended to help teams discuss practical controls.
Need a clear AI data boundary?
KIS helps teams map AI information flows, review provider controls and design practical guardrails for safer adoption.
Talk to our team →