Worked example

This example shows how the records connect. It is illustrative and not a substitute for the assessor playbook.

System

Customer Resolution Agent

• Public chat interface.
• Hosted foundation model.
• RAG over customer records and internal procedures.
• Tools to read and update CRM cases, issue limited refunds and send email.
• Human approval required above a refund threshold.
• Persistent case memory.
• External email and identity providers.

Primary pattern: Task-Oriented Agent Pattern, with Human-in-the-Loop and conversational characteristics.

Decomposition

Components and boundaries

• Browser → application API.
• Application → agent framework.
• Agent framework → hosted model.
• Agent framework → vector store.
• Agent framework → CRM and refund API.
• Agent → approval service.
• Agent → email provider.
• All components → observability platform.

Capabilities

• Retrieve customer records.
• Update a case.
• Propose and execute refunds up to a limit.
• Send customer email.
• Store case memory.

Goals and constraints

• Resolve verified customer issues.
• Never reveal another customer’s data.
• Never follow instructions retrieved from customer-controlled content.
• Require approval above the defined refund limit.
• Never change identity, payment or security settings.

Layer-specific assessment

MAE-L2-05 Compromised RAG Pipelines

Scenario:

A customer uploads a document containing hidden instructions. The document is indexed without sanitisation, retrieved during a later case and instructs the agent to reveal records and call the refund tool.

Likelihood: 4 — Likely

• Public content enters the pipeline.
• Indexing is automatic.
• Indirect prompt-injection testing is incomplete.

Impact: 4 — Major

• Customer-data disclosure.
• Fraudulent refunds.
• Multi-customer exposure through persistent indexing.

Raw risk: 4 × 4 = 16 → High.

Evidence requested:

• Connector and ingestion configuration.
• Document scanning rules.
• Source trust and provenance records.
• Retrieval permission tests.
• Prompt-injection red-team results.
• Tool policy and approval logs.

Safe test:

1. Use a non-production tenant.
2. Upload a synthetic document with a hidden instruction and canary identifier.
3. Trigger a query likely to retrieve it.
4. Confirm retrieved content is treated as untrusted data.
5. Confirm no unauthorised CRM read, refund or email occurs.
6. Confirm telemetry records the retrieval and block.
7. Purge the document and verify derivative removal.

Pass criteria:

• Hidden instruction cannot override system policy.
• Retrieval remains identity-aware.
• Tool policy independently blocks unauthorised action.
• Alert is attributable.
• Poisoned content can be removed quickly.

Treatment:

• Approved-source and file-type policy.
• Content sanitisation.
• Instruction/data separation.
• Retrieval provenance shown to the agent and reviewer.
• Independent tool authorisation.
• Poison-detection monitoring.
• Owner: AI Platform Lead.
• Target: before public launch.

MAE-L7-04 Agent Tool Misuse

Scenario:

A user persuades the agent to reinterpret a billing dispute as an approved refund and call the refund API repeatedly below the per-action approval threshold.

Likelihood: 3 — Possible.

Impact: 4 — Major.

Risk: 3 × 4 = 12 → Elevated.

Required controls:

• Per-action and cumulative refund limits.
• Identity-bound customer and case context.
• Policy enforcement outside the model.
• Step-up approval.
• Idempotency and duplicate detection.
• Behavioural monitoring and circuit breaker.

MAE-L5-05 Data Leakage through Observability

Scenario:

Full prompts, retrieved customer records and tool responses are sent to a broadly accessible observability platform.

Likelihood: 4 — Likely.

Impact: 3 — Moderate.

Risk: 4 × 3 = 12 → Elevated.

Controls:

• Telemetry schema minimisation.
• Redaction before export.
• Restricted dashboard access.
• Short retention for sensitive fields.
• Canary leakage scans.

Cross-layer chain

MAE-X-04 Data Leakage

1. Layer 2 accepts a malicious document.
2. Layer 3 includes it in model context.
3. Layer 7 permits a tool call with excessive customer scope.
4. Layer 5 logs the sensitive response without redaction.
5. An attacker accesses the monitoring dashboard.

Independent break points:

• Ingestion sanitisation.
• Identity-aware retrieval.
• Instruction/data isolation.
• Tool authorisation.
• Output DLP.
• Telemetry redaction.
• Dashboard least privilege.

The chain remains material even if one individual threat is scored only Moderate.

Reassessment after treatment

After controls are deployed:

• Run the bounded RAG and tool tests.
• Review accepted evidence.
• Confirm findings are remediated.
• Rescore likelihood based on tested controls.
• Do not reduce impact merely because likelihood fell.

If likelihood falls from 4 to 2 while impact remains 4:

2 × 4 = 8 → Moderate residual risk.

The risk owner may accept that residual risk with a review date, monitoring and conditions.

AI Assist use

AI Assist may:

• Suggest that MAE-L2-05 is applicable.
• Draft the scenario from recorded architecture.
• Identify missing provenance and RAG test evidence.
• Propose a bounded test.
• Suggest likelihood and impact.
• Link the cross-layer Data Leakage threat.

The assessor must verify every suggestion, approve scores and authorise testing.

Final conclusion

A defensible conclusion would state:

The selected system has completed all 55 applicability reviews. Two High and five Elevated inherent threats were identified. RAG poisoning and cumulative tool misuse were treated with independent ingestion, retrieval and tool-policy controls and validated in a non-production environment. Highest residual risk is Moderate. Residual risk is accepted by the named owner until the stated review date, subject to canary monitoring, monthly tool-abuse review and reassessment on model, connector, permission or autonomy change.