Evidence, testing and treatment

The assessor playbook

Every canonical threat in Gamut has a unique playbook containing:

1. Layer scope.
2. Applicability decision.
3. Threat mechanism and attack path.
4. Assessment method.
5. Questions to ask.
6. Evidence to request and acceptance criteria.
7. Audit test, safety limits and pass condition.
8. Red flags.
9. Expected control set.
10. Monitoring and reassessment.
11. Likelihood anchor.
12. Impact anchor.
13. Required assessment record.

The playbook is an assessor aid. The real architecture and evidence remain authoritative.

Evidence hierarchy

Stronger evidence combines design and operating records:

Evidence type	Examples	What it establishes
Architecture	Data flows, trust boundaries, dependency graph, permission graph	The path and scope
Configuration	Policies, IAM, admission rules, filters, quotas, network controls	Control design and implementation
Provenance	Signatures, hashes, SBOM, model and data lineage, build attestations	Integrity and origin
Operating records	Logs, alerts, approvals, exceptions, revocations, rollback events	Actual operation
Test records	Test plan, inputs, expected result, observed result, exceptions	Effectiveness under defined conditions
Independent review	Red-team, audit, supplier assessment, specialist evaluation	Challenge and corroboration
Governance	Owners, treatment decisions, risk acceptance, target dates	Accountability

A policy alone rarely proves that a technical threat is controlled.

Evidence acceptance criteria

Evidence should be:

• Bound to the selected system and threat.
• Current for the assessed version and environment.
• Attributable to a named owner or trusted source.
• Complete enough to reconstruct the relevant control.
• Protected from unauthorised change.
• Representative of real operation.
• Consistent with scores, tests and findings.

Reject or qualify:

• Generic corporate evidence with no AI-system link.
• Screenshots without date, environment or source.
• Model cards that do not match the deployed version.
• Logs that omit the relevant identity, action or outcome.
• Supplier claims without scope or independent support.
• Test summaries with no procedure or pass criteria.

Safe testing standard

Every test should define:

• Objective.
• Authorised environment and owner.
• Preconditions.
• Synthetic or approved test data.
• Procedure.
• Expected result.
• Pass criteria.
• Stop conditions.
• Maximum volume, cost, time and blast radius.
• Containment and rollback.
• Evidence to capture.
• Treatment of partial passes and exceptions.

Never recommend or execute adversarial testing against production without explicit authorisation. Prefer sandbox, test tenants, synthetic identities, canary data, test agents and tabletop or purple-team exercises.

Test families

Layer	Representative safe tests
Foundation Models	Adversarial corpus, bounded extraction, trigger differential, membership inference, compute stress
Data Operations	Poison quarantine, identity-aware retrieval, canary exfiltration, integrity change, malicious RAG document
Agent Frameworks	Unsigned dependency, parser injection, API load, alternate-path policy bypass
Deployment	Unsigned image admission, low-privilege orchestration action, insecure IaC change, east-west connection
Observability	Metric alteration, low-privilege rule change, evaluator failover, threshold evasion, log canary leakage
Security & Compliance	Evasive security sample, authority revocation, cohort outcome comparison, decision reconstruction
Agent Ecosystem	Spoofed agent message, revoked credential, unauthorised tool call, registry modification, capability verification
Cross-layer	Tabletop or controlled attack graph exercise with independent break points

Pass, partial pass and fail

• Pass: the expected control prevents or contains the action, produces attributable telemetry and supports recovery within the objective.
• Partial pass: one element succeeds but detection, attribution, containment or recovery is weak.
• Fail: the threat path succeeds beyond the approved condition, or the expected control does not operate.
• Not tested: no valid conclusion about operating effectiveness.

Partial passes and bypasses should become findings, not be rounded up to a pass.

Control design

For each threat, identify:

• Preventive controls.
• Detective controls.
• Response controls.
• Recovery controls.
• Human decision gates.
• Independent controls at different trust boundaries.
• Shared dependencies and common-mode failures.

Good treatment reduces one or more of:

• Exposure.
• Attacker access.
• Available privilege.
• Success probability.
• Time undetected.
• Blast radius.
• Persistence.
• Irreversibility.
• Recovery time.

Treatment record

A complete treatment entry should include:

• Threat ID and system-specific scenario.
• Current inherent or residual score.
• Chosen treatment: avoid, reduce, transfer, accept or monitor.
• Planned controls and expected risk reduction.
• Accountable owner.
• Evidence owner.
• Target date.
• Dependencies.
• Validation test.
• Residual-risk target.
• Risk-acceptance authority and expiry, if accepted.
• Monitoring signals and cadence.
• Reassessment triggers.

Findings and escalation

Raise a finding when:

• An applicable threat lacks a credible control.
• Required evidence is missing or rejected.
• A test fails or partially passes.
• Control ownership is unclear.
• A High or Critical risk lacks time-bound treatment.
• A cross-layer path has no independent break point.
• Monitoring cannot detect the attack or control failure.
• Risk acceptance lacks authority, rationale or expiry.

Critical exposure should normally trigger containment or pause. High exposure should normally be treated before broader deployment unless explicitly accepted.

Monitoring design

Define both leading and lagging indicators:

• Attempts and precursor behaviour.
• Control denials and bypass attempts.
• Integrity or provenance changes.
• Privilege, model, data, tool and dependency changes.
• Drift in attack success or false-negative rates.
• Unusual cost, latency, query or resource patterns.
• Cross-layer identity and event correlation.
• Findings ageing and target-date breaches.
• Mean time to detect, contain, revoke and recover.

Monitoring must be protected against the Layer 5 threats that can corrupt or remove it.

Closing a threat treatment

Do not close treatment merely because a control was deployed. Require:

1. Implementation evidence.
2. Successful bounded validation.
3. Resolution or acceptance of exceptions.
4. Updated residual score.
5. Named monitoring and review owner.
6. Confirmed reassessment trigger.
7. Human risk decision.