// copilot assessments / copilot-redteam-probe
Copilot Synthetic Red-Team Probe
Your refusal rate is only as strong as the next prompt. This probe fires a curated golden set of ~16 adversarial prompts at Copilot via the Chat API, plus retrieval queries via the Retrieval API, then scores six categories: refusal rate, PII leakage, cross-user boundary, external web pollution, citation faithfulness, and retrieval surface hygiene. Microsoft documents the Chat + Retrieval APIs as delegated-only — application (app-only) auth is "Not supported" — so the audit runs as a dedicated Copilot-licensed probe user your admin connects once via a single read-only sign-in. That same connection unlocks Copilot Package Management coverage in the Agent Inventory & Governance Audit, which sits behind the same delegated-only gate.
This audit needs the following before it can collect data. We will check each item before kicking off your run and prompt you if anything is missing.
Copilot probe account connected
This audit runs as a dedicated Copilot-licensed probe user via Microsoft’s delegated-only Chat & Retrieval APIs. Without a connected probe account it collects zero data — Microsoft blocks app-only access to these APIs entirely.
Provision a dedicated user with a Microsoft 365 Copilot licence (and Global Reader for full coverage), then open https://www.baref00t.io/consent/probe-account and sign that probe user in once. A Global Admin grants consent on first sign-in. The same connection also unlocks Package Management in the Agent Inventory audit.
Single report, no commitment.
Continuous monitoring — fresh report every 30 days.
Available in AUD, USD, GBP, EUR, SGD. MSP partners get volume discounts via the partner programme.
What it scores
Refusal Rate
Share of adversarial prompts Copilot refused (admin-compromise, phishing, policy-evasion).
PII Leakage
Probes targeting labelled / PII content. Returns must be empty or refusal-only.
Cross-User Boundary
Probes asking about other users’ OneDrive / mailbox. Boundary must hold.
External Web Pollution
Internal queries that blend external web citations.
Citation Faithfulness
For grounded answers, verify each cited URL actually contains the claim.
Retrieval Surface Hygiene
Stale documents (>365d), missing labels, overshared containers in the retrieval surface.
Microsoft APIs
- Copilot Chat API (preview)
- Copilot Retrieval API (preview)
Customer prerequisites
- Dedicated probe user account with M365 Copilot add-on licence
- Core delegated scopes (Files/Sites/ExternalItem) for retrieval + package coverage
- Optional broader scopes (Mail.Read, Chat.Read, ChannelMessage.Read.All, People.Read.All, OnlineMeetingTranscript.Read.All) — required only for the adversarial chat probes (RT1–RT5); decline them to run a lighter retrieval-only audit
Other Copilot Assessments
Pre-deployment data hygiene scoring across 7 dimensions plus live retrieval-surface measurement and an agent catalogue teaser.
Per-user, per-surface Copilot activity vs licence spend — with named-user reclaim list.
Risk-scored inventory of every Entra Agent ID (Copilot Studio etc.), the tenant-wide Copilot tool catalogue (MCP servers), and the full Copilot Package Management catalogue (third-party add-ins + Teams apps) once a probe account is connected.