Microsoft Security · AI Threat Intelligence

AI Attack Vectors
& Microsoft Defenses

A comprehensive map of threats targeting AI environments — with detection and response coverage across the Microsoft security stack.

11
Threat Vectors
14
MS Solutions
3
Actor Types
E5
Full Coverage SKU
🪤

Prompt Injection

// Direct & Indirect · Jailbreak · Agent Hijack

External Internal
▾
What It Is
An attacker embeds malicious instructions inside user input or external content (websites, documents, emails) that the AI model processes. The goal is to hijack the model's behavior — bypassing safety rules, leaking data, or triggering unauthorized actions.

Direct injection — User directly crafts malicious prompts.
Indirect injection — Malicious instructions hidden inside content the AI reads (e.g., a webpage summarized by a Copilot agent).
Threat Actor Profile
🌐 ExternalJailbreaking public-facing AI apps, manipulating chatbots, injecting prompts via RAG data sources.
👤 InternalEmployees bypassing enterprise AI guardrails, manipulating AI agents to exfiltrate corporate data.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
Azure AI Content SafetyPrompt Shields detect & block direct and indirect injection in real time before reaching the modelAzure AI Services
Defender for Cloud — AI Threat ProtectionMonitors Azure OpenAI & Azure ML for injection patterns; generates security alertsDefender P2
Microsoft SentinelIngests AI alerts; UEBA correlates anomalous query patterns; custom KQL detection rulesSentinel
Microsoft Purview DLPInspects prompts submitted to Copilot/AI tools for sensitive data being sent outboundM365 E5 Compliance
Entra ID Conditional AccessRestricts who can access AI-powered apps and from which device/network postureEntra P1/P2
☣️

Data Poisoning

// Training Data Corruption · Backdoor Injection · RAG Manipulation

External Internal
▾
What It Is
An attacker corrupts the training data (or fine-tuning data) fed into an AI model so it learns incorrect behaviors — creating backdoors, biased outputs, or manipulated classifications. This is a pre-deployment attack with long-lasting production consequences.
Threat Actor Profile
🌐 ExternalPoisoning publicly scraped datasets, supply chain attacks on open-source data, RAG database manipulation.
👤 InternalMalicious data engineers inserting backdoors into labeled datasets or fine-tuning pipelines.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
Purview — Sensitivity LabelsLabels and classifies training datasets; enforces access controls preventing unauthorized modificationM365 E3/E5
Purview IRMDetects suspicious bulk data modification by insiders in data lake / ML storage environmentsM365 E5 Compliance
Defender for Cloud — DevOps SecurityScans ML pipelines (GitHub Actions, ADO) for unauthorized changes; monitors Azure ML data storesDefender for Cloud
Microsoft Sentinel + UEBADetects anomalous write/modify patterns on training datasets in Azure Storage, ADLS, FabricSentinel
Entra ID PIMJust-in-time access to training data pipelines; limits who can write to ML data storesEntra P2
Purview Data MapTracks data lineage — where training data came from and whether it was alteredPurview
🕵️

Model Theft

// Model Extraction · API Harvesting · IP Theft

External Internal
▾
What It Is
An attacker queries a model thousands/millions of times via its API to reconstruct a functional "shadow model" without authorization. This allows IP theft, internal vulnerability analysis, and further targeted attacks — without ever accessing training data.
Threat Actor Profile
🌐 ExternalCompetitors or nation-state actors systematically harvesting model behavior via public APIs.
👤 InternalEmployees exfiltrating model weights or cloning behavior through authorized API access.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
MCAS / Defender for Cloud AppsDetects abnormal API call volume/patterns to AI endpoints; triggers anomaly alertsDefender for Cloud Apps
Azure API ManagementRate limiting, IP throttling, subscription key revocation — chokes extraction at the API layerAzure APIM
Microsoft SentinelCustom detection on Azure OpenAI diagnostic logs for high-volume systematic queriesSentinel
Entra ID + PIMRestricts access to model endpoints and Azure ML model registry; JIT for sensitive rolesEntra P2
Purview IRMMonitors insider behavioral signals for employees making large volumes of model queriesM365 E5 Compliance
Defender for Cloud — AI Security PostureIdentifies publicly exposed AI model endpoints without authenticationDefender for Cloud
🔓

Privacy Leakage

// Memorization · PII Exposure · GDPR/HIPAA Risk

External Internal
▾
What It Is
AI models — especially LLMs — can inadvertently memorize and reproduce fragments of their training data, including PII, credentials, proprietary documents, or medical records. This is a post-deployment risk that can result in GDPR/HIPAA violations and reputational damage.
Threat Actor Profile
🌐 ExternalCrafting adversarial prompts to extract memorized sensitive data from public-facing models.
👤 InternalEmployees using Copilot to inadvertently surface sensitive HR/legal/financial data they shouldn't access.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
Purview — AI HubVisibility into what sensitive data users submit to and receive from AI apps; detects privacy leakage in outputsM365 E5 Compliance
Purview DLPDetects and blocks responses containing SSNs, credit cards, health info from AI output channelsM365 E5 Compliance
Purview Sensitivity LabelsLabels flow into Copilot — won't surface Highly Confidential data to unauthorized usersM365 E3/E5
Azure AI Content SafetyOutput filtering — strips or blocks PII patterns from model responses at the API layerAzure AI Services
Entra ID — Access ReviewsEnsures only authorized users have access to AI apps that can surface sensitive dataEntra P2
Microsoft PrivaPrivacy risk management; identifies over-exposure of personal data in AI training sets or outputsMicrosoft Priva
📉

Model Drift

// Distribution Shift · Performance Degradation · Governance Gap

External Internal
▾
What It Is
Over time, the statistical distribution of real-world data diverges from training data, causing model predictions to degrade or become biased. While not always a direct attack, adversaries can deliberately trigger drift by feeding adversarial inputs continuously — a slow-burn attack. Shadow AI models running without governance are especially vulnerable.
Threat Actor Profile
🌐 ExternalSlowly poisoning production inference data to shift model behavior over time (concept drift injection).
👤 InternalNegligent ML ops ignoring drift signals; shadow AI models running without monitoring or governance.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
Azure Machine Learning — Data Drift MonitorTracks statistical drift between baseline and production datasets; triggers alerts when thresholds exceededAzure ML
Defender for Cloud — AI Security PostureFlags unmonitored AI models and pipelines lacking observability controlsDefender for Cloud
Microsoft SentinelIngests Azure ML monitoring logs; custom workbooks to visualize model performance degradation over timeSentinel
Purview — AI HubGovernance visibility into all AI models deployed in the tenant, including unmanaged/shadow modelsM365 E5 Compliance
Azure Monitor + Log AnalyticsCaptures inference telemetry; anomaly detection on output distribution changesAzure
🎭

Adversarial Examples

// Input Perturbation · Evasion · Misclassification

External
▾
What It Is
Specially crafted inputs — slightly perturbed images, audio, or text — cause a model to misclassify with high confidence. A stop sign with stickers fools an autonomous vehicle. A slightly modified PDF evades AI-based malware detection. These attacks exploit the mathematical fragility of neural networks.
Threat Actor Profile
🌐 ExternalAttackers crafting adversarial inputs to evade AI-powered security controls (malware detection, fraud scoring, content moderation).
Microsoft Defense Coverage
SolutionHow It HelpsLicense
Azure AI Content SafetyInput robustness checks; adversarial input filtering before reaching modelsAzure AI Services
Defender for CloudAI Threat Protection flags unusual inference patterns that may indicate adversarial probingDefender P2
Azure ML — Responsible AI DashboardAdversarial robustness evaluation during model development and testing phasesAzure ML
🔬

Model Inversion

// Training Data Reconstruction · Output Analysis

External
▾
What It Is
By querying the model repeatedly and analyzing confidence scores/probabilities in its outputs, an attacker reconstructs approximate samples of the original training data — effectively reversing the training process to recover sensitive individual records.
Threat Actor Profile
🌐 ExternalResearchers or attackers exploiting model confidence outputs to reconstruct faces, medical records, or proprietary data from API responses.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
Azure APIMRate limiting and output sanitization — suppresses raw probability scores from being exposed via API responsesAzure APIM
MCAS / Defender for Cloud AppsAnomaly detection on API access patterns consistent with inversion attack behaviorDefender for Cloud Apps
Microsoft PrivaPrivacy risk management; differential privacy tooling integration in Azure ML training pipelinesMicrosoft Priva
🔍

Membership Inference

// GDPR Right-to-Erasure · Training Data Detection

External
▾
What It Is
An attacker queries the model to determine whether a specific individual's data was used in training. This has direct GDPR implications (right to erasure — was my data used?). By comparing model confidence on known vs. unknown records, attackers can confirm membership in the training set.
Threat Actor Profile
🌐 ExternalIndividuals or adversaries confirming whether specific personal data appeared in a model's training set — used for legal pressure, extortion, or targeted privacy attacks.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
Microsoft PrivaPrivacy risk management and GDPR compliance tooling; data subject request managementMicrosoft Priva
Purview DLPBlocks model outputs that could confirm membership in sensitive data categoriesM365 E5 Compliance
Azure ML — Differential PrivacyApplies statistical noise during training to make membership inference mathematically infeasibleAzure ML
🔗

AI Supply Chain Attack

// Compromised Models · Malicious Libraries · Pipeline Tampering

External Internal
▾
What It Is
Compromising a pre-trained model, open-source ML library (e.g., a malicious Hugging Face model), or fine-tuning dataset before it enters the organization's environment. This is the SolarWinds of AI — trust in third-party components weaponized against the consumer.
Threat Actor Profile
🌐 ExternalNation-state actors or criminals poisoning open-source models, libraries, or datasets on public repositories.
👤 InternalNegligent intake of unverified pre-trained models or datasets without provenance validation.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
Defender for DevOpsScans ML pipeline code and dependencies for tampering; integrates with GitHub/ADODefender for Cloud
Defender for Cloud — Container ScanningScans model containers and base images for malicious components before deploymentDefender for Cloud
GitHub Advanced SecurityDependency scanning and secret detection in ML pipeline code repositoriesGHAS
Purview Data Map / LineageTracks provenance of models and datasets — where they came from and whether they were alteredPurview
🤖

Insecure Agentic / Plugin Execution

// LLM Agent Hijack · Tool Abuse · Autonomous Action

External Internal
▾
What It Is
AI agents with tool-use capabilities (web browsing, code execution, email sending) can be hijacked via indirect prompt injection to execute malicious actions — sending phishing emails, deleting files, or exfiltrating data — all under the guise of the authorized user. As Copilot agents proliferate, this surface area grows dramatically.
Threat Actor Profile
🌐 ExternalEmbedding injection payloads in websites or emails that AI agents browse, causing autonomous malicious actions.
👤 InternalEmployees crafting prompts to make AI agents perform unauthorized actions on corporate systems.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
Microsoft Copilot for SecurityDetection and investigation of agent-based attack chains across Defender XDR signalsCopilot for Security
Purview — AI HubAgent activity visibility — tracks what actions AI agents are taking on behalf of usersM365 E5 Compliance
Entra ID — OAuth Scope RestrictionRestricts OAuth permission scopes granted to AI plugins, limiting blast radius of hijacked agentsEntra P1/P2
MCAS — Session ControlsReal-time session monitoring on agent-connected SaaS apps; blocks abnormal automated actionsDefender for Cloud Apps
Azure AI Content SafetyIndirect prompt injection detection specifically for agentic workflows and RAG pipelinesAzure AI Services
👤

Shadow AI / Unauthorized AI Usage

// Unsanctioned Tools · Data Governance Bypass · BYOAI

Internal
▾
What It Is
Employees using unauthorized AI tools (consumer ChatGPT, personal Claude, free Gemini, etc.) and pasting sensitive corporate data into them — bypassing all enterprise data governance, DLP, and compliance controls. Often well-intentioned but highly risky; the data leaves the tenant boundary entirely.
Threat Actor Profile
👤 Internal (Negligent / Insider)Employees, contractors, and partners using personal AI tools with corporate data — often without malicious intent but creating significant data loss and compliance exposure.
Microsoft Defense Coverage
SolutionHow It HelpsLicense
MCAS / Defender for Cloud AppsBlocks or audits access to unsanctioned AI SaaS apps; cloud app catalog flags AI tools by risk scoreDefender for Cloud Apps
Purview — AI HubShows data submitted to external AI services; detects sensitive data flowing outside approved channelsM365 E5 Compliance
Endpoint DLPBlocks copy-paste of sensitive data to non-approved applications and browsers on managed endpointsM365 E5 Compliance
Entra ID — Conditional AccessRestricts non-managed/non-compliant devices from accessing corporate data, limiting BYOAI on personal devicesEntra P1/P2
Purview IRM — Adaptive ProtectionElevates DLP restrictions for users flagged as high risk based on AI tool usage patternsM365 E5 Compliance

Microsoft Solution Coverage Map

Azure AI Content Safety
Prompt Shields (injection), output filtering, jailbreak detection, agentic pipeline protection
Microsoft Defender for Cloud
AI Security Posture Management, ML pipeline threats, exposed endpoint detection, DevOps scanning
Microsoft Purview — AI Hub
Privacy leakage, shadow AI governance, agent activity visibility, all AI interaction oversight
Microsoft Purview DLP
Sensitive data in prompts & responses, exfiltration via AI channels, Endpoint DLP for copy-paste
Purview IRM + Adaptive Protection
Insider threats across data poisoning, model theft, privacy abuse, shadow AI usage
Microsoft Sentinel
SIEM correlation, UEBA across all AI threat signals, custom KQL detections, drift workbooks
Microsoft Defender XDR
Lateral movement post-AI compromise, agent abuse detection, cross-signal correlation
Entra ID (CA + PIM + Identity Protection)
Unauthorized access to AI endpoints, JIT privilege for ML pipelines, risk-based step-up
MCAS / Defender for Cloud Apps
Shadow AI blocking, model extraction via API, OAuth plugin scope abuse, session controls
Azure Machine Learning
Data drift monitoring, Responsible AI dashboard, adversarial robustness, pipeline integrity
Microsoft Priva
Privacy risk, membership inference defense, GDPR compliance for AI, differential privacy
Azure API Management
Rate limiting against model extraction, API gateway controls, subscription key management
Purview Sensitivity Labels
Data classification flowing into Copilot, training dataset access control, MIP encryption
GitHub Advanced Security
Dependency scanning in ML pipelines, secret detection, supply chain vulnerability alerts
⚠️

Licensing Reality Check

Full AI security coverage requires M365 E5 Compliance (Purview AI Hub, IRM, Adaptive Protection, full DLP) + Entra ID P2 (risk-based CA, PIM, Identity Protection) + Defender for Cloud P2 (AI Threat Protection, DevOps Security). Organizations on M365 E3 have significant gaps, particularly in AI Hub visibility, Adaptive Protection integration, and insider risk detection for AI-specific behaviors.