{"data":{"id":"SP-027","slug":"secure-llm-usage","title":"Secure LLM Usage","description":"Security architecture for integrating large language models (LLMs) into enterprise environments. Covers prompt injection and jailbreaking defences, output validation, provider security, privacy inference attacks on fine-tuned models, model extraction, and LLM interaction audit. For AI agents and agentic frameworks, see SP-047. For enterprise AI governance, see SP-045.","url":"https://www.opensecurityarchitecture.org/patterns/sp-027","metadata":{"release":"26.02","classification":"Application & Operations","status":"active","type":"pattern","datePublished":"2026-02-06","dateModified":"2026-02-22","authors":["Aurelius","Vitruvius"],"reviewers":[],"provenance":"Refocused from Secure AI Integration (covering agents and LLMs) to Secure LLM Usage (model-layer risks only). Agent and agentic framework security has been consolidated in SP-047. LLM-specific threats added: jailbreaking, model extraction, and privacy inference attacks on fine-tuned models. Content derived from OWASP LLM Top 10, NIST AI 100-2, MITRE ATLAS, and direct operational experience of LLM integration security."},"diagram":{"svg":"/images/sp-027-secure-ai-integration.svg","png":""},"content":{"description":"Large language models introduce fundamentally new security risks compared to traditional software components. A conventional application receives structured inputs, processes them through deterministic logic, and returns predictable outputs. An LLM receives natural language that cannot be formally validated, reasons through processes that are opaque to the caller, and produces outputs that are probabilistic and non-deterministic. These properties — not the power of the model, but the nature of its inputs and outputs — define the LLM security problem.\n\nThis pattern covers the model layer: the security risks that exist whenever an organisation uses an LLM, regardless of whether it is accessed via a chat interface, called through an API, used as the foundation for an agent, or fine-tuned on enterprise data. The threats addressed here — prompt injection, jailbreaking, hallucination, provider-side data exposure, privacy inference attacks on training data, and model extraction — are properties of the LLM interaction itself.\n\nScope: this pattern addresses the LLM model layer. For security architecture of AI agents that take autonomous actions, invoke tools, and orchestrate sub-agents, see SP-047 (Secure Agentic AI Frameworks). For the enterprise governance management system — training data governance, bias, transparency, and impact assessment — see SP-045 (AI Governance and Responsible AI). Reading order for enterprise AI security: SP-027 (LLM layer) → SP-047 (agent/framework layer) → SP-045 (governance layer).","keyControlAreas":["Data Classification for LLM Contexts (AC-04, PT-02, PT-03, SC-04): Every piece of data entering an LLM prompt or conversation context must be classified. Context windows create implicit data aggregation risk: individually innocuous data points may become sensitive in combination. Implement tiered context policies: public data flows freely, internal data requires justification, confidential data requires explicit approval, and restricted data (credentials, PII, regulated data) must never enter a context window without technical controls ensuring it cannot be persisted or exfiltrated. Information remnance (SC-04) is a specific risk: data shared with an LLM provider may persist in provider-side logs, training pipelines, or session caches beyond the user's awareness. PT-02 and PT-03 ensure lawful authority exists before personal data enters a context window, satisfying GDPR and equivalent privacy law obligations.","Prompt Security (SI-10, SI-03, SC-07): Prompt injection is the SQL injection of the AI era — an attacker who controls input content can subvert the LLM's intended behaviour. Direct injection attempts to override system instructions via user-supplied text. Indirect injection embeds malicious instructions in content the LLM fetches or processes: tool outputs, web pages, documents, PDFs, even image metadata. Defence requires strict separation between system instructions and user or retrieved content; output validation before the LLM's parsed data drives any action; content sanitisation for external content; and boundary protection (SC-07) between the LLM's context and untrusted content sources. System prompts should be treated as security policy: protect them from extraction and override, and never assume the LLM will enforce their constraints without application-level controls.","Jailbreaking and Safety Feature Bypass (SI-10, AC-03, SA-11): Jailbreaking refers to adversarial prompts that circumvent the LLM's built-in safety constraints — role-play scenarios ('pretend you are an AI without restrictions'), hypothetical framing, encoding tricks, and many-shot prompting that gradually shifts the model's response pattern. In enterprise contexts, safety features protect both ethical use and business policy enforcement: a safety bypass that produces harmful content also indicates a policy enforcement failure. Controls: red-team LLM safety boundaries as part of pre-deployment security testing (SA-11) to characterise the model's actual safety envelope; deploy application-layer guardrails that complement but do not solely rely on model-level safety mechanisms; monitor for anomalous prompt patterns indicative of jailbreak attempts; treat the model's built-in safety constraints as a baseline that must be reinforced by application-layer access enforcement (AC-03), not as a complete solution.","Output Validation and Hallucination Risk (SI-10, CA-07, SA-11): LLMs hallucinate — they produce plausible-sounding but factually incorrect outputs with apparent confidence. The security implications are direct: an AI-generated firewall rule that misinterprets the requirement, a compliance check citing a non-existent regulation, a vulnerability report fabricating CVE details, or infrastructure-as-code with a subtle misconfiguration. Controls: define output verification requirements scaled to the decision stakes (lower for informational output, mandatory human review for configuration changes, policy decisions, or security findings); never use LLM output as the sole authority for security decisions; implement continuous monitoring (CA-07) of LLM output quality with anomaly detection for output pattern shifts that may indicate model degradation or adversarial manipulation; include hallucination boundary testing in developer security testing (SA-11) for any LLM-integrated application.","Provider Security and Data Residency (SA-09, SA-04, PT-02): LLM providers are critical third-party dependencies. Assess provider security posture: SOC 2 Type II reports, data processing agreements (DPAs), breach notification commitments, and subprocessor chains. Data residency requirements must be met through regional API endpoint selection where providers offer it. Request explicit opt-out from training on your prompts and completions; seek zero data retention agreements for sensitive workloads. For fine-tuning and RAG scenarios, confirm that uploaded data is not used to improve the base model without consent. Evaluate provider concentration risk: if enterprise security tooling, code review, and infrastructure management all depend on a single model provider, a provider outage or compromise has cascading impact. SA-04 acquisition controls apply at the point of provider onboarding and annual review.","Privacy Inference Attacks on Fine-Tuned Models (PT-02, PT-03, RA-08): When an LLM is fine-tuned on enterprise data, two privacy attacks become relevant. Membership inference: an adversary queries the model in patterns designed to probabilistically determine whether a specific individual's data was in the training set — creating GDPR liability if data was processed without lawful basis or consent. Model inversion: systematic querying to reconstruct examples from the training data, potentially exposing PII, source code, or confidential content. Enterprise relevance: these attacks apply whenever fine-tuning uses customer records, employee data, proprietary documents, or any personally identifiable information. Mitigations: differential privacy techniques in fine-tuning pipelines, strict data minimisation before fine-tuning (only use data necessary and with clear lawful basis), careful review of fine-tuning datasets against these attack vectors. RA-08 (Privacy Impact Assessments) must be conducted before fine-tuning on any personal data, with membership inference risk explicitly identified as a privacy harm vector.","Model Extraction and System Prompt Protection (SR-02, SA-09, AU-02): Model extraction is systematic API querying designed to approximate or replicate a fine-tuned proprietary model — stealing the intellectual property and competitive advantage embedded in the fine-tuning investment. For organisations with proprietary fine-tuned models, rate limiting, query pattern monitoring (AU-02), and API key scoping are the primary defences. System prompt extraction — inferring or eliciting the system prompt through adversarial queries — leaks business logic, constraint definitions, and the organisation's AI configuration. Treat system prompts as security policy documents: classify them at the same level as the business logic they encode, never include them in logs or error messages, and implement application-layer controls to prevent their disclosure. SR-02 (Supply Chain Risk Management) covers the model itself as a supply chain dependency — pin model versions and test security-relevant behaviours after provider updates.","Audit of LLM Interactions (AU-02, AU-03, AU-06, CA-07): Log every LLM interaction with sufficient detail for forensic reconstruction: the initiating identity, the model and version used, the system prompt identifier (not content — the reference, to avoid logging sensitive prompts in plaintext), a hash of the user prompt, the response or a hash of it, and the timestamp. For tool-augmented or agentic use, also log what tool invocations were triggered by LLM outputs. Enable anomaly detection (CA-07) for interaction patterns: unusual prompt volumes, queries to topics outside normal usage, or output patterns inconsistent with the model's typical behaviour. Audit records (AU-03) must support attribution of LLM-assisted decisions to the human who initiated the session, maintaining accountability even when the model does the drafting. Retain audit logs independently of the LLM provider."],"assumptions":"Organisations access LLMs via cloud API services (Anthropic, OpenAI, Google, Azure OpenAI) over TLS-protected network connections. The pattern covers any use of LLMs: direct API integration, embedded enterprise products (Microsoft 365 Copilot, GitHub Copilot, Gemini for Workspace), and fine-tuned models on enterprise data. Some organisations additionally deploy open-weight models (Llama, Mistral) self-hosted on-premises, where provider-side risks shift but model-layer risks remain. The security risks addressed here apply at the model interaction layer regardless of whether the LLM underlies a chat interface, an application, or an agent. For security of AI agents that take autonomous actions and invoke tools, see SP-047 (Secure Agentic AI Frameworks). Model capabilities and the associated attack surface are advancing rapidly — this pattern should be reviewed quarterly.","typicalChallenges":"Prompt injection defence maturity remains at the equivalent of early SQL injection — the vulnerability class is well-understood but reliable systematic defences do not exist. No framework equivalent to parameterised queries has emerged for prompt injection, and mitigations (input/output filtering, canary tokens, instruction hierarchy) reduce risk but can be bypassed by determined attackers. Provider-side data handling is contractually constrained but practically opaque: what providers log, retain, and potentially use for training is difficult to audit independently. Privacy inference attacks on fine-tuned models (membership inference, model inversion) require ML engineering competence to assess and mitigate that most security teams have not yet developed. Output validation is inherently difficult — there is no deterministic way to verify LLM output correctness, and human review of high-volume LLM output at scale is impractical. Jailbreak techniques evolve continuously as researchers and attackers discover new ways to probe model safety constraints — defences effective today may fail after model updates. Provider lock-in grows as fine-tuned models and proprietary system prompts become tied to specific provider APIs.","indications":"Organisation uses LLMs via API or embedded enterprise products (Microsoft 365 Copilot, GitHub Copilot, Gemini for Workspace) for business-sensitive tasks including data processing, analysis, or decision support. LLMs are integrated into enterprise applications as a component. Organisation has or is planning to fine-tune LLMs on enterprise data including customer records, employee data, or proprietary content. LLMs are used in security tooling: AI-assisted SIEM analysis, AI-augmented code review, AI-based threat intelligence. Employees use consumer LLM tools for work purposes, introducing enterprise data into provider-side systems.","contraIndications":"Organisation has no AI integration plans and no employee exposure to LLM tools in any operational capacity. AI usage is strictly limited to isolated, air-gapped, non-sensitive tasks with no enterprise data input. Note: this contra-indication is increasingly rare — even organisations that do not formally deploy LLMs may have employees using AI tools informally, and many enterprise SaaS products now embed LLMs by default.","threatResistance":"Prompt injection — direct (system prompt override via user input) and indirect (malicious instructions embedded in fetched content, documents, or tool outputs) — addressed through strict instruction/data separation, output validation before LLM-parsed data drives any action, and boundary protection preventing untrusted content from reaching the system prompt context. Jailbreaking and safety feature bypass defended through adversarial pre-deployment testing that characterises the model's actual safety envelope, application-layer guardrails that complement model-level safety, and anomaly monitoring for adversarial prompt patterns. Hallucination-driven misconfiguration mitigated by output verification requirements scaled to decision stakes and mandatory human review for security-critical outputs — never using LLM output as the sole authority for security decisions. Provider-side data exposure addressed through DPAs, training opt-out agreements, zero data retention where required, and data classification controls governing what enters context windows. Privacy inference attacks on fine-tuned models (membership inference, model inversion) mitigated through privacy impact assessments before fine-tuning, differential privacy in training pipelines, and strict data minimisation in fine-tuning datasets. Model extraction and system prompt leakage addressed through rate limiting, query pattern monitoring, and treating system prompts as classified security policy documents. AI-amplified social engineering addressed through awareness training calibrated to the scale and sophistication of AI-generated phishing and pretexting."},"examples":{"Chat and completion APIs":["OpenAI GPT-4o and Claude API called from enterprise applications for text analysis, summarisation, and structured data extraction","Google Gemini API for document processing pipelines with enterprise data residency requirements","Azure OpenAI Service for regulated-industry deployments with data residency, DPA, and zero-retention configurations"],"Embedded LLMs in enterprise productivity tools":["Microsoft 365 Copilot integrated across Word, Excel, Outlook, and Teams — where data classification controls govern what the LLM context can access","Google Gemini for Workspace embedded in Docs, Gmail, and Meet","GitHub Copilot for code assistance — where system prompt protection and output validation apply to AI-suggested code"],"Fine-tuned models on enterprise data":["Domain-adapted models fine-tuned on proprietary legal, financial, or medical corpora — requiring privacy impact assessment and membership inference risk analysis","Custom LLMs trained on enterprise customer support history — where GDPR lawful basis and data minimisation must be established before training data selection","Code generation models fine-tuned on internal codebases — where model extraction risk applies to the embedded IP"],"LLMs in security tooling":["AI-assisted SIEM analysis: LLMs summarising alert clusters and suggesting triage priorities — where output validation prevents hallucinated threat escalations from driving incorrect responses","AI-augmented code review tools identifying security anti-patterns — where hallucination testing is part of tool validation","LLM-based threat intelligence: summarisation, indicator extraction, and report generation — where provider data retention agreements cover potentially sensitive intelligence content"],"Developing Areas":["Jailbreak research and defence maturity: jailbreaking techniques are catalogued by researchers (role-play, many-shot, encoding, context manipulation) but defences remain reactive rather than systematic. The community is debating whether model-level training mitigations or application-layer policy engines are the more durable solution. No consensus has formed on standardised adversarial testing scope or pass/fail criteria for LLM safety assessments.","Privacy inference attack tooling: membership inference and model inversion attack toolkits (LiRA, MI-BENCH) are maturing as research tools but are not yet routinely used in enterprise security assessments. The gap between academic attack capability and enterprise defensive awareness is significant — organisations fine-tuning on personal data may be exposed to GDPR liability risks they have not assessed.","EU AI Act Article 50 (synthetic content disclosure): the obligation to disclose AI involvement in content generation creates downstream security implications for phishing detection, forensic attribution, and content authenticity verification. The technical implementation of disclosure (watermarking, provenance metadata) is unresolved and the enforcement ecosystem is not yet mature.","NIST AI 100-2 and adversarial ML standardisation: NIST's Adversarial Machine Learning taxonomy provides the most structured framework for LLM attack classification, but enterprise implementation guidance translating the taxonomy into specific testing requirements and control baselines is still developing."]},"references":[{"title":"OWASP Top 10 for Large Language Model Applications","url":"https://owasp.org/www-project-top-10-for-large-language-model-applications/","note":"Comprehensive taxonomy of LLM-specific vulnerabilities including prompt injection, insecure output handling, training data poisoning, model denial of service, and supply chain vulnerabilities. Essential baseline for LLM application security assessment."},{"title":"NIST AI 100-1: AI Risk Management Framework","url":"https://www.nist.gov/itl/ai-risk-management-framework","note":"Voluntary framework for managing AI risks across the AI lifecycle. Maps well to enterprise governance structures and provides the Govern-Map-Measure-Manage functions as an organising structure for LLM risk management."},{"title":"NIST AI 100-2: Adversarial Machine Learning — A Taxonomy and Terminology of Attacks and Mitigations","url":"https://csrc.nist.gov/pubs/ai/100/2/e2025/final","note":"NIST's structured taxonomy of adversarial attacks against AI systems including evasion, poisoning, privacy attacks (membership inference, model inversion), and abuse attacks. Provides the authoritative classification framework for LLM-specific threats including those addressed in this pattern."},{"title":"MITRE ATLAS — Adversarial Threat Landscape for AI Systems","url":"https://atlas.mitre.org/","note":"Knowledge base of adversary tactics and techniques against AI systems, modelled on ATT&CK. Provides structured threat intelligence for AI-specific attacks including prompt injection, model extraction, and membership inference."},{"title":"ISO/IEC 42001:2023 — AI Management System","url":"https://www.iso.org/standard/81230.html","note":"International standard for establishing, implementing, and improving an AI management system. Provides the certifiable governance framework that complements this pattern's technical controls."},{"title":"NCSC Guidelines for Secure AI System Development","url":"https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development","note":"UK National Cyber Security Centre guidance covering secure design, development, deployment, and maintenance of AI systems. Co-signed by agencies from 18 countries. Addresses supply chain, monitoring, and secure-by-default principles for LLM integrations."},{"title":"EU General Data Protection Regulation — Article 22 (Automated Decision-Making)","url":"https://gdpr-info.eu/art-22-gdpr/","note":"Article 22 establishes rights around automated individual decision-making. Relevant to LLM usage where model outputs influence decisions about individuals. Also the legal basis for privacy impact assessment requirements when fine-tuning on personal data."},{"title":"EU AI Act (Regulation 2024/1689)","url":"https://artificialintelligenceact.eu/","note":"European regulatory framework with risk-based classification of AI systems. Article 50 establishes synthetic content disclosure obligations. High-risk AI systems involving LLMs face conformity assessments and transparency requirements."},{"title":"NIST SP 800-218A — Secure Software Development Practices for Generative AI","url":"https://csrc.nist.gov/pubs/sp/800/218/a/final","note":"Extension of the SSDF specifically addressing generative AI, covering training data integrity, model validation, and AI-specific development practices."}],"relatedPatterns":["SP-047","SP-045","SP-013","SP-025"],"relatedPatternNames":["Secure Agentic AI Frameworks","AI Governance and Responsible AI","Data Security","Advanced Monitoring and Detection"],"threats":[{"id":"T-AI-001","name":"Direct Prompt Injection","mitigatedBy":["SI-10","SC-07","AC-03"]},{"id":"T-AI-002","name":"Indirect Prompt Injection (via tool outputs, fetched content)","mitigatedBy":["SI-10","SI-03","SC-07","AC-04"]},{"id":"T-AI-003","name":"Data Exfiltration via Context Window","mitigatedBy":["AC-04","SC-07","PT-02","AU-02"]},{"id":"T-AI-004","name":"Credential Exposure in Conversation Context","mitigatedBy":["IA-05","SC-12","SC-04","AU-03"]},{"id":"T-AI-010","name":"AI-Amplified Social Engineering","mitigatedBy":["AT-02","AT-03","SI-04","IR-04"]},{"id":"T-AI-011","name":"Hallucination Leading to Security Misconfiguration","mitigatedBy":["SI-10","CA-02","SA-11","CA-07"]},{"id":"T-AI-012","name":"Provider-Side Data Retention and Logging","mitigatedBy":["SA-09","PT-02","PT-03","PS-07"]},{"id":"T-AI-013","name":"Jailbreaking / Safety Feature Bypass","mitigatedBy":["SI-10","SA-11","AC-03"]},{"id":"T-AI-014","name":"Model Extraction via Systematic API Querying","mitigatedBy":["SR-02","SA-09","AU-02"]},{"id":"T-AI-015","name":"Membership Inference Attack on Fine-Tuned Model","mitigatedBy":["PT-02","RA-08","PT-03"]}],"controls":[{"id":"AC-03","name":"Access Enforcement","family":"AC","emphasis":"critical"},{"id":"AC-04","name":"Information Flow Enforcement","family":"AC","emphasis":"critical"},{"id":"AT-02","name":"Literacy Training and Awareness","family":"AT","emphasis":"important"},{"id":"AT-03","name":"Role-based Training","family":"AT","emphasis":"important"},{"id":"AU-02","name":"Event Logging","family":"AU","emphasis":"critical"},{"id":"AU-03","name":"Content of Audit Records","family":"AU","emphasis":"critical"},{"id":"AU-06","name":"Audit Record Review, Analysis, and Reporting","family":"AU","emphasis":"important"},{"id":"CA-02","name":"Control Assessments","family":"CA","emphasis":"important"},{"id":"CA-07","name":"Continuous Monitoring","family":"CA","emphasis":"critical"},{"id":"IA-05","name":"Authenticator Management","family":"IA","emphasis":"important"},{"id":"IR-04","name":"Incident Handling","family":"IR","emphasis":"important"},{"id":"PS-07","name":"External Personnel Security","family":"PS","emphasis":"important"},{"id":"PT-02","name":"Authority to Process Personally Identifiable Information","family":"PT","emphasis":"important"},{"id":"PT-03","name":"Personally Identifiable Information Processing Purposes","family":"PT","emphasis":"important"},{"id":"RA-08","name":"Privacy Impact Assessments","family":"RA","emphasis":"critical"},{"id":"SA-04","name":"Acquisition Process","family":"SA","emphasis":"important"},{"id":"SA-08","name":"Security and Privacy Engineering Principles","family":"SA","emphasis":"important"},{"id":"SA-09","name":"External System Services","family":"SA","emphasis":"critical"},{"id":"SA-11","name":"Developer Testing and Evaluation","family":"SA","emphasis":"important"},{"id":"SC-04","name":"Information in Shared System Resources","family":"SC","emphasis":"critical"},{"id":"SC-07","name":"Boundary Protection","family":"SC","emphasis":"critical"},{"id":"SC-12","name":"Cryptographic Key Establishment and Management","family":"SC","emphasis":"important"},{"id":"SI-03","name":"Malicious Code Protection","family":"SI","emphasis":"important"},{"id":"SI-04","name":"System Monitoring","family":"SI","emphasis":"important"},{"id":"SI-10","name":"Information Input Validation","family":"SI","emphasis":"critical"},{"id":"SR-02","name":"Supply Chain Risk Management Plan","family":"SR","emphasis":"critical"}],"controlFamilySummary":{"AC":2,"AT":2,"AU":3,"CA":2,"IA":1,"IR":1,"PS":1,"PT":2,"RA":1,"SA":4,"SC":3,"SI":3,"SR":1}}}