AI Governance and Responsible AI

Key Control Areas

When to Use

Organisation deploys AI systems that make or support decisions affecting individuals (credit, employment, insurance, healthcare, law enforcement). EU AI Act applies to the organisation's AI systems. ISO 42001 certification is planned or required by customers. AI systems process personal data subject to GDPR or equivalent privacy regulation. Audit findings related to AI transparency, bias, or accountability. Board or executive requesting assurance that AI is governed responsibly. Customers or regulators asking how AI decisions are made and can be contested.

When NOT to Use

Organisation uses AI only for internal, non-consequential tasks (e.g., code completion, document formatting) with no impact on individuals. All AI usage is covered by SP-027's operational security controls and no regulatory AI governance requirements apply. Organisation has no current or planned AI deployments.

Typical Challenges

Defining 'fairness' is context-dependent and contested -- demographic parity may conflict with equalised odds, and different stakeholders may prefer different definitions. Training data provenance is difficult to establish for large-scale datasets, particularly foundation models trained on internet-scale data. Explainability techniques trade off fidelity with simplicity -- the most accurate explanation may not be the most understandable. AI impact assessments lack the methodological maturity of privacy impact assessments and often produce subjective conclusions. Responsible AI monitoring requires ML engineering capability that many governance teams lack. The pace of AI capability advancement outstrips governance framework development. Dual-use concerns: the same model may be low-risk in one context and high-risk in another. Board-level AI literacy is often insufficient for meaningful oversight of AI risk appetite decisions.

Threat Resistance

AI Governance addresses the governance failures that security controls alone cannot prevent. Training data bias leading to discriminatory outputs is mitigated through mandatory representativeness analysis, bias testing, and fairness metrics thresholds with automated deployment gates. Opaque AI decisions in regulated contexts are addressed through transparency obligations, explainability requirements scaled to risk tier, and decision-level audit trails that enable meaningful review and contestation. Uncontrolled model deployment is prevented by the model lifecycle with mandatory impact assessment, bias regression testing, and approval gates. Concept drift degrading fairness over time is detected through continuous monitoring with defined thresholds and automated escalation. Shadow AI models outside governance are contained through AI inventory requirements, acceptable use policies, and discovery mechanisms. The pattern's emphasis on human oversight with competence requirements ensures that human-in-the-loop controls are meaningful rather than rubber-stamp exercises.

Assumptions

The organisation deploys or intends to deploy AI systems that process personal data, make or support consequential decisions, or operate with a degree of autonomy. A centralised AI governance function exists or will be established (this may sit within risk, compliance, privacy, or a dedicated AI ethics team). The organisation has access to AI/ML engineering competence to implement bias testing, fairness metrics, and explainability techniques. NIST 800-53 Rev 5 controls are the security and privacy baseline; this pattern adds governance controls on top of that foundation. The regulatory landscape for AI is evolving rapidly -- this pattern should be reviewed quarterly.

Developing Areas

• EU AI Act implementation timelines are creating urgent compliance tooling gaps. The regulation entered into force in August 2024 with a phased implementation schedule: prohibited AI practices from February 2025, high-risk system obligations from August 2026, and full enforcement from August 2027. However, harmonised standards under the Act are still being developed by CEN/CENELEC, meaning organisations must build compliance programmes against requirements whose detailed technical specifications are not yet finalised. The conformity assessment ecosystem (notified bodies, technical documentation templates, testing methodologies) is at least 12-18 months from maturity.
• Algorithmic auditing standards are fragmented across multiple initiatives without convergence. IEEE 7003, NIST AI RMF, ISO 42001 Annex A, and emerging EU AI Act conformity requirements each define audit criteria differently. No single methodology has achieved the adoption and credibility that SOC 2 or ISO 27001 have for information security, leaving organisations unable to demonstrate responsible AI through a recognised, auditable standard. The IAASB (International Auditing and Assurance Standards Board) is developing assurance guidance for AI, but publication is not expected before 2027.
• AI incident reporting frameworks are in early development but lack the maturity of cybersecurity incident reporting. The OECD AI Incidents Monitor and the AI Incident Database collect reports voluntarily, but no jurisdiction has yet mandated AI incident reporting comparable to GDPR breach notification or DORA ICT incident reporting. The EU AI Act requires serious incident reporting for high-risk systems, but the definition of a serious AI incident, the reporting timeline, and the competent authority are still being clarified through implementing acts.
• Bias detection and fairness metrics standardisation remains contested because fairness itself is context-dependent and mathematically constrained. Demographic parity, equalised odds, and calibration are mutually incompatible in most scenarios (Chouldechova impossibility theorem), meaning organisations must choose which definition of fairness to optimise for -- a normative decision that technical metrics cannot resolve. The tooling for bias detection (IBM AI Fairness 360, Google What-If, Microsoft Fairlearn) is mature for tabular data but poorly adapted to generative AI outputs, where bias manifests in language, imagery, and reasoning patterns that resist quantitative measurement.
• AI supply chain governance -- particularly foundation model provenance -- is an urgent emerging challenge. Organisations building on third-party foundation models (GPT, Claude, Gemini, Llama) typically cannot verify training data composition, bias characteristics, or safety testing performed by the model provider. The EU AI Act places obligations on deployers of high-risk systems that use third-party models, creating a governance requirement that the supply chain does not yet support with adequate transparency documentation, model cards, or contractual assurances.

Related Patterns

Patterns that operate within or alongside this one. Click any to view.