Advanced Monitoring and Detection
This pattern starts where preventive controls end. It assumes that a motivated, skilled adversary has penetrated the perimeter, compromised an endpoint, and established an initial foothold in the environment. The question is no longer whether the attacker can be kept out, but how quickly the organisation can detect their presence, understand the scope of compromise, and mount an effective response. Advanced monitoring and detection is the architectural discipline of building the sensors, analytics, processes, and people needed to answer that question in hours rather than months.
The foundation of this pattern is comprehensive visibility. You cannot detect what you cannot see. This means collecting and centralising security-relevant data from across the environment: network traffic metadata and full packet capture at key boundaries, endpoint telemetry (process execution, file system changes, registry modifications, network connections), authentication and authorisation events from identity systems, DNS query logs, email gateway logs, cloud service activity logs, and application-layer audit trails. The volume of data is enormous, and the architecture must handle ingestion, normalisation, correlation, and retention at scale without creating blind spots due to cost-driven log exclusions.
SIEM (Security Information and Event Management) is the central nervous system, but a SIEM is only as good as the detection logic it runs and the analysts who investigate its output. Detection engineering must go beyond vendor-supplied correlation rules to include custom detections based on the organisation's specific threat model, ATT&CK-mapped detection coverage, and threat intelligence from sector-specific and commercial feeds. Behavioural analytics (UEBA -- User and Entity Behaviour Analytics) supplement rule-based detection by identifying anomalies in user activity, network traffic patterns, and data access that deviate from established baselines. Neither approach alone is sufficient; both must operate in concert.
The Security Operations Centre (SOC) operationalises detection capabilities. Whether internal, outsourced, or hybrid, the SOC must have defined triage procedures, escalation paths, investigation playbooks, and authority to take containment actions. Analyst burnout from alert fatigue is the single greatest operational risk -- if the SOC is drowning in false positives, real attacks will be missed. Detection engineering must be a continuous improvement loop: measure false positive rates, tune rules, retire ineffective detections, and invest analyst time in hunting activities where human intuition can find what automated rules miss.
Continuous monitoring extends beyond security events to include configuration drift detection, vulnerability state awareness, and compliance posture monitoring. The goal is to maintain a real-time understanding of the environment's security state so that deviations from expected baselines trigger investigation. This requires tight integration between monitoring platforms, asset inventory, configuration management, and vulnerability management systems. An alert about suspicious lateral movement is far more actionable when the SOC can immediately determine whether the target system is a development workstation or a domain controller, whether it has known unpatched vulnerabilities, and what data it has access to.
Click any control badge to view its details. Download SVG
Key Control Areas
Security Event Monitoring and SIEM
Configuration Baseline and Drift Detection
Vulnerability and Risk Intelligence
Access and Identity Monitoring
Incident Response Integration
Software and Information Integrity
System Architecture and Secure Development Practices
When to Use
Apply this pattern if your organisation may be a likely target of sophisticated blended attacks characteristic of Advanced Persistent Threats (APTs). This includes organisations in financial services, government, defence, critical infrastructure, healthcare, technology, and any sector handling high-value intellectual property or personally identifiable information. Regulatory requirements increasingly mandate continuous monitoring capabilities (NIST CSF DE.CM, PCI DSS Requirement 10, SOC 2 CC7, ISO 27001 A.8.15-16). Organisations that have experienced a breach or near-miss should implement this pattern as a priority. Any organisation with more than a few hundred endpoints should have at minimum a managed detection and response (MDR) capability.
When NOT to Use
Do not attempt to implement all elements of this pattern unless your organisation has high operational maturity with respect to change and configuration management. Key detective controls are dependent on accurate configuration management data -- without reliable baselines, anomaly detection generates unmanageable false positive volumes. Organisations at low security maturity should first establish basic controls (asset inventory, configuration management, centralised logging) before investing in advanced detection capabilities. Attempting to deploy a SIEM without the underlying data quality and staffing to operate it results in expensive infrastructure that provides little security value.
Typical Challenges
The primary operational challenge is alert fatigue: SOC analysts overwhelmed by false positives will miss genuine attacks buried in the noise. Addressing this requires continuous detection engineering -- tuning rules, retiring ineffective detections, enriching alerts with context, and automating repetitive triage tasks through SOAR (Security Orchestration, Automation, and Response) platforms. The volume of log data can be enormous, and cost management for SIEM licensing (often based on ingestion volume) frequently leads to difficult decisions about which log sources to include. Excluding sources creates blind spots that sophisticated adversaries will exploit. Achieving operational maturity with respect to configuration management is a prerequisite for many detection capabilities; organisations that cannot maintain accurate baselines will struggle with change detection and anomaly-based alerting. Staffing the SOC with skilled analysts is a persistent industry-wide challenge, with high turnover rates driven by burnout, shift work, and competitive demand for experienced security analysts. Measuring detection effectiveness is difficult -- you can count alerts and incidents, but the metric that matters most (attacks that were not detected) is inherently unknowable without regular red team exercises and assumed-breach testing.
Threat Resistance
This pattern is specifically designed to detect and limit the impact of advanced threats that bypass preventive controls. Advanced Persistent Threats (APTs) using multi-stage attack chains involving initial access, privilege escalation, lateral movement, and data exfiltration over extended dwell times. Living-off-the-land attacks that use legitimate system tools (PowerShell, WMI, PsExec) to avoid triggering signature-based detection. Credential-based attacks including pass-the-hash, Kerberoasting, and golden ticket attacks that operate within legitimate authentication channels. Insider threats where authorised users abuse their access for data theft or sabotage. Ransomware in its pre-encryption phases -- reconnaissance, lateral movement, and staging -- where early detection enables containment before encryption begins. Supply chain compromises where tampered updates or dependencies introduce backdoors. Zero-day exploitation where signature-based detection fails and behavioural analytics provide the primary detection mechanism. Data exfiltration through encrypted channels, DNS tunnelling, or steganography where network-based anomaly detection is required.
Assumptions
This pattern assumes that primary defensive layers (perimeter security, endpoint protection, access controls) have already been implemented but are insufficient against sophisticated adversaries who can bypass preventive controls. The organisation has sufficient operational maturity in change and configuration management to establish meaningful baselines against which anomalies can be detected -- without accurate configuration data, many detective controls produce excessive false positives and become operationally useless. The organisation is prepared to invest in skilled analysts or managed detection services, as monitoring technology without qualified human analysis generates noise rather than intelligence. Log sources across the environment are available and can be centralised, and sufficient storage and processing capacity exists to retain and analyse security data at the required scale.
Developing Areas
- AI-driven SOC triage is rapidly evolving from experimental to operational, with platforms using large language models to summarise alerts, correlate related events, and recommend analyst actions. Early deployments report 40-60% reduction in tier-1 triage time, but analyst trust in AI-generated recommendations remains a barrier -- analysts who cannot understand or verify the AI's reasoning are reluctant to act on its conclusions. The discipline is navigating the gap between AI capability and analyst confidence, with explainability and auditability of AI triage decisions as the critical unsolved design problems.
- Detection-as-code is maturing as a discipline but lacks standardised tooling and practices. The concept -- managing detection logic in version-controlled repositories with CI/CD testing, peer review, and automated deployment -- is well-established in principle, but the ecosystem is fragmented across incompatible formats (Sigma, KQL, SPL, YARA-L) and no universal detection language has emerged. Organisations investing in detection engineering face portability challenges when switching SIEM platforms and must maintain translation layers between detection formats.
- Security data lake architectures are challenging the traditional SIEM model by decoupling data storage from analytics. Platforms built on cloud object storage (Snowflake, Databricks, Amazon Security Lake) offer dramatically lower per-GB costs than traditional SIEM, enabling retention of telemetry that would be cost-prohibitive in SIEM-based architectures. However, the query performance, real-time alerting, and out-of-the-box detection capabilities of data lakes still lag purpose-built SIEMs, and most organisations are running hybrid architectures with SIEM for real-time detection and data lakes for threat hunting and long-term retention.
- SOAR effectiveness and maintenance burden is generating industry disillusionment after initial enthusiasm. Organisations that deployed SOAR platforms find that playbook development and maintenance requires dedicated engineering resources that were underestimated at purchase, and that the promise of fully automated response encounters practical barriers -- edge cases, API changes in integrated tools, and the need for human judgement at critical decision points. The emerging pattern is focused SOAR deployment for a small number of high-frequency, well-understood scenarios rather than attempting to automate the entire response lifecycle.
- Cloud-native detection coverage remains a significant gap for organisations with hybrid environments. Cloud provider audit trails (CloudTrail, Azure Activity Log, GCP Audit Logs) provide rich telemetry, but detection logic developed for on-premises environments does not translate directly to cloud-native attack patterns. Techniques like role assumption chaining in AWS, managed identity abuse in Azure, and service account key theft in GCP require purpose-built detections that most SOCs are still developing.
Related Patterns
Patterns that operate within or alongside this one. Click any to view.
References
- MITRE ATT&CK Framework
The industry-standard knowledge base of adversary tactics, techniques, and procedures. Essential for mapping detection coverage, identifying gaps, developing detection rules, and conducting adversary emulation exercises.
- NIST SP 800-137: Information Security Continuous Monitoring (ISCM)
Defines the ISCM process including establishing monitoring strategy, implementing monitoring programs, analysing data, and responding to findings. Provides the governance framework for continuous monitoring programs.
- NIST SP 800-92: Guide to Computer Security Log Management
Practical guidance on log management infrastructure, policies, and operational procedures. Covers log generation, storage, analysis, and the technical challenges of centralised log management at enterprise scale.
- SANS SOC Survey and Operational Best Practices
Annual survey data and practical guidance on SOC staffing, tooling, metrics, and operational maturity. Provides benchmarking data for SOC effectiveness across industries.
- CIS Controls v8: Control 8 (Audit Log Management) and Control 13 (Network Monitoring and Defense)
Prescriptive guidance on audit log collection, retention, and review (Control 8) and network monitoring, intrusion detection, and filtering (Control 13). Maps directly to the technical implementation of this pattern.
- Palantir Alerting and Detection Strategy Framework
Open-source framework for documenting, testing, and managing detection logic. Provides a structured approach to detection engineering that integrates with ATT&CK and supports detection lifecycle management.
- Verizon Data Breach Investigations Report (DBIR)
Annual analysis of breach data showing detection timelines, attack patterns, and the gap between attacker dwell time and organisational detection capability. Essential evidence base for justifying monitoring investments.