Advanced Monitoring and Detection

Key Control Areas

When to Use

Apply this pattern if your organisation may be a likely target of sophisticated blended attacks characteristic of Advanced Persistent Threats (APTs). This includes organisations in financial services, government, defence, critical infrastructure, healthcare, technology, and any sector handling high-value intellectual property or personally identifiable information. Regulatory requirements increasingly mandate continuous monitoring capabilities (NIST CSF DE.CM, PCI DSS Requirement 10, SOC 2 CC7, ISO 27001 A.8.15-16). Organisations that have experienced a breach or near-miss should implement this pattern as a priority. Any organisation with more than a few hundred endpoints should have at minimum a managed detection and response (MDR) capability.

When NOT to Use

Do not attempt to implement all elements of this pattern unless your organisation has high operational maturity with respect to change and configuration management. Key detective controls are dependent on accurate configuration management data -- without reliable baselines, anomaly detection generates unmanageable false positive volumes. Organisations at low security maturity should first establish basic controls (asset inventory, configuration management, centralised logging) before investing in advanced detection capabilities. Attempting to deploy a SIEM without the underlying data quality and staffing to operate it results in expensive infrastructure that provides little security value.

Typical Challenges

The primary operational challenge is alert fatigue: SOC analysts overwhelmed by false positives will miss genuine attacks buried in the noise. Addressing this requires continuous detection engineering -- tuning rules, retiring ineffective detections, enriching alerts with context, and automating repetitive triage tasks through SOAR (Security Orchestration, Automation, and Response) platforms. The volume of log data can be enormous, and cost management for SIEM licensing (often based on ingestion volume) frequently leads to difficult decisions about which log sources to include. Excluding sources creates blind spots that sophisticated adversaries will exploit. Achieving operational maturity with respect to configuration management is a prerequisite for many detection capabilities; organisations that cannot maintain accurate baselines will struggle with change detection and anomaly-based alerting. Staffing the SOC with skilled analysts is a persistent industry-wide challenge, with high turnover rates driven by burnout, shift work, and competitive demand for experienced security analysts. Measuring detection effectiveness is difficult -- you can count alerts and incidents, but the metric that matters most (attacks that were not detected) is inherently unknowable without regular red team exercises and assumed-breach testing.

Threat Resistance

This pattern is specifically designed to detect and limit the impact of advanced threats that bypass preventive controls. Advanced Persistent Threats (APTs) using multi-stage attack chains involving initial access, privilege escalation, lateral movement, and data exfiltration over extended dwell times. Living-off-the-land attacks that use legitimate system tools (PowerShell, WMI, PsExec) to avoid triggering signature-based detection. Credential-based attacks including pass-the-hash, Kerberoasting, and golden ticket attacks that operate within legitimate authentication channels. Insider threats where authorised users abuse their access for data theft or sabotage. Ransomware in its pre-encryption phases -- reconnaissance, lateral movement, and staging -- where early detection enables containment before encryption begins. Supply chain compromises where tampered updates or dependencies introduce backdoors. Zero-day exploitation where signature-based detection fails and behavioural analytics provide the primary detection mechanism. Data exfiltration through encrypted channels, DNS tunnelling, or steganography where network-based anomaly detection is required.

Assumptions

This pattern assumes that primary defensive layers (perimeter security, endpoint protection, access controls) have already been implemented but are insufficient against sophisticated adversaries who can bypass preventive controls. The organisation has sufficient operational maturity in change and configuration management to establish meaningful baselines against which anomalies can be detected -- without accurate configuration data, many detective controls produce excessive false positives and become operationally useless. The organisation is prepared to invest in skilled analysts or managed detection services, as monitoring technology without qualified human analysis generates noise rather than intelligence. Log sources across the environment are available and can be centralised, and sufficient storage and processing capacity exists to retain and analyse security data at the required scale.

Developing Areas

• AI-driven SOC triage is rapidly evolving from experimental to operational, with platforms using large language models to summarise alerts, correlate related events, and recommend analyst actions. Early deployments report 40-60% reduction in tier-1 triage time, but analyst trust in AI-generated recommendations remains a barrier -- analysts who cannot understand or verify the AI's reasoning are reluctant to act on its conclusions. The discipline is navigating the gap between AI capability and analyst confidence, with explainability and auditability of AI triage decisions as the critical unsolved design problems.
• Detection-as-code is maturing as a discipline but lacks standardised tooling and practices. The concept -- managing detection logic in version-controlled repositories with CI/CD testing, peer review, and automated deployment -- is well-established in principle, but the ecosystem is fragmented across incompatible formats (Sigma, KQL, SPL, YARA-L) and no universal detection language has emerged. Organisations investing in detection engineering face portability challenges when switching SIEM platforms and must maintain translation layers between detection formats.
• Security data lake architectures are challenging the traditional SIEM model by decoupling data storage from analytics. Platforms built on cloud object storage (Snowflake, Databricks, Amazon Security Lake) offer dramatically lower per-GB costs than traditional SIEM, enabling retention of telemetry that would be cost-prohibitive in SIEM-based architectures. However, the query performance, real-time alerting, and out-of-the-box detection capabilities of data lakes still lag purpose-built SIEMs, and most organisations are running hybrid architectures with SIEM for real-time detection and data lakes for threat hunting and long-term retention.
• SOAR effectiveness and maintenance burden is generating industry disillusionment after initial enthusiasm. Organisations that deployed SOAR platforms find that playbook development and maintenance requires dedicated engineering resources that were underestimated at purchase, and that the promise of fully automated response encounters practical barriers -- edge cases, API changes in integrated tools, and the need for human judgement at critical decision points. The emerging pattern is focused SOAR deployment for a small number of high-frequency, well-understood scenarios rather than attempting to automate the entire response lifecycle.
• Cloud-native detection coverage remains a significant gap for organisations with hybrid environments. Cloud provider audit trails (CloudTrail, Azure Activity Log, GCP Audit Logs) provide rich telemetry, but detection logic developed for on-premises environments does not translate directly to cloud-native attack patterns. Techniques like role assumption chaining in AWS, managed identity abuse in Azure, and service account key theft in GCP require purpose-built detections that most SOCs are still developing.

Related Patterns

Patterns that operate within or alongside this one. Click any to view.