← Patterns / SP-031

Security Monitoring and Response

Security Monitoring and Response is the operational backbone of any security programme. Without the ability to detect threats and respond effectively, preventive controls are a single point of failure. This pattern defines the architecture for a complete detection and response pipeline: from raw telemetry at the edge through centralised correlation and analytics to orchestrated containment and recovery. The pipeline has four stages. Collection gathers telemetry from every source that matters: endpoint detection and response (EDR) agents on workstations and servers, network sensors and flow data, cloud audit trails, application logs, identity provider events, and email/web gateway logs. Correlation brings these streams together in a Security Information and Event Management (SIEM) platform or data lake, normalising formats, enriching with context (asset inventory, threat intelligence, identity data), and applying detection logic -- rules, statistical baselines, and machine learning models. Orchestration coordinates the response through a Security Orchestration, Automation, and Response (SOAR) platform that executes playbooks: enriching alerts with additional context, deduplicating and prioritising, and routing to the right analyst or automated workflow. Response executes containment, eradication, and recovery actions -- isolating compromised hosts, revoking credentials, blocking indicators, preserving forensic evidence, and restoring services. The pattern addresses the fundamental challenge of modern security operations: the volume of telemetry vastly exceeds human capacity to review it. A mid-size enterprise generates millions of log events per day. Without automation at every stage -- collection, normalisation, correlation, enrichment, triage, and response -- security teams drown in alerts while missing the signals that matter. This pattern provides the architectural framework to build detection and response capability that scales with the threat landscape.
Release: 26.02 Authors: Aurelius, Vitruvius Updated: 2026-02-07
Assess
ATT&CK This pattern addresses 448 techniques across 13 tactics View on ATT&CK Matrix →
SECURITY MONITORING GOVERNANCE & POLICY AU-02 AU-06 SI-04 IR-04 | CA-07 AU-12 AU-03 AU-09 SOURCES EDR Syslog Cloud Network AU-02 AU-12 AU-03 Auditable Events Content of Audit Records Audit Record Generation All infrastructure generates structured audit events. Endpoints, servers, cloud workloads, network devices feed the detection pipeline. Breadth of coverage determines detection ceiling. COLLECTION Log Aggregation Centralise all sources into unified pipeline Normalisation Parse, enrich, timestamp Common schema (ECS) AU-09 SI-04 Audit Log Protection System Monitoring Tamper-proof log storage. Events normalised to a common schema before correlation. Retention policies enforced. Integrity verification on all stored records. CORRELATION SIEM Event correlation engine Cross-source detection ! Threat Intel IOCs, TTPs, STIX/TAXII Feeds and enrichment Rule Engine Detection rules, ML models Sigma, YARA, custom logic AU-06 CA-07 SI-04 Audit Review & Analysis Continuous Monitoring The intelligence layer. Correlate events across sources. Match against known threats. Detect anomalous behaviour. ORCHESTRATION SOAR Security Orchestration, Automation & Response Playbooks Automated runbooks Predefined workflows Enrichment Context lookup, asset correlation, risk scoring IR-04 IR-05 Incident Handling Incident Monitoring Automate the response. Playbooks drive consistent, repeatable actions. Human decisions where required. Speed kills threats. RESPONSE Alert Triage and classify severity and priority Contain Isolate, block, quarantine Limit blast radius Investigate Root cause analysis Forensic evidence Report Timeline, artefacts Lessons learned IR-04 IR-05 IR-06 IR-08 Incident Handling / Monitoring Incident Reporting / Plans MTTD Mean Time to Detect MTTR Mean Time to Respond Four response actions executed in sequence or parallel. MTTD and MTTR are the key performance indicators. Measure. Improve. Repeat. Events Normalised Alerts Actions Feedback Loop DETECTION → ANALYSIS → CONTAINMENT → ERADICATION → RECOVERY → LESSONS LEARNED SP-031: Security Monitoring and Response 8 key NIST 800-53 Rev 5 controls · EDR pipeline architecture · Authors: Aurelius, Vitruvius · Draft · 2026-02-07 Data flow Alert / action Feedback XX-00 NIST control (click to view) Pipeline stage Aligned with: MITRE ATT&CK Framework · NIST SP 800-61 Rev 2 · Elastic Common Schema opensecurityarchitecture.org

Click any control badge to view its details. Download SVG

Key Control Areas

Telemetry Collection and Log Management

AU-02AU-03AU-04AU-05AU-08AU-12SI-04
The foundation of detection is comprehensive telemetry. AU-02 defines which events must be logged: authentication events, privilege escalation, data access, configuration changes, process execution, network connections, and failed actions. AU-03 specifies the content of each record: who, what, when, where, outcome, and contextual metadata. AU-04 ensures sufficient storage capacity for log retention -- under-provisioned log storage is a common failure that creates blind spots exactly when forensic data is needed. AU-05 defines responses when logging fails: alerts, failover to backup collectors, and in critical systems, halting operations rather than operating blind. AU-08 mandates time synchronisation across all sources using NTP with authenticated time sources -- without accurate timestamps, cross-source correlation is unreliable. AU-12 ensures audit record generation at the point of event occurrence. SI-04 provides real-time system monitoring including network intrusion detection, file integrity monitoring, and host-based anomaly detection. Implementation requires deploying collection agents (EDR, syslog forwarders, cloud log connectors), centralising to a log management platform, and ensuring retention meets both operational and compliance requirements.

Correlation, Analytics, and Detection Engineering

AU-06AU-07SI-04CA-07RA-05PM-16
Raw telemetry becomes intelligence through correlation and analysis. AU-06 mandates review, analysis, and reporting of audit records -- the core SIEM function. This includes rule-based correlation (known attack patterns, sigma rules), statistical baselining (anomaly detection for user behaviour, network traffic, process execution), and threat hunting (hypothesis-driven investigation of telemetry). AU-07 provides audit record reduction and report generation: transforming millions of raw events into prioritised, actionable alerts. SI-04 at the analytics layer applies network behaviour analysis, protocol anomaly detection, and integration with threat intelligence feeds. CA-07 provides continuous monitoring that goes beyond point-in-time assessments to real-time security posture awareness. RA-05 integrates vulnerability scanning data into the correlation engine, enriching alerts with vulnerability context -- an exploit attempt against a known-vulnerable system is higher priority than one targeting a patched system. PM-16 maintains threat awareness: tracking adversary TTPs via MITRE ATT&CK, consuming threat intelligence, and updating detection logic as the threat landscape evolves. Detection engineering must be treated as a continuous discipline, not a one-time configuration.

Alert Triage, Enrichment, and Orchestration

IR-04IR-05IR-07SI-05PM-14
Orchestration bridges detection and response. IR-04 handles incident management including automated initial triage: when an alert fires, the SOAR platform enriches it with asset context (is this a crown jewel system?), identity context (is this a privileged account?), vulnerability context (is the target vulnerable?), and threat intelligence (is the indicator known malicious?). This enrichment transforms a raw alert into a decision-ready case. IR-05 provides continuous incident monitoring, correlating related alerts into coherent incidents rather than treating each alert in isolation. IR-07 delivers incident response assistance through automated playbooks and analyst decision support. SI-05 distributes security alerts and advisories from vendors and intelligence sources, integrating them into the triage workflow. PM-14 ensures testing and monitoring programmes validate that the detection and response pipeline works end-to-end -- regular purple team exercises, detection coverage assessments, and mean-time-to-detect/respond metrics.

Automated and Manual Response

IR-04IR-06IR-08SC-07AC-02SI-04
Response must be fast enough to limit blast radius. IR-04 at the response layer executes containment actions: network isolation of compromised hosts via EDR or SDN policy, credential revocation through identity provider integration, blocking of malicious IPs/domains at firewall and proxy, and quarantine of malicious files. IR-06 automates incident reporting to stakeholders, regulators, and law enforcement where required. IR-08 defines incident response plans with specific playbooks for common scenarios: ransomware, business email compromise, data exfiltration, insider threat, and supply chain compromise. SC-07 enables dynamic boundary reconfiguration for containment -- automatically tightening network segmentation to isolate affected zones. AC-02 supports response through emergency account disablement and forced credential rotation. SI-04 provides ongoing monitoring during response to confirm containment effectiveness and detect attacker pivoting. The balance between automated and manual response depends on confidence: high-confidence, low-risk actions (blocking a known-malicious hash) should be fully automated; high-impact actions (isolating a production server) should require analyst approval.

Forensics, Evidence Preservation, and Recovery

AU-09IR-04CP-09CP-10SI-07
Post-incident activities ensure lessons are learned and evidence supports legal proceedings. AU-09 protects audit information from tampering -- immutable log stores, write-once-read-many archives, and cryptographic integrity verification ensure that forensic evidence is admissible and trustworthy. IR-04 includes post-incident analysis: root cause identification, timeline reconstruction, and identification of detection gaps. CP-09 provides system backup that enables recovery to a known-good state after containment. CP-10 covers system recovery and reconstitution: bringing affected systems back online safely, verifying they are clean, and monitoring them closely post-recovery. SI-07 verifies software and firmware integrity during recovery, ensuring that restored systems have not been tampered with. Every incident should produce updated detection rules, improved playbooks, and refined response procedures.

SOC Operations and Workforce

AT-02AT-03IR-02PS-04PS-07
People are the irreducible core of security operations. AT-02 provides security awareness across the organisation -- users are sensors too, and trained users report phishing, anomalous behaviour, and policy violations. AT-03 delivers role-based training for SOC analysts: tier 1 triage, tier 2 investigation, tier 3 threat hunting, and incident commander roles. IR-02 provides incident response training including tabletop exercises, simulation drills, and post-incident reviews. PS-04 governs personnel termination procedures, ensuring access revocation is immediate and monitored -- a disgruntled leaver with active credentials is a high-risk insider threat. PS-07 manages third-party personnel security for outsourced SOC functions including managed detection and response (MDR) providers.

Threat Intelligence Integration

PM-16RA-03RA-05SI-05SR-10
Threat intelligence transforms reactive detection into proactive defence. PM-16 establishes a threat awareness programme that consumes strategic, tactical, and operational intelligence. RA-03 uses threat intelligence to inform risk assessments: which adversary groups target your sector, what TTPs they use, and which of your assets are most likely targets. RA-05 integrates vulnerability intelligence into monitoring: correlating CVE data with asset inventory to prioritise detection for exploitable vulnerabilities. SI-05 distributes security alerts from vendors, ISACs, and government agencies. SR-10 extends monitoring into the supply chain, detecting compromised components and malicious updates before they reach production systems. Intelligence should feed directly into detection engineering: every new intelligence report should be evaluated for detection opportunities.

When to Use

This pattern applies to any organisation that needs to detect and respond to security threats -- which in practice means every organisation. It is particularly critical for organisations in regulated industries (financial services, healthcare, critical infrastructure) where detection and response capabilities are mandated. Organisations with significant cloud workloads need cloud-native detection alongside traditional on-premises monitoring. Those experiencing rapid growth need scalable detection architecture before the estate outgrows ad-hoc monitoring. Any organisation that has suffered a breach where dwell time exceeded days should treat this pattern as urgent.

When NOT to Use

Very small organisations (under 20 users) with simple IT environments may find a full SIEM/SOAR deployment disproportionate and should consider managed detection and response (MDR) services as an alternative. Organisations without basic preventive controls (patch management, endpoint protection, access control) should establish foundations before investing in advanced detection -- you need to reduce the noise floor before detection becomes effective. Air-gapped environments with no internet connectivity have a fundamentally different threat model and monitoring approach.

Typical Challenges

Log volume is the first challenge: a single busy web server generates gigabytes of logs daily, and storage costs for multi-year retention are significant. Alert fatigue is endemic -- poorly tuned SIEM rules generate thousands of false positives that desensitise analysts and bury genuine threats. SIEM deployment often stalls at log collection without progressing to meaningful detection engineering. Correlating events across disparate sources requires normalisation that is never as clean as vendors promise. SOC analyst retention is an industry-wide problem: burnout from shift work, alert fatigue, and repetitive triage drives turnover above 30% annually in many organisations. Measuring SOC effectiveness is difficult -- mean-time-to-detect and mean-time-to-respond are useful but can be gamed. Cloud environments generate novel telemetry formats that traditional on-premises SIEM tools handle poorly. Encrypted traffic (TLS 1.3) limits network-based detection, shifting reliance to endpoint and identity telemetry. SOAR playbook maintenance requires ongoing engineering effort that is often underestimated. Executive expectations of 'real-time detection' clash with the reality that sophisticated adversaries operate within normal user behaviour patterns.

Threat Resistance

Security Monitoring and Response directly addresses the detection and containment phases of the kill chain. Advanced persistent threats with long dwell times are detected through behavioural analytics and anomaly detection (AU-06, SI-04, CA-07). Ransomware is detected at multiple stages: initial access via phishing detection, lateral movement via network anomaly detection, and pre-encryption staging via endpoint behavioural monitoring (SI-04, IR-04). Insider threats are identified through user behaviour analytics that detect anomalous data access, privilege use, and working patterns (AU-06, AC-02, PS-04). Supply chain compromise is detected through software integrity monitoring and update verification (SI-07, SR-10). Credential theft and abuse is detected through impossible travel, anomalous authentication patterns, and failed logon monitoring (AU-02, AC-07, IA-05). Data exfiltration is detected through DLP integration, network flow analysis, and cloud access monitoring (AC-04, SI-04, AU-06). The key architectural principle is defence in depth for detection: no single detection mechanism catches everything, but layered collection, correlation, and analytics create overlapping detection zones that are extremely difficult for adversaries to evade simultaneously.

Assumptions

The organisation has committed to building or procuring security operations capability (in-house SOC, managed SOC, or hybrid). Network architecture supports centralised log collection (agents can reach collectors, bandwidth is sufficient). A log management or SIEM platform is deployed or planned. Endpoints support EDR agent deployment across the estate. Identity providers can emit authentication and authorisation events. Cloud environments expose audit trails via APIs. The organisation has or will develop incident response procedures and playbooks.

Developing Areas

  • AI-driven triage accuracy versus analyst trust is the defining tension in modern SOC operations. AI copilots can summarise alerts, suggest investigation steps, and recommend containment actions with impressive speed, but SOC analysts report low confidence in AI recommendations they cannot independently verify. The fundamental challenge is that AI triage operates as a black box -- analysts cannot trace the reasoning chain from raw telemetry to recommendation -- and in a domain where false negatives have severe consequences, trust must be earned through demonstrated accuracy over extended operational periods.
  • Detection engineering as a formalised discipline is still being defined. While the concept of treating detection logic with the same engineering rigour as software development (version control, testing, peer review, CI/CD deployment) is gaining acceptance, the profession lacks standardised job descriptions, career paths, training curricula, and certification. The Sigma rules project provides a community-maintained detection corpus, but most organisations still write detections in vendor-specific query languages without systematic coverage analysis or quality assurance processes.
  • Security data lake economics are disrupting the traditional SIEM market but creating architectural complexity. The cost per GB of storing security telemetry in cloud object storage (S3, GCS, ADLS) is 10-50x lower than in traditional SIEM platforms, enabling retention of data sources that were previously excluded for cost reasons. However, the trade-off is query latency, real-time alerting capability, and the need to build or buy an analytics layer on top of the raw storage. Most organisations are converging on a hybrid architecture -- hot SIEM for real-time detection and warm data lake for hunting and historical analysis -- but the optimal split between tiers is still being established.
  • OCSF and OpenTelemetry convergence for security observability is a developing standards effort that could fundamentally improve cross-platform detection. The Open Cybersecurity Schema Framework (OCSF) provides a vendor-neutral data model for security events, while OpenTelemetry standardises telemetry collection for distributed systems. The convergence of these standards could eliminate the normalisation burden that consumes 30-40% of SIEM engineering effort, but adoption requires vendor commitment that is progressing unevenly, and organisations with existing SIEM investments face migration costs.
  • SOAR playbook maintenance burden is generating significant operational debt in organisations that invested heavily in automation. Playbooks that worked at deployment break as integrated APIs change, vendor products update, and the threat landscape evolves. Industry surveys report that the average SOAR deployment has 30-40% of playbooks in a degraded or broken state at any given time, and the engineering resources required for ongoing maintenance were systematically underestimated during procurement. The emerging pattern is fewer, more robust playbooks focused on high-frequency use cases rather than comprehensive automation of the entire incident lifecycle.

Related Patterns

Patterns that operate within or alongside this one. Click any to view.

SP-001 Client Module SP-002 Server Module SP-011 Cloud Computing SP-025 Advanced Monitoring and Detection SP-028 Secure DevOps Pipeline SP-029 Zero Trust Architecture SP-034 Cyber Resilience
AC: 3AT: 2AU: 9CA: 1CP: 2IA: 1IR: 6PL: 1PM: 2PS: 2RA: 2SC: 1SI: 3SR: 1
AC-02 Account Management
AC-04 Information Flow Enforcement
AC-07 Unsuccessful Logon Attempts
AT-02 Literacy Training and Awareness
AT-03 Role-Based Training
AU-02 Event Logging
AU-03 Content of Audit Records
AU-04 Audit Log Storage Capacity
AU-05 Response to Audit Logging Process Failures
AU-06 Audit Record Review, Analysis, and Reporting
AU-07 Audit Record Reduction and Report Generation
AU-08 Time Stamps
AU-09 Protection of Audit Information
AU-12 Audit Record Generation
CA-07 Continuous Monitoring
CP-09 System Backup
CP-10 System Recovery and Reconstitution
IA-05 Authenticator Management
IR-02 Incident Response Training
IR-04 Incident Handling
IR-05 Incident Monitoring
IR-06 Incident Reporting
IR-07 Incident Response Assistance
IR-08 Incident Response Plan
PL-02 System Security and Privacy Plans
PM-14 Testing, Training, and Monitoring
PM-16 Threat Awareness Program
PS-04 Personnel Termination
PS-07 External Personnel Security
RA-03 Risk Assessment
RA-05 Vulnerability Monitoring and Scanning
SC-07 Boundary Protection
SI-04 System Monitoring
SI-05 Security Alerts, Advisories, and Directives
SI-07 Software, Firmware, and Information Integrity
SR-10 Inspection of Systems or Components