Security information and event management

SIEM tools can be implemented as software, hardware, or managed services. SIEM systems log security events and generate reports to meet regulatory frameworks such as the Health Insurance Portability and Accountability Act (HIPAA) and the Payment Card Industry Data Security Standard (PCI DSS). The integration of SIM and SEM within SIEM provides organizations with a centralized approach for monitoring security events and responding to threats in real-time.

First introduced by Gartner analysts Mark Nicolett and Amrit Williams in 2005, the term SIEM has evolved to incorporate advanced features such as threat intelligence and behavioral analytics, which allow SIEM solutions to manage complex cybersecurity threats, including zero-day vulnerabilities and polymorphic malware.

In recent years, SIEM has become increasingly incorporated into national cybersecurity initiatives. For instance, Executive Order 14028 signed in 2021 by U.S. President Joseph Biden mandates the use of SIEM technologies to improve incident detection and reporting in federal systems. Compliance with these mandates is further reinforced by frameworks such as NIST SP 800-92, which outlines best practices for managing computer security logs.

Modern SIEM platforms are aggregating and normalizing data not only from various Information Technology (IT) sources, but from production and manufacturing Operational Technology (OT) environments as well.

History

Initially, system logging was primarily used for troubleshooting and debugging. However, as operating systems and networks have grown more complex, so has the generation of system logs. The monitoring of system logs has also become increasingly common due to the rise of sophisticated cyberattacks and the need for compliance with regulatory frameworks, which mandate logging security controls within risk management frameworks (RMF).

Starting in the late 1970s, working groups began establishing criteria for managing auditing and monitoring programs, laying the groundwork for modern cybersecurity practices, such as insider threat detection and incident response. A key publication during this period was NIST’s Special Publication 500-19.

In 2005, the term "SIEM" (Security Information and Event Management) was introduced by Gartner analysts Mark Nicolett and Amrit Williams. SIEM systems provide a single interface for gathering security data from information systems and presenting it as actionable intelligence. The National Institute of Standards and Technology provides the following definition of SIEM: "Application that provides the ability to gather security data from information system components and present that data as actionable information via a single interface." In addition, NIST has designed and implemented a federally mandated RMF.

With the implementation of RMFs globally, auditing and monitoring have become central to information assurance and security. Cybersecurity professionals now rely on logging data to perform real-time security functions, driven by governance models that incorporate these processes into analytical tasks. As information assurance matured in the late 1990s and into the 2000s, the need to centralize system logs became apparent. Centralized log management allows for easier oversight and coordination across networked systems.

On May 17, 2021, U.S. President Joseph Biden signed Executive Order 14028, "Improving the Nation's Cybersecurity," which established further logging requirements, including audit logging and endpoint protection, to enhance incident response capabilities. This order was a response to an increase in ransomware attacks targeting critical infrastructure. By reinforcing information assurance controls within RMFs, the order aimed to drive compliance and secure funding for cybersecurity initiatives.

7 more chapters are free with a Lably account.

Information assurance

Published in September 2006, the NIST SP 800-92 Guide to Computer Security Log Management serves as a key document within the NIST Risk Management Framework to guide what should be auditable. As indicated by the absence of the term "SIEM", the document was released before the widespread adoption of SIEM technologies. Although the guide is not exhaustive due to rapid changes in technology since its publication, it remains relevant by anticipating industry growth. NIST is not the only source of guidance on regulatory mechanisms for auditing and monitoring, and many organizations are encouraged to adopt SIEM solutions rather than relying solely on host-based checks.

Several regulations and standards reference NIST’s logging guidance, including the Federal Information Security Management Act (FISMA), Gramm-Leach-Bliley Act (GLBA), Health Insurance Portability and Accountability Act (HIPAA), Sarbanes-Oxley Act (SOX) of 2002, Payment Card Industry Data Security Standard (PCI DSS), and ISO 27001. Public and private organizations frequently reference NIST documents in their security policies.

NIST SP 800-53 AU-2 Event Monitoring is a key security control that supports system auditing and ensures continuous monitoring for information assurance and cybersecurity operations. SIEM solutions are typically employed as central tools for these efforts. Federal systems categorized based on their impact on confidentiality, integrity, and availability (CIA) have five specific logging requirements (AU-2 a-e) that must be met. While logging every action is possible, it is generally not recommended due to the volume of logs and the need for actionable security data. AU-2 provides a foundation for organizations to build a logging strategy that aligns with other controls.

NIST SP 800-53 SI-4 System Monitoring outlines the requirements for monitoring systems, including detecting unauthorized access and tracking anomalies, malware, and potential attacks. This security control specifies both the hardware and software requirements for detecting suspicious activities. Similarly, NIST SP 800-53 RA-10 Threat Hunting, added in Revision 5, emphasizes proactive network defense by identifying threats that evade traditional controls. SIEM solutions play a critical role in aggregating security information for threat hunting teams.

Together, AU-2, SI-4, and RA-10 demonstrate how NIST controls integrate into a comprehensive security strategy. These controls, supported by SIEM solutions, help ensure continuous monitoring, risk assessments, and in-depth defense mechanisms across federal and private networks.

Terminology

The acronyms SEM, SIM and SIEM have sometimes been used interchangeably, but generally refer to the different primary focus of products:

Log management: Focus on simple collection and storage of log messages and audit trails.
Security information management (SIM): Long-term storage as well as analysis and reporting of log data.
Security event manager (SEM): Real-time monitoring, correlation of events, notifications and console views.
Security information and event management (SIEM): Combines SIM and SEM and provides real-time analysis of security alerts generated by network hardware and applications.
Managed Security Service: (MSS) or Managed Security Service Provider: (MSSP): The most common managed services appear to evolve around connectivity and bandwidth, network monitoring, security, virtualization, and disaster recovery.
Security as a service (SECaaS): These security services often include authentication, anti-virus, anti-malware/spyware, intrusion detection, penetration testing and security event management, among others.

In practice many products in this area will have a mix of these functions, so there will often be some overlap – and many commercial vendors also promote their own terminology. Oftentimes commercial vendors provide different combinations of these functionalities which tend to improve SIEM overall. Log management alone doesn't provide real-time insights on network security, SEM on its own won't provide complete data for deep threat analysis. When SEM and log management are combined, more information is available for SIEM to monitor.

A key focus is to monitor and help manage user and service privileges, directory services and other system-configuration changes; as well as providing log auditing and review and incident response.

Capabilities

Data aggregation: Log management aggregates data from many sources, including networks, security, servers, databases, applications, providing the ability to consolidate monitored data to help avoid missing crucial events.
Correlation: Looks for common attributes and links events together into meaningful bundles. This technology provides the ability to perform a variety of correlation techniques to integrate different sources, in order to turn data into useful information. Correlation is typically a function of the Security Event Management portion of a full SIEM solution.
Alerting: The automated analysis of correlated events.
Dashboards: Tools can take event data and turn it into informational charts to assist in seeing patterns, or identifying activity that is not forming a standard pattern.
Compliance: Applications can be employed to automate the gathering of compliance data, producing reports that adapt to existing security, governance and auditing processes.
Retention: Employing long-term storage of historical data to facilitate correlation of data over time, and to provide the retention necessary for compliance requirements. The Long term log data retention is critical in forensic investigations as it is unlikely that the discovery of a network breach will be at the time of the breach occurring.
Forensic analysis: The ability to search across logs on different nodes and time periods based on specific criteria. This mitigates having to aggregate log information in your head or having to search through thousands and thousands of logs.

Components

SIEM architectures may vary by vendor; however, generally, essential components comprise the SIEM engine. The essential components of a SIEM are as follows:

A data collector forwards selected audit logs from a host (agent based or host based log streaming into index and aggregation point)
An ingest and indexing point aggregation point for parsing, correlation, and data normalization
A search node that is used for visualization, queries, reports, and alerts (analysis take place on a search node)

A basic SIEM infrastructure is depicted in the image to the right.

Use cases

Computer security researcher Chris Kubecka identified the following SIEM use cases, presented at the hacking conference 28C3 (Chaos Communication Congress).

SIEM visibility and anomaly detection could help detect zero-days or polymorphic code. Primarily due to low rates of anti-virus detection against this type of rapidly changing malware.
Parsing, log normalization and categorization can occur automatically, regardless of the type of computer or network device, as long as it can send a log.
Visualization with a SIEM using security events and log failures can aid in pattern detection.
Protocol anomalies that can indicate a misconfiguration or a security issue can be identified with a SIEM using pattern detection, alerting, baseline and dashboards.
SIEMS can detect covert, malicious communications and encrypted channels.
Cyberwarfare can be detected by SIEMs with accuracy, discovering both attackers and victims.

Modern SIEM platforms support not only detection, but response too. The response can be manual or automated including AI based response.

Correlation rules examples

SIEM systems can have hundreds and thousands of correlation rules. Some of these are simple, and some are more complex. Once a correlation rule is triggered the system can take appropriate steps to mitigate a cyber attack. Usually, this includes sending a notification to a user and then possibly limiting or even shutting down the system.

Brute force detection

Brute force detection is relatively straightforward. Brute forcing relates to continually trying to guess a variable. It most commonly refers to someone trying to constantly guess your password - either manually or with a tool. However, it can refer to trying to guess URLs or important file locations on your system.

An automated brute force is easy to detect as someone trying to enter their password 60 times in a minute is impossible.

Impossible travel

When a user logs in to a system, generally speaking, it creates a timestamp of the event. Alongside the time, the system may often record other useful information such as the device used, physical location, IP address, incorrect login attempts, etc. The more data is collected the more use can be gathered from it. For impossible travel, the system looks at the current and last login date/time and the difference between the recorded distances. If it deems it's not possible for this to happen, for example traveling hundreds of miles within a minute, then it will set off a warning.

Many employees and users are now using VPN services which may obscure physical location. This should be taken into consideration when setting up such a rule.

Excessive file copying

The average user does not typically copy or move files on the system repeatedly. Thus, any excessive file copying on a system could be attributed to an attacker wanting to cause harm to an organization. Unfortunately, it's not as simple as stating someone has gained access to your network illegally and wants to steal confidential information. It could also be an employee looking to sell company information, or they could just want to take home some files for the weekend.

Network anomaly detection

Monitoring network traffic against unusual patterns that includes any threats or attacks ranging from DDOS to network scans. Note SIEM can monitor data flow in the network and to detect and prevent potential data exfiltration efforts. In general dedicated Data loss prevention (DLP) take care about data loss prevention.

DDoS attack

A DDoS (distributed denial of service) attack could cause significant damage to a company or organization. A DDoS attack can not only take a website offline, it can also make a system weaker. With suitable correlation rules in place, a SIEM should trigger an alert at the start of the attack so that the company can take the necessary precautionary measures to protect vital systems.

File integrity change

File integrity and change monitoring (FIM) is the process of monitoring the files on your system. Unexpected changes in your system files will trigger an alert as it's a likely indication of a cyber attack.

Alerting examples

Some examples of customized rules to alert on event conditions involve user authentication rules, attacks detected and infections detected.