SIEM: The Complete Guide
Learn all about Security Information and Event Management
- The basics of SIEM and how it operates.
- What’s changed over a decade of SIEM evolution.
- Use cases for beginners and seasoned veterans alike.
- Things to look for when buying a SIEM solution.
How does SIEM work?
All about data sources, SIEM architecture, correlation and analytics in SIEMLearn More
Incident Response and Automation with SIEM
About incident response, the benefits of security automation and orchestrationLearn More
Advanced SIEM Capabilities: UEBA and SOAR
Beyond alerting and compliance, detecting threats never seen beforeLearn More
The Basics of SIEM
What is SIEM?
Security information and event management (SIEM) technology supports threat detection and security incident response through the real-time collection and historical analysis of security events from a wide variety of event and contextual data sources. It also supports compliance reporting and incident investigation through analysis of historical data from these sources. The core capabilities of SIEM technology are a broad scope of event collection and the ability to correlate and analyze events across disparate sources.While you may be familiar with SIEM software as a tool to help with regulatory compliance, SIEM offers much more than this. It can form the foundation of your security strategy, a fact that more and more organizations are becoming aware of.
Why do companies need SIEM?
SIEM Market Share
How does SIEM work
10 SIEM Capabilities At A Glance
Log data aggregation
Collect and aggregate data from multiple data sources, like network devices, security devices and cloud services.
Custom data enrichment with external threat intelligence providers.
Correlation and analytics
Connect the dots between related security events to see the complete picture.
Real-time security monitoring
Monitor key metrics and traffic profiles to identify anomalies.
Detect issues and notify incident handlers for further investigation or remediation.
Give your team a unified security overview, making it easier to identify anomalies with threat storylines.
Identify and investigate incidents faster by bringing in relevant context and threat information.
Keep your organization audit-ready with comprehensive reporting.
Derive insights from your log and event data by writing queries.
Detect, validate and respond to threats without lifting a finger, thanks to built-in security orchestration, automation and response (SOAR) functionality.
How SIEM Works
Ingesting and Interpreting LogsA SIEM at its core comprises of a log collection and storage frameworks, that facilitates log management and analysis.
Data collection and IngestionSIEM platforms can collect data from a variety of data sources, such as network devices, servers, applications, DLP/IDS/IPS engines and BYOD platforms. Usually, data collection is performed in four ways:
- Log forwarding software collects and forwards logs from a device to a SIEM platform. This is known as agent collection.
- Logs are forwarded directly over HTTP(S) or via APIs.
- Devices forward logs directly in the syslog format.
- Event streaming protocols like NETFLOW or SNMP can forward events to a SIEM platform as they occur.
Data indexing and storageAfter data ingestion, the log data is parsed and stored in a database. Traditional SIEM solutions typically use RDBMS storage systems; today, in response to their schema and scalability limitations, next-gen SIEMs have replaced this technology with distributed, horizontally scalable architecture and storage setups.
Log management and retentionIn most organizations, SIEM platforms must rapidly ingest huge volumes of log data in a variety of formats. Over a period of time, the stored data generally goes through two stages, as determined in a log retention policy:
- Tiered: Most tiered data has been recently collected. Given its relevance to the current situation, it must be readily accessible, such as via queries. Historical data that may be needed in an investigation must also remain accessible alongside the most recently gathered data. Therefore, these data sets must be stored on storage media with high throughput rates, which improves performance..
- Archived: Data sets older than the log retention threshold can be transferred to archival storage (using less expensive storage media), as these data are unlikely to be useful for heuristic analysis and investigations.
SIEM and threat intelligence feeds
Data enrichmentMost SIEM solutions today have the ability to enrich stored log data in some way. This added information may include geographic location, inter-departmental metadata, and much more. Modern SIEM solutions have provisions to enrich event data in real time with information pertaining to user risk scores and anomalies in usage patterns as well.
Threat validationValidating threats and alerts is crucial to threat hunting. Traditional SIEM solutions tend to have limited threat correlation capabilities. This means that more manual investigation is required, relying heavily on analysts’ experience and skills. In addition, this approach is time-consuming and can result in a high rate of false positives. Most SIEMs now are able to augment event data with threat information (in form of Indicators of Compromise IOCs) from third party intel providers. Given the broad range of scenarios that benefit from automated validation, the use of a next-generation SIEM platform significantly reduces the overall time to respond. By providing relevant threat information and focusing on mission-critical events, analysts can investigate more rapidly.
Correlation and analyticsTraditional SIEM solutions rely on pre-written correlation rules for threat hunting purposes. These basic correlation rules are prone to producing false positives and false negatives, because rule-based correlation is geared toward finding known threats and patterns. Pre-written rules cannot adapt to today’s constantly shifting threat landscape, in which APTs and insider threats have become more prominent. The result is unreliable security alerts, as well as missed alerts that may quickly lead to catastrophic incidents. Data-driven cybersecurity analytics, however, now plays a crucial role in helping vendors and security analysts do much more with their log data. The ability to correlate event data using machine learning (ML) techniques while monitoring user behavior has made it possible for modern SIEM platforms to address risks beyond the scope of traditional, signature-based detection models.
Security alerts and data visualization
Real-time event data enrichment reduces overall investigation times, giving analysts a complete contextual threat timeline in the form of a dashboard featuring drill-down analysis.
Most SIEM platforms feature dashboards, which allow analysts to visualize and interact with data. Security teams can customize dashboards to include information and tools relevant to any given situation.
Security compliance and advanced reporting
Regulatory compliance remains a key driver for SIEM deployment, and SIEM solutions continue to be go-to tools in deriving insights and automatically generating reports for audit purposes.
- Integration with non-standard, customized devices and applications requires analysts to fine-tune parsers.
- Relational database management systems like MySQL have schema limitations that make it difficult to store and access large volumes of data.
- Enriching event data with contextual information is impossible.
- Detection strategies are primarily static or point-based.
- Vertical scalability promotes rip-and-replace architecture, which is expensive.
- Traditional platforms weren’t designed with ML in mind.
- Scaling is difficult and expensive.
- Creating and customizing context-based reports is difficult.
- Once ingested, data cannot be easily used to derive insights beyond the scope of security.
- Deployment is complex and time-consuming, with timelines in excess of two months.
- Options for custom or external event enrichment and data validation are limited or nonexistent.
Next Gen SIEM
- Real-time threat detection: Malicious activity can be detected as soon as it begins, and appropriate measures can be taken to limit the resulting damage or negate it entirely.
- Improved decision-making and predictions: The accuracy of modern solutions improves over time by “learning” from wrong decisions. These metadata systems are often further enriched to improve decision quality. Preparing for the future gives businesses a competitive edge, providing a more flexible framework for decision-making and risk handling.
- Enhanced security posture: Using historical data, organizations can understand what is typical and what isn’t. By detecting these outliers and prioritizing security concerns correctly, an organization can improve its overall security posture.
- Extended data storage: Storing more data for a longer period means there’s more time to analyze patterns and proactively plan for future security incidents.
- Data lake-based storage: unlimited scalability at low cost, with improved fault tolerance and high throughput, is perfect for organizations experiencing issues with centralized data management.
SIEM deployment models
- Where do you want to deploy your SIEM platform? (Cloud, on-premise or hybrid)
- Will an in-house security team be monitoring and managing your SIEM solution?
In this scenario, an in-house team takes care of both deployment and management. Purchasing, hardware provisioning and data archival are also typically handled internally.
Managed security services (MSS)
This is the most widely adopted model. Purchasing and provisioning hardware for are handled internally, while. other activities (deployment, event collection setup, correlation, analysis, alerting and dashboards) are handled by your MSS partner.
The Evolution Of SIEM
Problems Associated With Traditional SIEM Platforms
Lack of integration
The volume and variety of data that today’s SOCs handle can be a challenge for traditional SIEM solutions. Data often arrives in inconsistent formats, since SOCs often use products from numerous vendors.Additionally, modern security teams are increasingly leveraging external data sources as a supplement to their internal data. Many traditional solutions lack the integration support they need to make use of this external data, forcing analysts to spend time manually correlating separate data sets.
Reliance on human intervention
One of the most significant drawbacks of traditional SIEM systems when compared to their modern counterparts is their reliance on human intervention: they can ingest data and generate alerts, but they stop short of taking any action themselves.This places a tremendous burden on the shoulders of the SOC team, who must manually respond to each and every alert. In today’s high-volume SOCs, analysts working with traditional solutions can quickly find themselves buried under a mountain of alerts as a result.
No dynamic profiling
In today’s constantly shifting security landscape, signature-based detection isn’t enough. Relying too heavily on these strategies leaves networks open to more sophisticated attacks, including advanced persistent threats (APTs) and exploits involving zero-day vulnerabilities.Modern SIEM platforms incorporate statistical analysis and machine learning techniques are applied to establish baselines for what constitutes normal activity. When the software detects unusual activity, like an unusually high login frequency for a particular account, it generates an alert. These baselines are also continuously updated to avoid generating an unacceptable number of false positives.
Limited capacity for validation
In an SOC using a traditional SIEM solution, each alert requires the attention of an analyst, who must investigate manually to determine whether a threat is present at all (and what to do about it, if one exists).Over time, that translates into hours and hours of analysts’ time spent investigating threats that don’t exist, thereby increasing the time it takes for the SOC to respond to real threats, too. To help make security teams more efficient and effective, modern SIEM platforms can do some of the investigative work on their own. By taking advantage of external threat intelligence sources, like lists of malicious domains and file hashes, the software can validate many threats independently. This reduces the amount of time spent manually investigating threats and enables SOC staff to respond more rapidly to confirmed threats.
Inefficient resource use
In light of the standard set by modern SIEM platforms, traditional platforms lead to inefficient resource use at a number of levels.In a traditional setup, analysts are regularly distracted from legitimate threats by false positives. Legitimate threats, in turn, must be validated and responded to manually. This approach to security puts constant pressure on analysts, who must work quickly and without stopping to prevent a backlog of alerts from forming. By contrast, next-generation SIEM solutions are able to reduce the pressure on analysts, ensuring that the SOC’s resources can be focused where they’re needed most at any given time.
How Has SIEM EvolvedSince its introduction, SIEM systems have come a long way. Today’s platforms incorporate technologies like SOAR (security orchestration, automation and response), machine learning and artificial intelligence, reducing the amount of manual work SOC teams are forced to do and giving them the tools they need to respond effectively to a variety of threats. Significant advances have also been made in the approach SIEM solutions take to data storage and retrieval. While older solutions used relational database technologies like MySQL to store ingested data after parsing, modern solutions have shifted to using data lakes. Data lakes can include data in so-called “semi-structured” formats, such as XML and JSON, making them more flexible than relational databases. Data lake storage is also more scalable, making it easy for organizations of all sizes to get started using SIEM without investing in expensive dedicated storage hardware.
Incident Response and Automation With SIEM
What is incident response?Incident response includes a variety of activities whose purpose is to detect, investigate, and handle security incidents in an organization. Incident response activities can be broken down into two main types: reactive and proactive.
Reactive responseReactive response activities are those begun after the presence of a threat has become apparent. Depending on the progress and nature of the incident, this may include limiting the damage an attacker can cause, denying an attacker continued access to the network, eliminating malware, restoring from backups, patching vulnerabilities, and so on.
Proactive responseProactive response activities are those begun before the presence of a threat is apparent. In contrast to reactive response, proactive response is about actively searching for signs of a cyberattack or the means by which an attack could occur. For example, network traffic can be monitored to discover the presence of a backdoor before an attacker is able to exfiltrate sensitive information.
The SOC’s tools in practiceWith an extensive array of tools at its disposal, the ability to apply the right tools at the right time is critical to success in an SOC. A SIEM system is one of these tools, keeping analysts informed by aggregating information forwarded from multiple sources as it arrives. This includes internal sources of information, like firewall logs; as well as external sources of information, like threat intelligence data from third parties. Based on this information, the SIEM system generates alerts to let security staff know about suspicious activity that may require their attention. In a modern SOC, it is common for analysts to be faced with a high volume of alerts, which makes triage essential. In this context, triage is the process of assessing and prioritizing alerts based on their severity. Some alerts will also be false positives — identifying them as such early on is key to ensuring efficiency and keeping valuable SOC resources available to handle any critical incidents that may arise.
Tying It All Together: Incident Management and Collaboration
The first component of SOAR is orchestration. Orchestration involves retrieving and/or generating data about a potential incident in order to evaluate it:
- Some of the needed information will typically already exist locally, like log entries ingested by a SIEM solution.
- Some information may need to be retrieved. For example, connections to suspicious domains can be verified against prepared lists of known malicious domains.
- SOAR tools can also generate information themselves, such as by running a suspicious executable in a sandbox to observe its behavior.
Automation and response
If orchestration confirms the existence of a security incident, a SOAR tool can often take action without human intervention. Using sets of rules called playbooks, a SOAR tool matches the incident to one or more rules and takes the action called for by those rules.
- If a malicious file is discovered on an endpoint, for example, the endpoint can be automatically isolated from the network before any damage occurs.
- If the file arrived as an email attachment, the system can delete or quarantine other emails from the same address.
- If a user downloaded the file from a website, the system can block access to that website indefinitely.
Automated alert handling
In traditional environments, analysts were usually forced to manually triage and investigate every alert generated by the SIEM system. Now, systems with integrated SOAR capabilities can automatically triage alerts.By correlating internal and external information, false positives can often be sorted out automatically, greatly enhancing efficiency in SOCs that deal with high volumes of alerts. Validated security incidents can be forwarded to case management with more contextual information in place, so security staff can spend more time responding to incidents and less time poring over pages of log data.
Journaling and record-keeping
Modern SIEM solutions can keep records of every investigation from start to finish. These records can become a historical reference for your organization, providing an overview of the types of incidents encountered previously and how they were handled.Besides functioning as the SOC’s own self-writing history book, SIEM records can be used as part of a regulatory compliance strategy.
Advanced SIEM Capabilities: UEBA and SOAR
UBA and UEBAUEBA involves observing the normal conduct of users and entities, and detecting any anomalous behavior by looking for deviations from established, “normal” patterns. UEBA evolved out of UBA (user behavior analysis), which was limited to analyzing user activity. Soon, the need to apply behavior analysis to security events became too great to ignore, which gave rise to UEBA solutions.
Advanced Applications of UEBAWhile most SIEM platforms offer some basic functionality in this area, traditional solutions are limited in their ability to account for varying patterns of behavior from one attacker to the next. Fortunately, the UEBA features of modern SIEM solutions are markedly more effective at addressing these threats. Using statistical models powered by machine learning and artificial intelligence, next-generation SIEM platforms can establish dynamic baselines for what constitutes “normal” behavior. When the software detects a deviation from those baselines, an alert is generated, so that the security team can investigate and take action. Below, you’ll find a few examples of UEBA features and their applications:
One application of UEBA involves detecting login attempts outside of specified office hours. If some users in an organization can only be expected to log in within a given period, and a login attempt is detected outside that period, this may indicate that an insider is trying to perform a suspicious activity.It is, however, important to keep in mind that an alert produced by such activity could also be a false positive, so dynamic, constantly updated baselines are essential to making UEBA efficient. In the case of an organization with offshore clients, a user logging in at 4 A.M. may be considered a normal event. Traditional SIEM tools that with lacking UEBA support might assume that any login at that time is suspicious, generating spurious alerts and distracting analysts from other tasks..
Monitoring login/logout frequencies
By monitoring how many times users log in and out in a given time period, a UEBA-equipped SIEM platform can identify behavior likely to be associated with unauthorized account use.Using a statistical model, the software learns how frequently users typically log in and out, and this serves as a baseline for detecting unusual behavior. For example, a user who wishes to gain access to confidential information or protected resources may be able to obtain the account credentials of a more privileged user. The extra logins from the unauthorized user will be detected, generating an alert..
Monitoring dormant user accounts
Sudden activity on an account that has not been used in some time may also indicate an attempted attack. If the credentials to an old or disabled account with root privileges are leaked, for example, an attacker may attempt to use the account to infiltrate an organization’s network.Even if the login is unsuccessful, as in the case of a login attempt on a disabled account, the resulting alert from the SIEM platform serves to notify SOC staff that an attack may be in progress..
Security Orchestration, Automation and Response (SOAR)Security orchestration, automation and response — or SOAR, for short — refers to a range of powerful technologies that streamline the incident response process and help you make the most of your security data. Integration with third-party threat intelligence providers like VirusTotal and Kaspersky gives you the context you need to make informed decisions, and automation tools save your analysts time by responding to many alerts without requiring human intervention. Need a crash course on SOAR?
SOAR + SIEM: Integrated SolutionsOne example of an integrated solution is DNIF, a unified SIEM solution based on modern data lake technology. With built-in SOAR and UEBA functionality, there’s no need to spend time and money buying and configuring plugins to complete your security toolkit. To better understand how an integrated solution like DNIF works, let’s first take a look at a typical security pipeline.
Existing Cybersecurity PipelineA cybersecurity pipeline starts far before an event is classified as a threat — in fact, it starts right where events are received. Response is at the very end of the pipeline. Since most events don’t indicate the presence of a threat, relatively few events actually reach the end.. A typical pipeline within a SIEM system might look like this:
- Collection: events are received using various protocols
- Parsing: events are broken down and entered into fields
- Enrichment: contextual information is added to events
- Indexing: events are stored in the database
- Correlation: similar events are connected and analyzed
- Validation: event details are checked across multiple sources
- Response: action is taken to counter a threat
Enrich Events and Add ContextThe enrichment phase adds context to events, which makes finding correlations between events easier and more productive.. Adding geographical information and whois records are common means of enriching data. However, much more can be done with SOAR:
Data enrichmentAdvanced examples of event enrichment include:
- Layering a user’s UEBA score on top of Active Directory or proxy events
- Extracting and adding IP context from AD, and adding it to proxy events
- Aggregating third-party threat intelligence and enriching events with context
- Adding user attributes extracted from AD to database users
Validate Threats Using LookupsPreviously, correlation rules were limited to simple models and groupings. There were no opportunities to automatically validate correlation results or incorporate third-party threat intelligence. Today, SOAR can also validate correlations, ensuring that the right decisions are made in the response phase. By calling on external sources to validate decisions and add context to correlated outcomes, SOAR makes more effective validation systems possible. Examples of ways threats can be validated include:
- Validate domains / IPs / file hash etc using remote threat intelligence providers.
- Validate and contextualize event details with additional fields from either ITSM or Active Directory.
Trigger Outbound Events and Respond to AttacksResponse is the last phase in the event pipeline. Any event reaching this point has become a confirmed alert that requires action. Cross-verification of the threat has already been performed, so action must be taken to counter the threat. Traditional systems stop short of this stage, simply raising an incident or ticket on a handler’s screen. It is then the handler’s responsibility to validate and respond to the threat manually. This time-consuming approach lets attackers remain in the system longer than necessary, by modern standards. Examples of triggerable responses include:
- Raising a ticket in the ITSM application
- Notify users using email or chat apps like Google Hangouts / Slack etc
- Trigger a vulnerability scanner on an unrecognized device
- Choke a process, block disk and network access using an endpoint product
Benefits of SOAR
- Automate data enrichment: SOAR not only minimizes response times, but also enriches generated alerts.
- Security playbooks reduce manual investigation: Security playbooks can be built based on gathered intelligence or trend profiles based on suspicious IP addresses, domains or even file hashes. Automated threat validation from external threat intelligence providers makes investigation and remediation faster. This helps security analysts investigate every incident while giving them ample time to address alerts.
SIEM Use Cases
Process WhitelistingSecurity teams have long been haunted by thoughts of a threat slipping through all their defenses undetected, perhaps leaving some malware active within the network in “stealth mode.” Process whitelisting significantly reduces the risk of such a situation occurring, making it one of the most powerful applications of SIEM.
How it worksImplementing process whitelisting is simple. There are three steps:
- Scan a clean system’s folders and drives to detect the applications and processes you wish to allow. Executable files detected in this scan will be added to a whitelist.
- If necessary, modify the generated list. For example, you can remove an automatically detected application from the list.
- Enforce the whitelist, putting the system into a protected state. An application that attempts to run on your system will first be checked against the whitelist. If it’s not on the list, the application will not be allowed to run.
How process whitelisting helps
DGA DetectionDetecting the use of a domain generation algorithm is unquestionably critical. Attackers implement DGAs in malware to periodically generate a large number of domain names. Malware authors register a small set of the generated domains as means for the malware to “phone home,” potentially receiving updates or relaying information to an attacker. Repeated attempts to connect to invalid domains may indicate the presence of DGA-equipped malware, so detecting those connection attempts can be critical to stopping the spread of malware and preventing the exfiltration of sensitive data..
How it worksDetecting DGA-related network activity is summarized in the following steps:
- Monitor domain names present in log data.
- Validate or enrich domain name information with external threat intelligence sources for reputation, registration date and domain owner.
- Correlate the domain information with traffic data to identify anomalies.
- Generate alerts and/or block connections to flagged domains. This can be done automatically using built-in automated response tools.
How DGA detection helps
Detecting DDE InjectionDynamic Data Exchange (DDE) injection is another technique attackers use to infiltrate environments where Microsoft Office is used. DDE is a technology that facilitates information sharing between applications. It is sometimes used to populate Excel spreadsheets with information generated elsewhere, for example. However, the ubiquity of Microsoft Office makes it an attractive target for attackers, who have been known to use DDE as a means of executing malicious code..
How it works
- Monitor Sysmon event IDs for new process creation and registry changes.
- Monitor Microsoft Office processes and correlate the actions of these processes with the above events.
- Trigger an alert if such processes are found.
How detecting DDE injection helps
User Entity and Behavior Analytics (UEBA)Security experts must also be attuned to the risk of insider threats. Profiling user behavior makes it possible to spot suspicious account activity and respond to it preemptively, rather than waiting for a security incident to occur.
How UEBA worksUEBA profiling involves monitoring user behavior to detect anomalies. Common components of this approach include:
- Monitoring failed login attempts.
- Creating a profiler to monitor and analyze common trends related to successful logins, login/logout frequency and session lengths.
- Monitoring dormant user activity.
- Provisioning privileged access and monitoring user access.
How UEBA helps
Threat Intelligence: Data Enrichment and Threat ValidationModern SIEM solutions support integration with third-party data sources. Incorporating external threat intelligence helps to refine datasets, adding context and making it easier to validate threats.
How it worksUEBA profiling involves monitoring user behavior to detect anomalies. Common components of this approach include:
- Events forwarded to your SIEM solution are supplemented with threat intelligence from your chosen intel providers in real time.
- This data can be further used to validate threats and provide additional insight during incident investigation.
- Using integrated security automation tools, internet resources and users can be blocked automatically after a threat is validated. Alternatively, an alert can be sent to the relevant team for further investigation.