As organizations pursue their digital transformation dreams, they’ll migrate from on-premises SIEM to cloud-based SIEM. In the process of doing so, CISOs are taking a closer look at their previous security incident and event log retention policies, and revisiting past assumptions and processes.
For organizations needing to maintain a smorgasbord of industry compliance and regulatory requirements, overall event log retention will range from one year through to seven. Many organizations find that a minimum of one year meets most mandated requirements but err on the side of retaining between three to four years – depending on what their legal counsel advises.
With public cloud, data retention spans many different options, services, and price points. Backups, blob storage, “hot” access, “cold” access, etc. – there are endless ways to store and access security events and logs. With cloud storage dropping in price year-on-year, it’s cheap and easy to just store everything forever – assuming there’s no rush or requirement to inspect the stored data. But hot data, more expensive than the cold option, gives defenders the quick access they need for real-time threat hunting. Keeping data hot for SIEM use is inevitably one of the more expensive data storage options. A balance needs to be struck between having instant access to SIEM for queries and active threat hunting, and long-term regulatory-driven storage of event and log data. Can an optimal storage balance be achieved?
Widely available public threat reports for the last couple of years provide a “mean-time” to breach discovery ranging from 190 to 220 days and a breach containment window of between 60 to 100 days. Therefore, keeping 220 days of security event logs “hot” and available in a cloud SIEM would statistically only help with identifying half of an organization’s breaches. Obviously, a higher retention period makes sense – especially for organizations with less mature or less established security operations capabilities.
However, a sizable majority of SIEM-discoverable threats and correlated events are detectable in a much shorter timeframe – and rapidly detecting these breaches naturally makes it considerably more difficult for an adversary to maintain long-time persistence. For example, automatically piecing together the kill chain for an email phishing attack that led to a malware installation, that phoned home to a malicious C&C, which had then brute-forced the administrative access to a high value server is almost trivial for cloud SIEM (assuming appropriate logging was enabled). Nowadays, such a scenario (or permutation of that scenario) likely accounts for near half of all enterprise network breaches.
My advice to organizations new to cloud SIEM is to begin with a rolling window of one year’s worth of event logs while measuring both the frequency of breaches and time to mitigate. All older event logs can be stored using cheaper cloud storage options and needn’t be immediately available for threat hunting.
Depending on the security operations teams’ capacity for mitigating the events raised by cloud SIEM, it may be financially beneficial to reduce the rolling window if the team is overwhelmed with unresolvable events. I’d be hesitant to reduce that rolling window. Instead, I would recommend CISOs with under-resourced teams find and engage a managed security services provider to fill that skills gap.
A question then arises as to the value of retaining multiple years of event logs. Is multi-year log retainment purely a compliance tick-box?
While day-to-day cloud SIEM operations may focus on a one-year rolling window, it can be beneficial to organize a twice-annual threat hunt against several years of event logs using the latest available threat intelligence and indicator of compromise (IoC) information as seeds for investigation. These periodic events have two objectives: reduce your average monthly cloud SIEM operating costs (by temporarily loading and unloading the historic data) and allow teams to change mode and “deep dive” into a broader set of data while looking for “low and slow” compromises. If an older breach is detected, incrementally older event logs could be included in the quest to uncover the origin point of an intruder’s penetration or full spectrum of records accessed.
Caution over infinite event log retention may be warranted, however. If the breached organization only has a couple years of logs, versus being able to trace breach inception to, say, four years earlier, their public disclosure to customers may sound worse to some ears (including regulators). For example, disclosing “we can confirm customers over the last two years are affected” is a weaker disclosure than “customers since July 4th 2015 are affected”. Finding the sweet-spot in log retention needs to be a board-level decision.
Having moved to cloud SIEM, CISOs also need to decide what logs should be included and what log settings should be used.
Ideally, all event logs should be passed to the cloud SIEM. That is because the AI and log analytics systems powering threat detection and automated response thrive on data. Additionally, inclusion of logs from the broadest spectrum of enterprise devices and applications will help reduce detection times and remove potential false positives, which increase overall confidence in the system’s recommendations.
Most applications and networked appliances allow for different levels of logging, including scaling from error messages to alerts and error messages through to errors, warnings, status messages, and debugging information. In general, the greater the detail in the event logs, the greater the value they bring to cloud SIEM. In this way, upgrading from “normal” to “verbose” log settings can offer several threat response advantages – particularly when it comes to handling misconfigurations and criticality determination.
The symbiotic development of cloud SIEM and cloud AI innovation continues at an astounding pace. While cloud SIEM may be new for most organizations, its ability to harness the innate capabilities of public cloud are transforming security operations. Not only are threats being uncovered quicker and responses managed more efficiently, but continual advancements in the core AI makes the technology more valuable while costs of operating SIEM and storing data in the cloud continue to drop. This makes it possible for companies to make pragmatic use of the intelligent cloud by operating on a one-year window of hot data while getting value out of older data, stored cold, on twice a year threat hunts.
-- Gunter Ollmann
First Published: SecurityWeek - July 22, 2019
