Getting Your SOC SOARing Despite AI

It’s a fact: enterprise security operations centers (SOCs) that are most satisfied with their investments in Security Information and Event Management (SIEM) and Security Orchestration, Automation, and Response (SOAR) operate and maintain less than a dozen playbooks. This is something I’ve uncovered in recent years whilst building SIEM+SOAR and autonomous SOC solutions – and it perhaps runs counterintuitive to many security leaders’ visions for SOAR use and value.

SOAR technology is one of those much-touted security silver bullets that have tarnished over time and been subsumed into broader categories of threat detection and incident response (TDIR) solutions, yet it continues to remain a distinct must-have modern SOC capability.

Why do satisfied SOCs run so few playbooks? After all, a core premise of SOAR was that you can automate any (perhaps all) security responses, and your SOC analysts would spend less time on ambient security noise – giving them more time to focus on higher-priority incidents. Surely “success” would include automating as many responses as possible?

Beyond the fact that “security is hard,” the reality is that threat detection and response is as dynamic as the organization you’re trying to protect. New systems, new policies, new business owners, new tools, and a sea of changing security products and API connector updates mean that playbooks must be dynamic and vigilantly maintained, or they become stale, broken, and ineffective.

Every SOC team has at least one playbook covering their phishing response. It’s one of the most common and frequently encountered threats within the enterprise, yet “phishing” covers an amazingly broad range of threats and possible responses, so a playbook-based response to the threat is programmatically very complex and brittle to environmental changes.

From a SOC perspective of automating and orchestrating a response, you would either build a lengthy single if/then/else-stye playbook or craft individual playbooks for each permutation of the threat. Smart SOC operators quickly learn that the former is more maintainable and scalable than the latter. A consequence of this is that you need analysts with more experience to maintain and operate the playbook. Any analyst can knock-up a playbook for a simple or infrequently encountered threat vector, but it takes business knowledge and vigilance to maintain each playbook’s efficacy beyond the short term.

Surely AI and a sprinkle of LLM magic will save the day though, right?

I can’t count the number of security vendors and startups that have materialized over the last couple of years with AI and LLM SOAR capabilities, features, or solutions – all with pitches that suggest playbooks are dead, dying, replaced, accelerated, automatically maintained, dynamically created, managed, open sourced, etc., so the human SOC analyst does less work going forward. I remain hopeful that 10% of any of that eventually becomes true.

For the immediate future, SOC teams should continue to be wary of any AI stories that emphasize making it easier to create playbooks (or their product-specific equivalent of a playbook). More is NOT better. It’s too easy (and all too common) to fall down the rathole of creating a new playbook for an existing threat because it’s too hard to find and maintain an earlier iteration of that threat’s playbook. Instead, focus on a subset of the most common and time-consuming threats that your SOC already faces daily, and nail them using the smallest number of playbooks you can get away with.

With the Rolling Stones’ “(I Can’t Get No) Satisfaction” playing in the background, perhaps you’ll get some modicum of SIEM+SOAR satisfaction by keeping your playbook playlist under a dozen.

-- Gunter Ollmann

First Published: IOActive Blog - October 15, 2024

Why Cybersecurity is Critical in MLOps

Larger and more sophisticated businesses will lean into building out their in-house data science teams and capabilities

If your business relies on machine learning (ML) to drive strategic decision-making, you’re in good company. A recent report by ClearML shows the technology clearly entering the mainstream, with 60% of organizations’ ML leaders planning to increase ML investments by more than a quarter in 2023. The same study revealed that 99% of respondents either already have dedicated budgets for ML operations (MLOps) or plan to implement them this year.

But, as MLOps mature, they also carry more risk. According to a recent study by NCC Group, organizations are deploying ML models in more applications without considering security requirements. In a separate survey by Deloitte, nearly two-thirds of AI and ML users describe cybersecurity risks as a significant or extreme threat, but only 39% feel prepared to combat those risks.

MLOps model creation pipelines are vulnerable and easily attacked in three separate ways: by malicious insiders, through software supply chain manipulation, and via compromised systems. If the SolarWinds supply chain attack taught the industry anything, it’s that continuous build processes are both a target for sophisticated adversaries and a blind spot for in-house security operations teams.

In 2023, continuous build processes will continue to be a target for threat actors. As these attacks start to impact enterprises’ bottom lines, they will have to start paying more attention to the cybersecurity side of MLOps.

Here are some ways to make MLOps projects safer and more secure.

Secure the Whole Pipeline

Part of the challenge of securing MLOps is the sheer length and depth of typical machine learning pipelines. They include a half dozen or more phases – data collection and preparation, along with the creation, evaluation, optimization, deployment and usage of an ML model. Vulnerabilities can crop up at any point in the process.

Early on, in data collection, threat actors can taint the data, manipulate the annotation or conduct adversarial attacks on the metadata stores. In later phases, open-source models and frameworks can include hidden vulnerabilities. Potential bias and system performance need to be addressed. And as models are deployed and used, new data is often introduced, expanding the attack surface and opening an organization up to all kinds of threats – including evasion attacks, model theft, code injections and privacy attacks.

At the tail end of the process, there’s a lot of intellectual property inside an ML model. Decades of transactional data and learnings from financial models that are built and trained into models may only be 10s of kilobytes in size. It’s easier to steal that model than to steal the actual source data.

These models tend to be exposed. Attackers have become skillful at querying models and reproducing them somewhere else. This requires a new way of thinking about the value of the model. Tooling and alerting not only around the theft of data but around the manipulation of the models is important to an overall MLOps security strategy.

Invest in Tooling to Scale Across SOCs

It’s no secret that security is no longer siloed in a single department. It cuts across all functions, and organizations are creating Security Operations Centers (SOCs) to improve the visibility, manageability and auditing of their overall security posture. To extend the SOC’s capability to MLOps, organizations need to incorporate tooling that scales to much larger uses than ever before.

Meeting MLOps’ data needs forces SOCs to adapt in two ways. Existing SOC operations teams are now accountable, forcing them to build the additional tooling and reporting to support MLOps teams from a security perspective. Plus, MLOps teams that are specialized in data science curation are able to leverage larger toolsets – including logging analytics platforms that provide higher levels of threat detection.

Double Down on Security Best Practices

Some of the best defense tactics for MLOps are practices organizations deploy regularly across the rest of their operations. A zero trust security policy requires the authentication and authorization of anybody trying to access applications or data used in the development of ML models. It also tracks their activity. Applying the principle of least privilege (PLoP) limits users’ access to the exact data sets and models they are authorized to touch. This reduces the attack surface by prohibiting hackers who have gained access to one data trove from moving freely throughout the system.

Use Analytics to Observe and Log ML Tasks

An important step in protecting an ML system is to understand the system’s behavior in healthy and unhealthy states. To do this, organizations need to set up alerts that trigger action before an incident occurs. This is called “observability.” A vulnerability introduced early in the training data will affect the model’s performance down the line. Tracking performance data and logging metrics of ML tasks gives organizations insights into any and all security issues that could affect the ML model.

Future Monitoring of Model Development Lifecycle

The continuous lifecycle of MLOps necessitates the continuous monitoring of a deployed model’s response to adversarial manipulation and corruption. In the future, expect to see larger and more sophisticated businesses lean into building out their in-house data science teams and capabilities, detecting threats with security analytics, and pruning and filtering data from unknowns to improve the advancements in the next generation of ML and AI.

-- Gunter Ollmann

First Published: Security InfoWatch - June 20, 2023

Why the SOC Needs to Learn from the Aviation Industry

The cybersecurity industry has spent a lot of time talking about improving the analyst experience while not making significant improvements, as much of the efforts have been too focused on finding a silver bullet solution. Combine that with a global pandemic and now things are just getting worse. A recent study published by Devo, the 2021 SOC Performance Report, found that on a 10-point scale, where 10 indicates SOC staff have a “very painful” experience performing their jobs, 72% of respondents rated the pain of SOC analysts at a 7 or above.

Instead of thinking about the aforementioned silver bullet to alleviating SOC pain, I wanted to focus on one of the top sources, alert fatigue, and how the cybersecurity industry might be able to take a page out of another field to find a solution.

In the SOC Performance Report, a whopping 61% said a cause of SOC pain was that there are too many alerts to chase. I think it’s safe to draw the connection that “alert fatigue” will expand to “posture fatigue” and “policy fatigue,” as it adversely affects both recruitment and all too critical retention of experienced SOC professionals.

Alert fatigue may exit the aircraft

So, if we can’t figure out within the security industry, let’s learn from others. There are many non-cyber industries and professions that suffer similarly with alert fatigue, and perhaps the cybersecurity industry can reapply some of those learnings. Across these compatriots of alert fatigue, if we ask the question “how do alarms, warnings, and alerts differ?” I think we’ll find much similarity and overlap in answers — in both the theory and practice of how human operators are supposed to respond and how they do so in reality.

For the purpose of this article, I want to take a look at the aviation industry as our example to the SOC. They have navigated many of the problems SOC operators face today and have made the most progress in governing and managing the ergonomics of sensory overload and automation. Picture this: the inside of an airplane cockpit with all its knobs, buttons, lights, and alerts isn’t too dissimilar to the combined dashboards SOC analysts have to navigate when triaging, investigating, and responding to threats.

In 1988, The Washington Post reported on a “glass cockpit” syndrome in the aviation industry, that reads eerily similar to what many say or think about the SOC today. Researchers from the American Psychological Association noted that pilots would “fall victim to information overload and ignore the many bits of data pouring from myriad technical systems,” and that in airline crashes they studied it was found that “black box recordings showed that the crews talked about ‘how the systems sure were screwed up’ but did not verify what was wrong. In both cases, the systems worked but crews failed to check the information and crashed.”

Similarly, research published in 2001 by the Royal Institute of Technology examined “the alarm problem” in aviation, meaning, “in the most critical situations with the highest cognitive load for the pilots, the technology lets you down.” The reports noted that “the warning system of the modern cockpits are not always easy to use and understand. The tendency is to overload the display with warnings, cautions and inoperative system information accompanied by various audio warnings.” It went on to identify one of the main problems as a result of this overload as “a cognitive problem of understanding and evaluating from the displayed information which is the original fault and which are the consecutive faults.” Sound familiar? You would likely hear something extremely similar from someone working in today’s SOC.

In the decades that followed, aircraft cockpit design has progressively applied new learnings and automation to dynamically manage alert volume and the attention of the pilot to priorities. In the Royal Institute of Technology’s report, researchers identified accident simulation as an effective tool for improving cockpit alert systems, finding more associable ways to present alerts such as differentiating sounds and the introduction of context, which would allow pilots to “immediately understand what part or function of the aircraft is suffering a malfunction.” More context would also include guidance on what to do next. In its conclusion the study noted:

Such simulations would hopefully result in less cognitive stress on behalf of the pilots: they would know that they have started to solve the right problem. They would not have to worry that they have entered the checklist at the wrong place. With a less stressful situation even during malfunctions there is greater hope for correct actions being taken, leading to increased flight safety.

SOC systems need to embrace and apply many of these same learnings that have spanned decades for aviation. The majority of the cybersecurity industry seems to have only gotten as far as color coding alert and warning significance, leaving the analyst faced with a hundred flashing red priorities, even after triaging it. It’s no surprise that they’re both overwhelmed and unable to respond to complex threats across a broadening attack surface.

Beware of Autopilot

When it comes to solving the issue of alert fatigue, automation is typically one of the first things to come to mind. The same went for aviation in 1988, where the previously mentioned Washington Post report quoted researchers saying what could have been taken right from a security trade publication in 2022:

Research is badly needed to understand just how much automation to introduce — and when to introduce it — in situations where the ultimate control and responsibility must rest with human operators, said psychologist Richard Pew, manager of the experimental psychology department at BBN Systems and Technologies Corp. in Cambridge, Mass.

“Everywhere we look we see the increasing use of technology,” Pew said. “In those situations where the operator has to remain in control, I think that we have to be very careful about how much automation we add.”

The growing use of high-tech devices in the cockpit or on ships can have two seemingly contradictory effects. One response is to lull crew members into a false sense of security. They “regard the computer’s recommendation as more authoritative than is warranted,” Pew said. “They tend to rely on the system and take a less active role in control.” Sometimes crews are so mesmerized by technological hardware that they are lulled into what University of Texas psychologist Robert Helmreich calls “automation complacency.”

And while automation of course has an important part to play in incident response and investigation — just as it does in modern aircraft cockpit design — it comes with some key warnings:

1. Situational awareness is lost. Automation is often brittle, unable to operate outside of the situations it is programmed for, and subject to inappropriate performance due to faulty sensors or limited knowledge about a situation.
2. Automation creates high workload spikes (such as when routine changes or a problem occurs) and long periods of boredom (in which attention wavers and response to exceptions may be missed). If you’re staffing for automation-level activities, how do you manage capacity for spikes?

The SOC Earns its Wings

As an industry we have to take a page from the aircraft handbook and avoid increasing cognitive demands, workload and distractions, and make tasks easier to perform. But we must also understand how to manage automation failure and exceptions better.

• Embrace AI and autocomplete: Like the more advanced sentence autocomplete functions appearing in email and word processing applications, SOC analysts are still in charge of managing an incident, but there is an opportunity to further guide and preemptively enrich a threat investigation, thereby increasing the speed and robustness of response.
• Distill and prioritize at the incident level, not the alert level: It’s not about filtering/correlating/aggregating alerts, it’s about contextualizing both events and alerts in the background and only articulating an incident in plain single-sentence language. Analysts can double-click down from there.
• Leverage a community of experts: As attack surfaces increase and vertical technology specialization becomes tougher for in-house SOCs to cover (particularly in times of competing incident prioritization), it becomes increasingly important to be able to “phone-a-friend” and access an on-demand global pool of expert talent. It’s like having several Boeing engineers sitting in the cockpit with the pilot to troubleshoot a problem with the plane.

-- Gunter Ollmann

First Published: Medium - March 29, 2022

The Cusp of a Virtual Analyst Revolution

Security Analytics and Threat Investigation Are in the Midst of a Sea Change

Once live stomping around vendor-packed expo halls at security conferences returns, it is highly probable that “Virtual Analyst” will play a starring role in buzzword bingo. Today, the loosely defined term represents an aspiration for security vendors and managed service providers but may be perceived as a threat by internal day-to-day security operations and threat hunting teams.

For context, security analytics and threat investigation are in the midst of a sea change. Cloud log analytics platforms now enable efficient and timely analysis of ever-increasing swathes of enterprise logs, events, and alerts dating back years. Threat Intelligence platforms are deeply integrated into cloud SIEM solutions—enabling both reactive and proactive threat hunting and automated incident investigation—and are entwined with a growing stack of sophisticated AI and ML capabilities. However, smart event correlation and alert fusion engines automatically triage the daily deluge of suspiciousness down to a manageable stack of high-priority incidents—replete with kill-chain reassembly and data enrichment.

In many environments the traditional tier-one security analyst responsibilities for triaging events (removing false positives and “don’t care” noise) and maintaining operational health of scale-limiting SOC systems (e.g., device connectors, log retention and storage parameters, ticket response management) have already been subsumed by modern SIEM solutions. Meanwhile, platform-native no-code/low-code-powered orchestration and automation capabilities, along with growing libraries of community-sourced investigation and response playbooks, have greatly accelerated incident response and efficacy for tier-two analysts—alleviating time-consuming repetitive tasks and increasing focus on new and novel incidents.

Arguably, the Virtual Analyst is already here—captured within the intelligent automation and efficiencies of modern cloud SIEM— and I believe the journey has just begun.

The near future evolution of the Virtual Analyst is being driven by two competing and intwined motions —the growing need for real-time threat response, and the inaccessibility of deep security knowledge and expertise.

Real-time threat response has long been thought an achievable target for in-house security operations teams and has underpinned many historic CISO security purchasing decisions. As the enterprise attack surface has grown, adversaries (external and internal) have increased the breadth and pace of attack, and in response businesses continue to invest heavily in instrumenting their environments with an “assume breach” mindset—widening the visibility aperture and exponentially increasing the volume and timeliness of threat-relatable data. Advanced log analytics capabilities and AI-powered event fusion processes are identifying more incidents earlier along the kill-chain and consequently providing more opportunities to conditionally mitigate a budding threat or disrupt a sequence of suspicious events. 

To successfully capitalize on that shrinking window of opportunity, responses need to occur at super-human speeds. The speed bump introduced by requiring a human in that response loop will increasingly materialize as the difference between having been attacked versus being breached. In this context, the Virtual Analyst represents the super-human capabilities AND responsibilities for real-time threat identification AND trusted automated mitigation of a live incident.

Although that Virtual Analyst capability will be tightly bound to a product (e.g., Cloud SIEM, SOC-as-a-Service), the second Virtual Analyst motion centers around access to deep security expertise.

If a product-bound Virtual Analyst can be considered a quick-learning high-speed generalist, the second motion can be thought of as a flexible “on-call” specialist—augmenting the security operations team’s investigative and response capabilities as needed—and may be conceptually akin to the on-demand specialist services provided by traditional managed security service and incident response providers. 

The differentiated value of cloud-based Virtual Analyst solutions will lie in leveraging broader internet-spanning datasets for threat detection and attribution, and powerful, rapid, ad hoc forensic-level investigation of incidents and response. For example, the in-house SOC team may engage the Virtual Analyst to augment an ongoing investigation by temporarily connecting it to their on-premises SIEM, and receive targeted direction for capturing and collecting incident-relevant non-SIEM data (e.g., PCAPs, VM images, storage snapshots, configuration files) that are uploaded and automatically investigated by the virtual analyst as well as incorporated for real-time instruction on system recovery and attack mitigation.

It’s tempting to think that on-premises security analysts’ days are numbered. Virtual analyst advancements will indeed increase the speed, fidelity, and efficacy of threat detection and incident response within the enterprise—replacing almost all repeated and repeatable analyst tasks. But AI-powered virtual analyst solutions will do so with little knowledge or context about the business and its priorities. 

With the day-to-day noise and incident investigation drudgery removed, security operations teams may evolve into specialist business advisors—partnering with business teams, articulating technology risks, and providing contextual security guidance.

-- Gunter Ollmann

First Published: SecurityWeek - March 23, 2021

Reinventing Managed Security Services’ Detection and Response

Managed security services are undergoing a timely and significant transformation, armed with new hyper-scalable technology stacks, hybrid enterprise and cross-cloud protection complexities, and a demand to evolve from 24/7 eyes-on-glass into hands-on customer-integrated early warning and response. If it wasn’t a tired industry cliché we’d probably be adding “next-generation” or NG prefixes to many of these newly transformed managed services.

That transformation of traditional managed security services provider (MSSP) offerings, combined with an explosion of software product vendors and consulting services providers entering the fray with their newly hybridized managed detection and response solutions, is confusing to many. Whether from an MSSP, security product vendor or a consulting services provider pitch, the same vocabulary and acronyms increasingly mean different things.

Endpoint protection platforms (EPP) have evolved over the past decade from alert-generating megaphones into standalone, powerful endpoint detection and response (EDR) solutions. Key to that evolution is the incorporation of progressively advanced machine learning threat detection capabilities and automated remediation. Although EDR continues to grow smarter and more capable, a diverse portfolio of managed services is being welded to them to enhance threat protection and remedy capabilities.

As such, in broad strokes EDR solutions (and the market in general) is evolving into managed detection and response (MDR). 

For product vendors, MDR is tied to advanced manageability and proprietary services almost exclusively built around enhancing their own stand-alone EDR product (and may extend to data ingestion and analytics of integrated partnered products). For MSSPs and consulting services providers, MDR typically refers to the addition of human-led detection and response capabilities provided 24/7 and layered upon EDR suites from one or more vendors.

Recently the term XDR (extended detection and response) has become a catch-all for combined endpoint, network, and cloud-based detection and response. MSSPs tend to imply that “XDR” is a managed service and often a component of an outsourced and managed SOC (security operations center) offering. 

Tight product integrations and security ecosystem fusions with automation have made it exponentially easier to provide managed detection and response across a broader range of security products and technologies — and easier for service providers to offer highly scalable and managed XDR (or EDR, NDR or MDR) detection and response solutions. 

Meanwhile, some software and SaaS vendors have launched stand-alone XDR products that aggregate detection alerts and automate response across multivendor EDR, NDR (Network Detection and Response) and CSPP (Cloud Security Protection Platform) products — enabling third-party human specialists to transform their XDR product into a managed service. 

It can feel like a Monty Python sketch when speaking with a product vendor: XDR and managed XDR (MXDR) are different solutions, but EDR and MDR may mean the same thing because vendor-provided management is often part of the product purchase subscription. For an MSSP, managed EDR is different from MDR, but MDR and XDR may be the same thing. 

The inclusion and advancement of machine learning is the key ingredient to modern managed detection and response solutions. For example, supervised and deep learning methods play such a fundamental role in bulky detection triage processes that they’ve effectively eliminated the traditional mind-numbing tier-one security analyst role. Meanwhile, Natural Language Processing (NLP) and anomaly clustering, along with no-code playbook automation, is simplifying threat hunting and response — removing the daily grind tier-two analysts tend to face. “Virtual analyst” is the term we’ll hear with growing regularity.

MSSPs and EDR vendors may have commenced their detection and response journeys from different starting points, but they are converging to roughly the same solution and reinventing the managed security services market along the way. Virtual analyst technology (and the advances in ML and AI that lay behind its efficacy) will assuredly drive further innovation in managed services, with the next click-stop on this journey likely being autonomous SOC (or SOC-as-a-Service).

-- Gunter Ollmann

First Published: SecurityWeek - Feb 9th, 2021

Enterprise Threat Visibility Versus Real-World Operational Constraints

The phrase “assume breach” has been transformational to enterprise security investment and defensive strategy for a few years but may now be close to retirement. 

When the vast majority of information security expenditure was focused on impermeable perimeter defenses and reactive response to evidence-based compromise, it served as a valuable rallying cry for organizations to tool their enterprise for insider-threat detection, adopt zero-trust network segmentation, and pursue widespread deployment of multifactor authentication systems and conditional access controls.

Sizable investments in enterprise-wide visibility should have reversed the much older adage “a defender needs to be right all the time, while the attacker needs to be right only once” into something like “an attacker needs to be invisible all the time, while the defender needs them to slip up only once.” Unfortunately, security operations and threat-hunting teams have found that instead of automatically spotting needles in a haystack, they must now manage haystacks of needles—if they’re properly equipped. For under-resourced security teams (which appears the majority), advances in enterprise-wide visibility have in the best case added hundreds of daily alerts to their never-completed to-do lists.

As security budgets have morphed, a higher percentage of spend has been allocated to increasing visibility on the premise that more threats will be preemptively detected, blocked, and mitigated.

An appropriate analogy for the situation would be installing dozens of video cameras in and around your home with overlapping fields of view and relying on that as the primary alerting mechanism for preventing break-ins. The primary assumption is that someone will be continually monitoring all those video feeds, will recognize the build up and execution of the break-in, and can initiate a response to stop the thief. 

The consequences of such a strategy (by way of continuing the analogy) are pretty obvious:

1. Because 24/7 monitoring is expensive, automated detection is required. Automatic detection comes at the cost of high false-positive rates and baseline tuning; in home CCTV terms, ignoring the rabbits, golf balls, and delivery men that cross a field of vision, while desensitizing movement thresholds and setting up hot zones for alerting. Even rarish false positive events such as lighting strikes during a storm or the shadow of a passing airplane are unfortunately enough to fill an inbox or message tray and result in wariness delays and wasted investigative cycles. To counter the problem, use at least two disparate and independent detection technologies to detect and confirm the threat (for example, CCTV movement zones and a break-glass sensor).
2. Automatic detection without an automatic response limits value to post-break-in cleanup and triage—not prevention. Because of potential false positives, automatic responses also need to be reversible throughout the period of alert response. If CCTV movement and break-glass sensors are triggered, perhaps an automatic request for a patrol car visit is initiated. Meanwhile the original alert recipient can review footage and cancel the callout if it was clearly a false positive (e.g., the neighbor’s kids kicked a ball over the fence and broke a window).
3. Balance between detection and prevention is critical and will change over time. 24/7 CCTV monitoring may serve as a key detection capability, but locking all external doors with deadbolts shouldn’t be neglected. Deadbolted doors won’t stop the future threat of a $50 miniature drone flying down the chimney and retrieving the spare front-door key laying on the kitchen table. Prevention investments tend to be threat reactive, while modern detection technologies tend to be increasingly successful in identifying behavioral anomalies.

“Assume breach” served its purpose in changing the ways organizations thought about and invested in their security technologies (and operational programs). As with many well-intentioned initiatives, the security pendulum may have swung a little too far and now needs a balanced redressing.

Although I think cloud-SIEM and the advanced machine intelligence platforms being wedded to it will eventually meet most organizations’ 24/7 visibility and detection needs, SecOps teams will continue to battle against both alert fatigue and posture fatigue. The phrase I’d like to see the industry focus on for the next five years is “automatically mitigated.”

-- Gunter Ollmann

First Published: SecurityWeek - September 17, 2020

Retooling Cyber Ranges

Cloud-based Cyber Ranges Will Change the Future of Training and Certifying Security and DevOps Professionals

A half-decade ago, with much fanfare, cyber ranges were touted as a revolutionary pivot for cybersecurity professionals’ training. Many promises and investments were made, yet the revolution has been slow coming. What may have been a slow start appears to be picking up speed and, with the accelerated adoption of work-from-home business practices, may finally come of age.

The educational premise behind almost all cyber range training platforms is largely unchanged from decades-old war-gaming and capture the flag—nothing beats hands-on practice in refining attack and defense strategies or building responder muscle memory. Carefully scripted threat scenarios guide the training program—often gamifying the experience with mission scores and leaderboards. Many of the interfaces and scenario scene-setting often appear like they came from the imagination of developers who grew up on a diet of 1990’s video games like Command & Conquer; the militaristic adversary overtone is strong yet adds positively to the immersive experience for users.

For many years, gamified security training has required significant infrastructure investment by the provider—investments capable of replicating the complex environments of their customers and the apparatus to generate realistic network traffic. Like the customers that subscribe, cyber-range platforms are undergoing their own digital transformation and moving to the cloud—ephemeral virtual environments, dynamic scaling to the number of participants, global anytime delivery, etc., are all obvious advantages to building and running cyber ranges within the public cloud.

What may be less obvious is how cloud-based cyber ranges will change the future of training and certifying security and DevOps professionals.

Some of the changes underway (and maybe a couple years down the road for mainstream availability) that excite me include:

• At-home cyber-range training and hands-on mastery of operational security tasks and roles. Past cyber-range infrastructure investments necessitated classroom-based training or regional traveling roadshows. Cloud-based cyber ranges can remove the physical classroom and scheduling constraints—offering greater flexibility for employees to advance practical skills at their own pace and balance time investments against other professional and personal commitments. I’m particularly encouraged with the prospect of delivering a level field for growing and assessing the practical skills and operational experiences of security professionals coming from more diverse backgrounds.
• Train against destructive scenarios within your own business environment. As businesses run more of their critical systems within the cloud, it becomes much easier to temporarily spin up a clone, mirror, or duplicate of that environment and use it as the basis for potentially destructive training scenarios. Cyber ranges that apply threat scenarios and gamify the training regime for users across the replicated workloads of their customers significantly increase the learning value and response applicability to the business.
• Shift-left for security mastery within DevOps. Cyber range environments and the scenarios they originally embraced focused on security incident responders and SOC operators—the traditional Blue Team members. With security becoming a distributed responsibility, there is a clear need to advance from security awareness to hands-on experience and confidence for a broader range of cyber-professional. Just as SIEM operations have been a staple of cyber ranges, a new generation of cyber-range platforms will “shift left” to replicate the complex CI/CD environments of their customers—enabling DevOps teams to practice responding to zero-day bugs in their own code and cascading service interruptions, for example.

It will be interesting to see how enterprise SOC leaders will embrace SecOps teams that trained and certified via cyber ranges at home. I’m sure many CISOs will miss the ability to escort senior executives, investors, and business partners around a room filled with security professionals diligently staring at screens of graphs and logs, and a wall of door-sized screens showing global pew-pew animated traffic flows. 

There is a difference between a knowledge certificate and the confidence that comes with hands-on experience—and that confidence applies not only to the employee, but to their chain of command.

The coming of age for cyber ranges is both important and impactful. It is important that we can arm a greater proportion and more diverse range of cyber-professionals with the hands-on practical experience to tackle real business threats. It is impactful because cyber-range scenarios provide real insights into an organization’s capabilities and resilience against threats, along with the confidence to tackle them when they occur.

-- Gunter Ollmann

First Published: SecurityWeek - March 31, 2020

Defending Downwind as the Cyberwar Heats up

The last few weeks have seen a substantial escalation of tensions between Iran and the US as regional cyberattacks gain pace and sophistication with Iran’s downing of a US drone, possibly leveraging its previously claimed GPS spoofing and GNSS hacking skills (to trick it into Iranian airspace) and a retaliatory US cyberattack knocking out Iranian missile control systems. 

While global corporations have been targeted by actors often cited as supported by or sympathetic to Iran, the escalating tensions in recent weeks will inevitably bring more repercussions as tools and tactics change with new strategic goals. Over the last decade, at other times of high tension, sympathetic malicious actors have often targeted the websites or networks of Western corporations – pursuing defacement and denial of service strategies. Recent state-level cyberattacks show actors evolving from long-cycle data exfiltration to include tactical destruction.

State sponsored attacks are increasingly focused on destruction. Holmium, a Middle Eastern actor, has been observed recently by Microsoft to target oil & gas and maritime transportation sectors – using a combination of tactics to gain access to networks, including socially engineered spear phishing operations and password spray attacks – and are increasingly associated with destructive attacks.

Many businesses may be tempted to take a “business as usual” stand but there is growing evidence that, as nation state cyber forces square off, being downwind of a festering cyberwar inevitably exposes organizations to collateral damage. 

As things heat up, organizations can expect attacks to shift from data exfiltration to data destruction and for adversarial tooling to grow in sophistication as they expose advanced tools and techniques, such as zero-day exploits, in order to gain a temporary advantage on the cyber battlefield.

Against this backdrop, corporate security teams and CISOs should focus on the following areas:

1. Pivot SOC teams from daily worklist and ticket queue response to an active threat hunting posture. As state-sponsored attackers escalate to more advanced tools and break out cherished exploits, some attacks will become more difficult to pick up with existing signature and payload-based threat detection systems. Consequently, SOC teams will need to spend more time correlating events and logs, and hunting for new attack sequences.
2. Prepare incident responders to investigate suspicious events earlier and to mitigate threats faster. As attackers move from exfiltration to destruction, a timely response becomes even more critical.
3. Review the organization’s back-up strategy for all critical business data and business systems, and verify their recoverability. As the saying goes, a back-up is only as good as its last recovery. This will provide continuity in the event actors using ransomware no longer respond to payment, leaving your data unrecoverable.
4. Update your business response plan and practice disaster recovery to build your recovery muscle memory. Plan for new threat vectors and rapid destruction of critical business systems, both internal and third-party.
5. Double-check the basics and make sure they’re applied everywhere. Since so many successful attack vectors still rely on social engineering and password guessing, use anti-phishing and multi-factor authentication (MFA) as front-line defenses for the cyberwar. Every privileged account throughout the organization and those entrusted to “trusted” supplier access should be using MFA by default.
6. Engage directly with your preferred security providers and operationalize any new TTPs and indicators associated with Middle Eastern attack operators that they can share with you. Make sure that your hunting tools account for the latest threat intelligence and are capable of alerting the right teams should a threat surface.
7. For organizations that have adopted cyber-insurance policies to cover business threats that cannot be countered with technology, double-check which and what “acts of war” are covered.

While implementing the above advice will place your organization on a better “cyberwar footing”, history shows that even well-resourced businesses targeted by Iranian state-sponsored groups fall victim to these attacks. Fortunately, there’s a silver lining in the storm clouds. Teaming up in-house security teams with public cloud providers puts companies in a much better position to respond to and counter such threats because doing so lets them leverage the massively scalable capabilities of the cloud provider’s infrastructure and the depth of security expertise from additional responders. For this reason, organizations should consider which critical business systems could be duplicated or moved for continuity and recovery purposes to the cloud, and in the process augment their existing on-premises threat response.

-- Gunter Ollmann

First Published: SecurityWeek - July 2, 2019

The Purple Team Pentest

It’s not particularly clear whether a marketing intern thought he was being clever or a fatigued pentester thought she was being cynical when the term “Purple Team Pentest” was first thrown around like spaghetti at the fridge door, but it appears we’re now stuck with the term for better or worse.

Just as the definition of penetration testing has broadened to the point that we commonly label a full-scope penetration of a target’s systems with the prospect of lateral compromise and social engineering as a Red Team Pentest – delivered by a “Red Team” entity operating from a sophisticated hacker’s playbook. We now often acknowledge the client’s vigilant security operations and incident response team as the “Blue Team” – charged with detecting and defending against security threats or intrusions on a 24x7 response cycle.

Requests for penetration tests (Black-box, Gray-box, White-box, etc.) are typically initiated and procured by a core information security team within an organization. This core security team tends to operate at a strategic level within the business – advising business leaders and stakeholders of new threats, reviewing security policies and practices, coordinating critical security responses, evaluating new technologies, and generally being the go-to-guys for out-of-ordinary security issues. When it comes to penetration testing, the odds are high that some members are proficient with common hacking techniques and understand the technical impact of threats upon the core business systems.

These are the folks that typically scope and eventually review the reports from a penetration test – they are however NOT the “Blue Team”, but they may help guide and at times provide third-line support to security operations people. No, the nucleus of a Blue Team are the front-line personnel watching over SIEM’s, reviewing logs, initiating and responding to support tickets, and generally swatting down each detected threat as it appears during their shift.

Blue Teams are defensively focused and typically proficient at their operational security tasks. The highly-focused nature of their role does however often mean that they lack what can best be described as a “hackers eye view” of the environment they’re tasked with defending.

Traditional penetration testing approaches are often adversarial. The Red Team must find flaws, compromise systems, and generally highlight the failures in the targets security posture. The Blue Team faces the losing proposition of having to had already secured and remediated all possible flaws prior to the pentest, and then reactively respond to each vulnerability they missed – typically without comprehension of the tools or techniques the Red Team leveraged in their attack. Is it any wonder that Blue Teams hate traditional pentests? Why aren’t the Red Team consultants surprised that the same tools and attack vectors work a year later against the same targets?

A Purple Team Pentest should be thought of as a dynamic amalgamation of Red Team and Blue Team members with the purpose of overcoming communication hurdles, facilitating knowledge transfer, and generally arming the Blue Team with newly practiced skills against a more sophisticated attacker or series of attack scenarios.

How to Orchestrate a Purple Team Pentest Engagement

Very few organizations have their own internal penetration testing team and even those that do regularly utilize external consulting companies to augment that internal team to ensure the appropriate skills are on hand and to tackle more sophisticated pentesting demands.

A Purple Team Pentest almost always utilizes the services of an external pentest team – ideally one that is accomplished and experienced in Red Team pentesting.

Bringing together two highly skilled security teams – one in attack, the other in defense – and having them not only work together, but to also achieve all the stated goals of a Purple Team pentest, requires planning and leadership.

To facilitate a successful Purple Team Pentest, the client organization should consider the following key elements:

• Scope & Objectives - Before reaching out and engaging with a Red Team provider, carefully define the scope and objectives of the Purple Team Pentest. Be specific as to what the organizations primary goals are and what business applications or operational facilities will be within scope. Since a key objective of conducting a Purple Team Pentest is to educate and better arm the internal Blue Team and to maximize the return on a Red Team’s findings, identify and list the gaps that need to be addressed in order to define success.
• Blue Team Selection - Be specific in defining which pieces of the organization and which personnel constitute the “Blue Team”. Go beyond merely informing various security operations staff that they are now part of a Blue Team. It is critical that the members feel they are a key component in the company’s new defensive strategy. Educate them about the roles and responsibilities of what the Blue Team entails. Prior to engaging with a Red Team provider and launching a Purple Team Pentest, socialize and refine the scope and objectives of the proposed Purple Teaming engagement with the team directly.
• Red Team Selection - It is important that the client select a Red Team that consists of experienced penetration testers. The greater the skills and experience of the Red Team members, the more they will be able to contribute to the Purple Team Pentest objectives. Often, in pure Red Team Pentest engagements, the consulting team will contain a mix of experienced and junior consultants – with the junior consultants performing much of the tool-based activities under the supervision of the lead consultant. Since a critical component of a Purple Team Pentest lies in the ability to communicate and educate a Blue Team to the attacker’s methodologies and motivations, junior-level consultants add little value to that dialogue. Clients are actively encouraged to review the resumes of the consultants proposed to constitute the Red Team in advance of testing.
• Playbook Definition - Both sides of the Purple Teaming exercise have unique objectives and methodologies. Creation of a playbook in advance of testing is encouraged and so too is the sharing and agreement between the teams. This playbook loosely defines the rules of the engagement and is largely focused on environment stability (e.g. rules for patch management and rollout during the testing period) and defining exceptions to standard Blue Team responses (e.g. identifying but not blocking the inbound IP addresses associated with the Red Team’s C&C).
• Arbitrator or Referee - Someone must be the technical “Referee” for the Purple Team Pentest. They need to be able to speak both Red Team and Blue Team languages, interpret and bridge the gap between them, manage the security workshops that help define and resolve any critical threat discoveries, and generally arbitrate according to the playbook (often adding to the playbook throughout the engagement). Ideally the arbitrator or referee for the engagement is not directly associated with, or a member of, either the Red or Blue teams.
• Daily Round-table Reviews - Daily round-table discussions and reviews of Red Team findings are the center-piece of a successful Purple Team Pentest. Best conducted at the start of each day (mitigating the prospect of long tired days and possible overflow of working hours – curtailing discussion), the Red Team lays out the successes and failures of the previous days testing, while the Blue Team responds with what they detected and how they responded. The review facilitates the discussion of “What and Why” the Red Team members targeted, explain the “How” they proceeded, and allows the Blue Team to query and understand what evidence they may have collected to detect and thwart such attacks. For example, daily discussions should include discussions covering what traffic did the tool or methodology generate, where could that evidence have been captured, how could that evidence be interpreted, what responses would pose the biggest hurdle to the attacker?
• Pair-down Deep Dives - Allowing members of the teams to “pair down” after the morning review to dive deeper in to the technical details and projected responses to a particular attack vector or exploitation is highly encouraged.
• Evaluate Attack and Defense Success in Real-time - Throughout the engagement the “Arbitrator” should engage with both teams and be constantly aware of what attacks are in play by the Red Team, and what responses are being undertaken by the Blue Team. In some attack scenarios it may be worthwhile allowing the Red Team to persist in an attack even if it has been detected and countered by the Blue Team, or is known to be unsuccessful and unlikely to lead to compromise. However, the overall efficiency can be increased and the cost of a Purple Team Pentest can be reduced by brokering conversations between the teams when attack vectors are stalled, irrelevant, already successful, or known to eventually become successful. For example, the Red Team are able to get a foothold on a compromised host and then proceed to bruteforce attack the credentials of an accessible internal database server. Once the Red Team have successfully started their brute-force attack it may be opportune to validate with the Blue Team that they have already been alerted to the attack in process and are initiating countermeasures. At that point in time, in order to speed up the testing and to progress with another approved attack scenario, a list of known credentials are passed directly to the Red Team and they may progress with a newly created test credential on that (newly) compromised host.
• Triage and Finding Review - Most Red Team pentests will identify a number of security vulnerabilities and exploit paths that were missed by the Blue Team and will require vendor software patches or software development time to remediate. In a pure Red Team Pentest engagement, a “Final Report” would be created listing all findings – with a brief description of recommended and generic best practice fixes. In a Purple Team Pentest, rather than production of a vulnerability findings report, an end-of-pentest workshop should be held between the two teams. During this workshop each phase of the Red Team testing is reviewed – discoveries, detection, remediation, and mitigation – with an open Q&A dialogue between the teams and, at the conclusion of the workshop, a detailed remediation plan is created along with owner assignment.

The Future is Purple

While the methodologies used in Purple Team penetration testing are the same as those of a stand-alone Red Team Pentest, the business objectives and communication methods used are considerably different. Even though the Purple Team Pentest concept is relatively new, it is an increasingly important vehicle for increasing an organizations security stature and reducing overall costs.

The anticipated rewards from conducting a successful Purple Team pentest include increased Blue Team knowledge of threats and adversaries, muscle-memory threat response and mitigation, validation of playbook response to threats in motion, confidence in sophisticated attacker incident response, identification and enumeration of new vulnerabilities or attack vectors, and overall team-building.

As businesses become more aware of Purple Teaming concepts and develop an increased understanding of internal Blue Team capabilities and benefits, it is anticipated that many organizations will update their annual penetration testing requirements to incorporate Purple Team Pentest as a cornerstone of their overall information security and business continuity strategy.

-- Gunter Ollmann