Get Strategic About Cyber Risk Management

With global cybercrime damage costs exceeding $11 trillion last year and moving toward an estimated $20 trillion by 2026, robust cybersecurity risk management has never been more imperative.

The interconnected nature of modern technology means that, by default, even small vulnerabilities can lead to catastrophic losses. And it’s not just about finances. Unmitigated risk raises the specter of eroded customer confidence and tainted brand reputation. In this comprehensive guide, we’ll give enterprise defenders a holistic, methodical, checklist-style approach to cybersecurity risk management. We’ll focus on practical applications, best practices, and ready-to-implement strategies designed to mitigate risks and safeguard digital assets against ever-more numerous—and increasingly capable—threats and adversaries.

What is Cybersecurity Risk Management?

This subspecialty of enterprise risk management describes a systematic approach to identifying, analyzing, evaluating, and addressing cyber threats to an organization’s assets and operations. At its core, it involves a continuous cycle of risk assessment, risk decision-making, and the implementation of risk controls intended to minimize the negative impact of cyber incidents.

A proactive cyber risk mitigation approach helps organizations protect critical digital assets and bolster business continuity, legal compliance, and customer trust. By integrating risk management with the organization’s overall strategic planning, cybersecurity teams can prioritize resources efficiently and align their efforts with the business’s risk appetite and objectives.

Why Has Cyber Risk Management Become So Critical?

Getting control over cyber risk is quickly becoming a core requirement for businesses operating in today’s digital ubiquity. The proliferation of digital information and internet connectivity have paved the way for sophisticated cyber threats that can penetrate many of our most robust defenses. With the digital footprint of businesses expanding exponentially, the potential for data breaches, ransomware attacks, and other forms of cybercrime has escalated dramatically.

These incidents can result in devastating financial losses, legal repercussions, and irreparable damage to an organization’s reputation. Furthermore, as regulatory frameworks around data protection become more stringent, failure to comply can lead to significant penalties. Given these conditions, an aggressive and comprehensive approach to managing cybersecurity risks is crucial for safeguarding an organization’s assets, ensuring operational continuity, and maintaining trust with customers and stakeholders.

Effective Cyber Risk Management: A Framework-Based Approach

Adopting a structured, framework-based approach to cybersecurity risk management lets security teams corral the complexity of digital environments with a methodical, strategic mitigation methodology. For most enterprise applications, there’s no need to reinvent the wheel. There are a myriad of established frameworks that can be modified and customized for effective use in nearly any environment.

Perhaps the best known is the National Institute of Standards and Technology (NIST) Risk Management Framework (RMF), a companion to NIST’s well-tested and widely implemented Cybersecurity Framework (CSF). The NIST RMF offers a structured and systematic approach for integrating security, privacy, and risk management processes into an organization’s system development life cycle.

Such frameworks provide a comprehensive set of guidelines that help identify and assess cyber threats and facilitate the development of effective strategies to mitigate these risks. By standardizing cybersecurity practices, organizations can ensure a consistent and disciplined application of security measures across all departments and operations.

This coherence and uniformity are crucial for effectively addressing vulnerabilities and responding to incidents promptly. Equally important, frameworks incorporate best practices and benchmarks that help guide organizations toward achieving compliance with regulatory requirements, thus minimizing legal risks and enhancing the safeguarding of customer data. In essence, a framework-based approach offers a clear roadmap for managing cyber risk in a way that’s aligned with organizational strategic objectives and industry standards.

What follows is a checklist based on the 7-step RMF process. This is just a starting point. A framework to-do list like this can and should be tweaked to aid in reducing and managing specific cyber risks in your unique enterprise environment.

1. Preparation

In this initial phase, organizations focus on establishing the context and priorities for the Risk Management Framework process. This involves identifying critical assets, defining the boundaries, and codifying a risk management strategy that aligns with the organization’s objectives and resources. This is the foundation upon which a tailored approach to managing cybersecurity risk will ultimately be built throughout the system’s lifecycle.

• Establish the context for risk management and create a risk management strategy.
• Define roles and responsibilities across the organization.
• Develop a taxonomy for categorizing information and information systems.
• Determine the legal, regulatory, and contractual obligations.
• Prepare an inventory of system elements, including software and hardware.

2. Systems Categorization

Expanding on the categorization step (above), this phase involves identifying the types of information processed, stored, and transmitted to determine potential impact as measured against the information security CIA triad (confidentiality, integrity, and availability). Organizations can assign appropriate security categories to their systems by leveraging a categorization standard such as the Federal Information Processing Standard (FIPS) 199, ensuring that the protective measures taken are tailored to the specific needs and risks associated with the information being handled. This step is crucial as it lays the groundwork for selecting suitable security controls in the later stages of the risk management process.

• Identify the types of information processed, stored, and transmitted by the system.
• Assess the potential impact of loss of Confidentiality, Integrity, and Availability (CIA) associated with each type.
• Document findings in a formal security categorization statement.

3. Selecting Appropriate Security Controls

This critical step begins the safeguarding of information systems against potential threats and vulnerabilities in earnest. Based on the categorization of the information system, organizations select a baseline of security and privacy controls (NIST Special Publication 800-53 or some equivalent controls standard is a good starting point here), corresponding to the system’s impact level. This baseline acts as the jumping-off point for the security controls, which can be tailored to address the specific risks identified throughout the risk assessment process. Customization involves adding, removing, or modifying controls to ensure a robust defense tailored to the unique requirements and challenges of the organization.

• Select an appropriate baseline of security controls (NIST SP 800-53 or equivalent).
• Tailor the baseline controls to address specific organizational needs and identified risks.
• Document the selected security controls in the system security plan.
• Develop a strategy for continuously monitoring and maintaining the effectiveness of security controls.

4. Implementing the Selected Controls

Implementing security controls involves the physical and technical application of measures chosen during the previous selection phase. This step requires careful execution to ensure all controls are integrated effectively within the environment, aligning with its architecture and operational practices. Documenting the implementation details is crucial to provide a reference for future assessments and maintenance activities.

• Implement the security controls as documented in Step 3.
• Document the security controls and the responsible entities in place.
• Test thoroughly to ensure compatibility and uninterrupted functionality.
• Prepare for security assessment by documenting the implementation details.

5. Assessing Controls Performance

Assessing security controls involves evaluating effectiveness and adherence to the security requirements outlined in the overall security plan. This phase is critical for identifying any control deficiencies or weaknesses that could leave the information system vulnerable. Independent reviewers or auditors typically conduct assessments to ensure objectivity and a comprehensive analysis.

• Develop and implement a plan to assess the security controls.
• Perform security control assessments as per the plan.
• Prepare a Security Assessment Report (SAR) detailing the effectiveness of the controls.
• Determine if additional controls are needed and append the master security plan accordingly.

6. Authorizing the Risk Management Program

The authorization phase is a vital decision-making interval where one or more senior executives evaluate the security controls’ assessment results and decide whether the remaining risks to the information systems are acceptable to the organization. Upon acceptance, authorization is granted to operate the mitigation program for a specific time period, during which its compliance and security posture are continuously monitored. This authorization is formalized through the issuance of what is known as an Authorization to Operate (ATO) in some organizations, particularly in the public sector.

• Compile the required authorization package, including the master plan, the SAR, and the so-called Plan of Action and Milestones (POA&M).
• Assess the residual risk against the organizational risk tolerance.
• Document the authorization decision in an Authorization Decision Document.

7. Monitoring and Measuring Against Performance Metrics

The monitoring phase ensures that all implemented security controls remain effective and compliant over time. Continuous surveillance, reporting, and analysis can promptly address any identified vulnerabilities or changes in the operational environment. This ongoing process supports the kind of flexible, adaptive security posture necessary for dealing with evolving threats while steadfastly maintaining the integrity and availability of the information system.

• Implement the plan for ongoing monitoring of security controls.
• Report the system’s security state to designated leaders in the organization.
• Perform ongoing risk assessments and response actions, updating documentation as necessary.
• Conduct reviews and updates regularly, in accordance with the organizational timelines, or as significant changes occur.

Conclusion: Formalizing Cyber Risk Mitigation

A solid risk management framework provides a comprehensive guide for enhancing the security and resilience of information systems through a structured process of identifying, implementing, and monitoring security controls.

Sticking to a framework checklist helps ensure a successful, systematic adoption. As noted throughout, engaging stakeholders from across the organization, including IT, security, operations, and compliance, is critical to ensuring a truly comprehensive risk management program. Additionally, periodic training and awareness for team members involved in various phases of the risk management project will contribute to the resilience and security of the organization’s digital assets.

Organizations can effectively safeguard their digital assets and mitigate unacceptable risks by following the outlined steps, tailoring the program to fit specific organizational needs, involving stakeholders, conducting regular training, and adapting to the evolving cybersecurity landscape. Ultimately, this kind of formal, structured cyber risk management fosters a culture of continuous improvement and vigilance in an enterprise, contributing to the overall security posture and the success of the organization.

-- Gunter Ollmann

First Published: IOActive Blog - August 20, 2024

The Security Imperative in Artificial Intelligence

Artificial Intelligence (AI) is transforming industries and everyday life, driving innovations once relegated to the realm of science fiction into modern reality. As AI technologies grow more integral to complex systems like autonomous vehicles, healthcare diagnostics, and automated financial trading platforms, the imperative for robust security measures increases exponentially.

Securing AI is not only about safeguarding data but also about ensuring the core systems — in particular, the trained models that really put the “intelligence” in AI — function as intended without malicious interference. Historical lessons from earlier technologies offer some guidance and can be used to inform today’s strategies for securing AI systems. Here, we’ll explore the evolution, current state, and future direction of AI security, with a focus on why it’s essential to learn from the past, secure the present, and plan for a resilient future.

AI: The Newest Crown Jewel

Security in the context of AI is paramount precisely because AI systems increasingly handle sensitive data, make important, autonomous decisions, and operate with limited supervision in critical environments where safety and confidentiality are key. As AI technologies burrow further into sectors like healthcare, finance, and national security, the potential for misuse or harmful consequences due to security shortcomings rises to concerning levels. Several factors drive the criticality of AI security:

• Data Sensitivity: AI systems process and learn from large volumes of data, including personally identifiable information, proprietary business information, and other sensitive data types. Ensuring the security of enterprise training data as it passes to and through AI models is crucial to maintaining privacy, regulatory compliance, and the integrity of intellectual property.
• System Integrity: The integrity of AI systems themselves must be well defended in order to prevent malicious alterations or tampering that could lead to bogus outputs and incorrect decisions. In autonomous vehicles or medical diagnosis systems, for example, instructions issued by compromised AI platforms could have life-threatening consequences.
• Operational Reliability: AI is increasingly finding its way into critical infrastructure and essential services. Therefore, ensuring these systems are secure from attacks is vital for maintaining their reliability and functionality in critical operations.
• Matters of Trust: For AI to be widely adopted, users and stakeholders must trust that the systems are secure and will function as intended without causing unintended harm. Security breaches or failures can undermine public confidence and hinder the broader adoption of emerging AI technologies over the long haul.
• Adversarial Activity: AI systems are uniquely susceptible to certain attacks, whereby slight manipulations in inputs — sometimes called prompt hacking — can deceive an AI system into making incorrect decisions or spewing malicious output. Understanding the capabilities of malicious actors and building robust defenses against such prompt-based attacks is crucial for the secure deployment of AI technologies.

In short, security in AI isn’t just about protecting data. It’s also about ensuring safe, reliable, and ethical use of AI technologies across all applications. These inexorably nested requirements continue to drive research and ongoing development of advanced security measures tailored to the unique challenges posed by AI.

Looking Back: Historical Security Pitfalls

We don’t have to turn the clock back very far to witness new, vigorously hyped technology solutions wreaking havoc on the global cybersecurity risk register. Consider the peer-to-peer recordkeeping database mechanism known as blockchain.  When blockchain exploded into the zeitgeist circa 2008 — alongside the equally disruptive concept of cryptocurrency — its introduction brought great excitement thanks to its potential for both decentralization of data management and the promise of enhanced data security. In short order, however, events such as the DAO hack —an exploitation of smart contract vulnerabilities that led to substantial, if temporary, financial losses — demonstrated the risk of adopting new technologies without diligent security vetting.

As a teaching moment, the DAO incident highlights several issues: the complex interplay of software immutability and coding mistakes; and the disastrous consequences of security oversights in decentralized systems. The case study teaches us that with every innovative leap, a thorough understanding of the new security landscape is crucial, especially as we integrate similar technologies into AI-enabled systems.

Historical analysis of other emerging technology failures over the years reveals other common themes, such as overreliance on untested technologies, misjudgment of the security landscape, and underestimation of cyber threats. These pitfalls are exacerbated by hype-cycle-powered rapid adoption that often outstrips current security capacity and capabilities. For AI, these themes underscore the need for a security-first approach in development phases, continuous vulnerability assessments, and the integration of robust security frameworks from the outset.

Current State of AI Security

With AI solutions now pervasive, each use case introduces unique security challenges. Be it predictive analytics in finance, real-time decision-making systems in manufacturing systems, or something else entirely,  each application requires a tailored security approach that takes into account the specific data types and operational environments involved. It’s a complex landscape where rapid technological advancements run headlong into evolving security concerns. Key features of this challenging  infosec environment include:

• Advanced Threats: AI systems face a range of sophisticated threats, including data poisoning, which can skew an AI’s learning and reinforcement processes, leading to flawed outputs; model theft, in which proprietary intellectual property is exposed; and other adversarial actions that can manipulate AI perceptions and decisions in unexpected and harmful ways. These threats are unique to AI and demand specialized security responses that go beyond traditional cybersecurity controls.
• Regulatory and Compliance Issues: With statutes such as GDPR in Europe, CCPA in the U.S., and similar data security and privacy mandates worldwide, technology purveyors and end users alike are under increased pressure to prioritize safe data handling and processing. On top of existing privacy rules, the Biden administration in the U.S. issued a comprehensive executive order last October establishing new standards for AI safety and security. In Europe, meanwhile, the EU’s newly adopted Artificial Intelligence Act provides granular guidelines for dealing with AI-related risk. This spate of new rules can often clash with AI-enabled applications that demand more and more access to data without much regard for its origin or sensitivity.
• Integration Challenges: As AI becomes more integrated into critical systems across a wide swath of vertical industries, ensuring security coherence across different platforms and blended technologies remains a significant challenge. Rapid adoption and integration expose modern AI systems to traditional threats and legacy network vulnerabilities, compounding the risk landscape.
• Explainability: As adoption grows, the matter of AI explainability  — or the ability to understand and interpret the decisions made by AI systems — becomes increasingly important. This concept is crucial in building trust, particularly in sensitive fields like healthcare where decisions can have profound impacts on human lives.Consider an AI system used to diagnose disease from medical imaging. If such a system identifies potential tumors in a scan, clinicians and patients must be able to understand the basis of these conclusions to trust in their reliability and accuracy. Without clear explanations, hesitation to accept the AI’s recommendations ensues, leading to delays in treatment or disregard of useful AI-driven insights. Explainability not only enhances trust, it also ensures AI tools can be effectively integrated into clinical workflows, providing clear guidance that healthcare professionals can evaluate alongside their own expertise.

Addressing such risks requires a deep understanding of AI operations and the development of specialized security techniques such as differential privacy, federated learning, and robust adversarial training methods. The good news here: In response to AI’s risk profile, the field of AI security research and development is on a steady growth trajectory. Over the past 18 months the industry has witnessed  increased investment aimed at developing new methods to secure AI systems, such as encryption of AI models, robustness testing, and intrusion detection tailored to AI-specific operations.

At the same time, there’s also rising awareness of AI security needs beyond the boundaries of cybersecurity organizations and infosec teams. That’s led to better education and training for application developers and users, for example, on the potential risks and best practices for securing A-powered systems.

Overall,  enterprises at large have made substantial progress in identifying and addressing AI-specific risk, but significant challenges remain, requiring ongoing vigilance, innovation, and adaptation in AI defensive strategies.

Data Classification and AI Security

One area getting a fair bit of attention in the context of safeguarding AI-capable environments is effective data classification. The ability to earmark data (public, proprietary, confidential, etc.) is essential for good AI security practice. Data classification ensures that sensitive information is handled appropriately within AI systems. Proper classification aids in compliance with regulations and prevents sensitive data from being used — intentionally or unintentionally — in training datasets that can be targets for attack and compromise.

The inadvertent inclusion of personally identifiable information (PII) in model training data, for example, is a hallmark of poor data management in an AI environment. A breach in such systems not only compromises privacy but exposes organizations to profound legal and reputational damage as well. Organizations in the business of adopting AI to further their business strategies must be ever aware of the need for stringent data management protocols and advanced data anonymization techniques before data enters the AI processing pipeline.

The Future of AI Security: Navigating New Horizons

As AI continues to evolve and tunnel its way further into every facet of human existence, securing these systems from potential threats, both current and future, becomes increasingly critical. Peering into AI’s future, it’s clear that any promising new developments in AI capabilities must be accompanied by robust strategies to safeguard systems and data against the sophisticated threats of tomorrow.

The future of AI security will depend heavily on our ability to anticipate potential security issues and tackle them proactively before they escalate. Here are some ways security practitioners can prevent future AI-related security shortcomings:

• Continuous Learning and Adaptation: AI systems can be designed to learn from past attacks and adapt to prevent similar vulnerabilities in the future. This involves using machine learning algorithms that evolve continuously, enhancing their detection capabilities over time.
• Enhanced Data Privacy Techniques: As data is the lifeblood of AI, employing advanced and emerging data privacy technologies such as differential privacy and homomorphic encryption will ensure that data can be used for training without exposing sensitive information.
• Robust Security Protocols: Establishing rigorous security standards and protocols from the initial phases of AI development will be crucial. This includes implementing secure coding practices, regular security audits, and vulnerability assessments throughout the AI lifecycle.
• Cross-Domain Collaboration: Sharing knowledge and strategies across industries and domains can lead to a more robust understanding of AI threats and mitigation strategies, fostering a community approach to AI security.

Looking Further Ahead

Beyond the immediate horizon, the field of AI security is set to witness several meaningful advancements:

• Autonomous Security: AI systems capable of self-monitoring and self-defending against potential threats will soon become a reality. These systems will autonomously detect, analyze, and respond to threats in real time, greatly reducing the window for attacks.
• Predictive Security Models: Leveraging big data and predictive analytics, AI can forecast potential security threats before they manifest. This proactive approach will allow organizations to implement defensive measures in advance.
• AI in Cybersecurity Operations: AI will increasingly become both weapon and shield. AI is already being used to enhance cybersecurity operations, providing the ability to sift through massive amounts of data for threat detection and response at a speed and accuracy unmatchable by humans. The technology and its underlying methodologies will only get better with time. This ability for AI to remove the so-called “human speed bump” in incident detection and response will take on greater importance as the adversaries themselves increasingly leverage AI to generate malicious attacks that are at once faster, deeper, and potentially more damaging than ever before.
• Decentralized AI Security Frameworks: With the rise of blockchain technology, decentralized approaches to AI security will likely develop. These frameworks can provide transparent and tamper-proof systems for managing AI operations securely.
• Ethical AI Development: As part of securing AI, strong initiatives are gaining momentum to ensure that AI systems are developed with ethical considerations in mind will prevent biases and ensure fairness, thus enhancing security by aligning AI operations with human values.

As with any rapidly evolving technology, the journey toward a secure AI-driven future is complex and fraught with challenges. But with concerted effort and prudent innovation, it’s entirely within our grasp to anticipate and mitigate these risks effectively. As we advance, the integration of sophisticated AI security controls will not only protect against potential threats, it will foster trust and promote broader adoption of this transformative technology. The future of AI security is not just about defense but about creating a resilient, reliable foundation for the growth of AI across all sectors.

Charting a Path Forward in AI Security

Few technologies in the past generation have held the promise for world-altering innovation in the way AI has. Few would quibble with AI’s immense potential to disrupt and benefit human pursuits from healthcare to finance, from manufacturing to national security and beyond. Yes, Artificial Intelligence is revolutionary. But it’s not without cost. AI comes with its own inherent collection of vulnerabilities that require vigilant, innovative defenses tailored to their unique operational contexts.

As we’ve discussed, embracing sophisticated, proactive, ethical, collaborative AI security and privacy measures is the only way to ensure we’re not only safeguarding against potential threats but also fostering trust to promote the broader adoption of what most believe is a brilliantly transformative technology.

The journey towards a secure AI-driven future is indeed complex and fraught with obstacles. However, with concerted effort, continuous innovation, and a commitment to ethical practices, successfully navigating these impediments is well within our grasp. As AI continues to evolve, so too must our strategies for defending it. 

-- Gunter Ollmann

First Published: IOActive Blog - May 30, 2024

GenAI Influencing the Secure Code Review Profession

 It’s tough to be a secure code reviewer. There are already over 700 programming languages according to Wikipedia, and seemingly more languages materializing every year. Expectations are high that rapid developments in Generative Artificial Intelligence (GenAI) will bring a new suite of languages and security issues that’ll have an oversized impact on software development. Consequently, secure software development lifecycle (SDL) processes and security code review are having to evolve rapidly.

I’m both excited and nervous about GenAI advancements in the world of software development and secure application design. It’s exciting to see how prompt engineering of Large Language Models (LLM) and adoption of AI augmentation in the form of copilots and chatbots are increasing the pace of ideation into new products. I’m nervous about the hallucinations and code quality being generated in response though.

English as a Programming Language

2023 was the breakthrough year for AI, with LLM and GenAI permeating every industry, technology, and product. Today, the most in-demand languages currently are Python, C, and C++ but, controversially, the future star programming language may in fact be English; something that’ll take some time to adjust to.

For over a decade we’ve been told that the supply of experienced cybersecurity professionals has trailed the market’s requirements, with a deficit growing year-on-year, and a casual scan across office desks and cubicles will highlight a more significant gender gap across the cybersecurity (and software development) industry. I think AGI and emergence of English as a critical programming language are fundamental to correcting both industry problems.

GenAI, particularly those based upon LLM advancements, are increasingly sophisticated language machines – and women may have an advantage over men in maximizing utility and productivity from them.

Multiple studies over the last 30 years have constantly highlighted that women are better communicators than men. “Better” is obviously an explosive and controversial term even amongst the academics who published the studies, but in general women have more expansive vocabularies and stronger interpretative communication skills. Modern neuroscience and studies in children and adolescents identify girls as more garrulous than boys, with greater complexity and sophistication of language, and tend to develop more in the realm of listening with greater focus and concentration as they age. This historically translates into women being better coders than men (once you remove the bias in the system).

As I look to GenAI and the expanding world of prompt engineering, I anticipate that women will have an advantage over their male developer counterparts. Strong and well-developed communication skills (and the reasoning and understanding that underlays those polished skills) are prerequisites for maximizing efficiency of GenAI-returned results and tuning responses – both now and for the immediate future.

Starter-job Experience

But what about experience? The “experience gap” is often called out as a chasm for newly minted degree-holding graduates and landing a starter-job in cybersecurity.

It’s rare to find an entry-level job in our industry that doesn’t require multiple years of hands-on security experience nowadays as many of those traditional starter roles – network scanning, alert triage, playbook maintenance, patch management – have been automated away, with many more projected to disappear as AI adoption increases.

Most successful new entrants into the cybersecurity profession come from adjacent technical industries making a career jump rather than direct from a college or university. Armed with transferable skills and technical experience, they’re capable of crossing the chasm left in the wake of cyber automation. However, the security knowledge gap between a cybersecurity veteran and a recent transfer remains large and a growing concern for the industry.

I’m excited to think AI augmentation and copilot technologies will have one of the largest impacts on our industry – removing much of the security knowledge gap and reducing the overall impact of the experience gap – like what is happening in other industries, such as the medical field. For example, AI use in patient triage, predictive analytics, and virtual assistants are augmenting generalist regional nurses (two-year qualification) and Bachelor of Science in Nursing (four-year qualification) graduates, and allowing them to perform many of the roles and responsibilities traditionally associated with a completed medical doctor degree (10 to 12 years).

Secure Code Reviews

It’s tough to be a secure code reviewer. There aren’t enough of them. The job requires tremendous amounts of experience and advanced security knowledge, and it’s tiring and hard work.

GenAI is going to have a huge impact on their job.

On the positive side, English as a programming language and AI augmentation and copilots is going to help increase both the breadth and depth of the cybersecurity talent pool available to perform this critical job. The tools available to code reviewers to automatically review and assess code security are advancing quickly and, while still in their first generation of AI adoption, are anticipated to mature rapidly and identify vulnerabilities and logic flaws with higher fidelity and trust. I’m sure there’ll be a gap between the best that a tool can achieve versus the best-of-the-best human expert though – especially when that expert is augmented and using similar tools themselves.

Meanwhile, GenAI is spearheading prompt engineering of new software applications. A new generation of product developers may have little to no influence over the code that powers the application. Indeed, I’ve previously argued that the role of product manager will change greatly in the coming years as their skills in product design and requirement setting pivot from being directed to engineering teams and into GenAI prompts instead.

What does an GenAI-generated application look like under the covers? Time will tell. We anticipate it’ll increasingly become more secure – using best security practices and recycling pretested code behind the scenes – and that it’ll constantly learn, optimize, and apply best and better security practices – but we’ll still need those human secure code reviewers for some time to come, specially when it comes to high-impact applications and certification.

A concern though as GenAI does more of the application development software developers will have less direct influence over the underlying code is that powering the application and business logic. It would be a nightmare if the AGI produced entirely different code throughout the application each time it received new design criteria and re-optimized – making vulnerability triage, reporting, reconciliation, and tracking near impossible, and code reviews and code certifications (human or tool led) largely nonsensical.

Watch this space!

I think we’re still some years away from having to worry about continuously reimagined code generated by GenAI without human software developers tweaking and refining the underlaying application code, but it is tremendously exciting to see the rapid advances in prompt engineering and how LLM’s are being incorporated into both old and new products.

Our industry has consistently struggled to attract and retain women. GenAI has the potential to not only level the field and make it easier to join the game, but to also leverage previously poorly-tapped communication skills for the betterment of both application development and security. There’s a lot of work ahead. There’s a lot of research to be done. There’s a lot of opportunities to make code more secure!

-- Gunter Ollmann

First Published: IOActive Blog - March 1, 2024

A SAFE Journey to Selling Devices to Cloud and Datacenter Providers

Observations from the OCP Global Summit | San Jose, CA | October, 18, 2023

If you missed it, there was a significant launch of the Open Compute Project (OCP) Foundation’s new community-led security program for improving device security underpins a fundamental change in the way device vendors and manufacturers engage and sell their products to the worlds leading cloud and datacenter providers.

Beyond standing up a framework for driving continuous security conformance assurance, the Security Appraisal Framework and Enablement (S.A.F.E.) program redefines(and optimizes) the relationship between the device vendor, the device adopter, and the security review provider.

Back when I served as chief security officer for Microsoft’s Cloud and AI Security group, I encountered a scaling problem with the way Azure worked with device vendors (think in terms of CPU’s, GPU’s, SSD’s, network cards, etc.) in validating the integrity and security of their devices. The short story is that before any device or it’s firmware could be deployed within a datacenter a security assessment had to be undertaken – and only devices (and their firmware) that came back with “clean” security reports could be purchased and ultimately deployed. Through a combination of Microsoft’s in-house and trusted third-party penetration testing and code review teams, each key device would be tested and an iterative improvement cycle would continue until the security assurance report came back clean.

There were two obvious problems with that mode of security assurance and interaction. As a purchaser and adopter of devices, how do you scale an in-house security practices to assess and assure the thousands of devices from hundreds of vendors and their ever-changing firmware? And, as a device vendor, how do you meet and coordinate the overlapping yet distinct security requirements of every purchaser and adopter of your technology?

Which brings me back to S.A.F.E. While it has been a couple years since I moved on from Microsoft, it’s fantastic to see that those first meetings with the OCP team and like-minded cloud service providers to tackle those problems that culminated this week in the official launch of the solution – and doubly proud of IOActive’s key involvements in both the creation of the S.A.F.E. device security checklist and Security Review Provider (SRP) criteria, and for being one of the founding SRP providers.

For device vendors, I think S.A.F.E. is great news.

A few points to justify that statement:

1. Just one set of device security requirements (complete with device checklists) that address all OCP member purchasers and adopters. Better yet, community-driven and publicly available security requirements (with input from purchasers, vendors, and SRP providers alike) that will evolve and adapt to the changing world
2. Accredited and trusted OCP SRP providers that device vendors can engage with at anytime in their device and firmware’s development lifecycle – cost-effectively accelerating and simplifying a vendors journey to “clean” S.A.F.E.-Approved product status; ultimately ensuring a vendor’s devices can be purchased and deployed by cloud and datacenter providers quicker than ever before
3. Devices with advanced and proprietary technologies can be security tested and assured to meet adopter security requirements without the vendor having to share proprietary source code with the adopter.

One last perspective to share – oriented specifically to device vendors: there’s fantastic innovation going on across the cloud and datacenter device ecosystem. As I navigate the OCP conference Expo hall, passing by liquid cooling-systems, quantum computing data connectors, petabyte storage arrays, and even specialized forklifts for moving data racks- I’m reminded that many vendors are only just now starting their security journey, and the prospect of having to meet the detailed requirements of S.A.F.E. may feel daunting.

In all my time at Microsoft (and IBM for that matter), I never encountered a device vendor that “nailed security” the first time, nor the third time. Reaching S.A.F.E. approval will be a journey that requires a trusted security partner. When you’re ready for that journey, you’ll not find a stronger and more accomplished or experienced partner than the team at IOActive.

-- Gunter Ollmann

First Published: IOActive Blog - October 19 2023

Why Cybersecurity is Critical in MLOps

Larger and more sophisticated businesses will lean into building out their in-house data science teams and capabilities

If your business relies on machine learning (ML) to drive strategic decision-making, you’re in good company. A recent report by ClearML shows the technology clearly entering the mainstream, with 60% of organizations’ ML leaders planning to increase ML investments by more than a quarter in 2023. The same study revealed that 99% of respondents either already have dedicated budgets for ML operations (MLOps) or plan to implement them this year.

But, as MLOps mature, they also carry more risk. According to a recent study by NCC Group, organizations are deploying ML models in more applications without considering security requirements. In a separate survey by Deloitte, nearly two-thirds of AI and ML users describe cybersecurity risks as a significant or extreme threat, but only 39% feel prepared to combat those risks.

MLOps model creation pipelines are vulnerable and easily attacked in three separate ways: by malicious insiders, through software supply chain manipulation, and via compromised systems. If the SolarWinds supply chain attack taught the industry anything, it’s that continuous build processes are both a target for sophisticated adversaries and a blind spot for in-house security operations teams.

In 2023, continuous build processes will continue to be a target for threat actors. As these attacks start to impact enterprises’ bottom lines, they will have to start paying more attention to the cybersecurity side of MLOps.

Here are some ways to make MLOps projects safer and more secure.

Secure the Whole Pipeline

Part of the challenge of securing MLOps is the sheer length and depth of typical machine learning pipelines. They include a half dozen or more phases – data collection and preparation, along with the creation, evaluation, optimization, deployment and usage of an ML model. Vulnerabilities can crop up at any point in the process.

Early on, in data collection, threat actors can taint the data, manipulate the annotation or conduct adversarial attacks on the metadata stores. In later phases, open-source models and frameworks can include hidden vulnerabilities. Potential bias and system performance need to be addressed. And as models are deployed and used, new data is often introduced, expanding the attack surface and opening an organization up to all kinds of threats – including evasion attacks, model theft, code injections and privacy attacks.

At the tail end of the process, there’s a lot of intellectual property inside an ML model. Decades of transactional data and learnings from financial models that are built and trained into models may only be 10s of kilobytes in size. It’s easier to steal that model than to steal the actual source data.

These models tend to be exposed. Attackers have become skillful at querying models and reproducing them somewhere else. This requires a new way of thinking about the value of the model. Tooling and alerting not only around the theft of data but around the manipulation of the models is important to an overall MLOps security strategy.

Invest in Tooling to Scale Across SOCs

It’s no secret that security is no longer siloed in a single department. It cuts across all functions, and organizations are creating Security Operations Centers (SOCs) to improve the visibility, manageability and auditing of their overall security posture. To extend the SOC’s capability to MLOps, organizations need to incorporate tooling that scales to much larger uses than ever before.

Meeting MLOps’ data needs forces SOCs to adapt in two ways. Existing SOC operations teams are now accountable, forcing them to build the additional tooling and reporting to support MLOps teams from a security perspective. Plus, MLOps teams that are specialized in data science curation are able to leverage larger toolsets – including logging analytics platforms that provide higher levels of threat detection.

Double Down on Security Best Practices

Some of the best defense tactics for MLOps are practices organizations deploy regularly across the rest of their operations. A zero trust security policy requires the authentication and authorization of anybody trying to access applications or data used in the development of ML models. It also tracks their activity. Applying the principle of least privilege (PLoP) limits users’ access to the exact data sets and models they are authorized to touch. This reduces the attack surface by prohibiting hackers who have gained access to one data trove from moving freely throughout the system.

Use Analytics to Observe and Log ML Tasks

An important step in protecting an ML system is to understand the system’s behavior in healthy and unhealthy states. To do this, organizations need to set up alerts that trigger action before an incident occurs. This is called “observability.” A vulnerability introduced early in the training data will affect the model’s performance down the line. Tracking performance data and logging metrics of ML tasks gives organizations insights into any and all security issues that could affect the ML model.

Future Monitoring of Model Development Lifecycle

The continuous lifecycle of MLOps necessitates the continuous monitoring of a deployed model’s response to adversarial manipulation and corruption. In the future, expect to see larger and more sophisticated businesses lean into building out their in-house data science teams and capabilities, detecting threats with security analytics, and pruning and filtering data from unknowns to improve the advancements in the next generation of ML and AI.

-- Gunter Ollmann

First Published: Security InfoWatch - June 20, 2023

What to Consider When Building an Autonomous SOC

Today’s threat landscape demands more from IT and security professionals than ever before. Schools are being forced to shut down due to ransomware attacks, major brands are falling victim to reputation-harming data breaches, and an explosion of connected devices has broadened the attack surface. At the same time, cyber-criminals are getting smarter and savvier, developing new ways to evade detection software and make money.

As cyber-criminals are getting more creative, the cybersecurity industry is improving and developing innovative solutions to protect businesses. Earlier this year, the FBI revealed it had turned the tables on the notorious Hive ransomware gang by secretly hacking the group’s systems, saving $130 million in ransomware demands for more than 300 victims. Despite our best efforts, there are still elements holding us back as an industry and continuing to make organizations vulnerable to cyber-attacks. Prevention, monitoring, and mitigation all happen in the Security Operations Center (SOC), and, right now, SOCs are facing the perfect storm for cyber-crime: lack of visibility into complex operating environments, inability to analyze cloud-scale volumes of data, and an industry-wide shortage of cybersecurity talent. As a result, security professionals are experiencing widespread burnout and unrealistic workloads, which lowers their productivity and creates higher security risks.

Autonomous SOC: Building Yours Right

A lot of the burnout security practitioners face is caused by alert fatigue. When alerts about potential cyber-attacks come in at a rate faster than SOC analysts can handle, analysts work longer hours and still miss important threats. To cut through the noise and focus on the attacks that matter most, SOCs need to take a cue from cyber-criminals and adapt to the current threat landscape: They need to start their journey to the autonomous SOC.

Understanding the Autonomous SOC

An autonomous SOC (ASOC) is composed of an artificial intelligence (AI) and/or machine learning (ML) system that receives all of the data points coming in and assists with cybersecurity monitoring and mitigation. An ASOC is ideal for threat investigation since it can automatically detect suspicious activity, learn and correlate everything about the attack quickly, and provide analysts with the context they need to detect, isolate, and neutralize the attack easily and efficiently. The ASOC also filters out false negatives, allowing analysts to direct their focus on real threats and take immediate action.

The ASOC helps alleviate many of the issues organizations face with their security posture — limited resources, overwhelmed analysts, and repetitive, monotonous tasks. AI and ML’s ability to identify patterns and outliers boosts analysts with an actionable plan of prevention and mediation. An ASOC running in the background provides a much-needed extra layer of coverage to protect organizations, especially those dealing with understaffed SOCs due to recent layoffs throughout the tech industry.

There are a lot of questions around autonomy and the SOC. Namely, will ASOCs replace human analysts? The short answer is no. Human and machine collaboration is necessary for success, especially regarding cybersecurity. ASOCs constantly evolve as they ingest data and assess new threats, which is why they will always need human analysts to create guardrails and provide feedback. ASOCs are designed to make analysts’ jobs easier, not to steal them.

What Organizations Should Consider Before Investing

The ASOC is not a passing trend. It’s where our industry is headed. IDC predicts that by 2026, 30 percent of large enterprise organizations will migrate to ASOCs for faster remediation, incident management, and response. Still, many executives misguidedly view the SOC as a department that exclusively costs the company money and does little — if anything — to drive revenue. As such, the shift to an ASOC may seem daunting or unrealistic to some organizations. Proponents of plans to build an ASOC might face pushback from others in the organization and need to justify the investment costs. The bottom line of an ASOC is to get more value out of the tools and workflows at the SOC’s disposal. In the short-term, this means SOC analysts who are less burnt out, more engaged, and stay at the company longer. In the long-term, investments in cybersecurity save the company money in terms of reputational damage and customer loss if an attack or breach occurs.

Another aspect to consider is timing. Ask yourself, “Is my organization ready for this transition?” Assess the maturity of the SOC and bring SOC analysts and leaders into the conversation. It’s also important to note that moving to an ASOC doesn’t have to be all or nothing; it’s a journey.

Keeping these elements in mind will help you to seamlessly transition to an automated SOC, the future of cybersecurity.

-- Gunter Ollmann

First Published: Solutions Review - March 30, 2023

It’s Time for Security Analytics to Embrace the Age of Science Over Art

Security analytics has traditionally been approached with a “hunt and peck” mentality, which has made the process of uncovering and responding to cyberthreats more art than science. A human analyst has an idea of what they are looking for when they begin to hunt across the available data, performing that task based on their own experience. They’ve been taught to celebrate when they find something, and that the trickier and more obscure the discovery, the greater the celebration of their skills.

This situation is, I believe, an “art” because the results will always differ between analysts — the day of the week, what they had for breakfast, or how their weekend went — and there are too many outside factors that can affect the individual doing the hunting. The situation has only been perpetuated by an industry that has for too long touted the value of this “art.”

We’re no longer working with a simple canvas

We’ve all heard it before and will continue to hear it — data volumes and the enterprise landscape have been growing exponentially and that’s not going to stop. This was put into hyperdrive with the rapid adoption of cloud computing, which challenges organizations to collect and analyze complete data from multiple sources, including new cloud data as well as from legacy, on-premises infrastructures. This has resulted in limited visibility that ultimately compromises overall security.

What we’re not hearing enough is that applying to this challenge the long-held belief in the “art” of hunt-and-peck doesn’t scale and isn’t a reliable or repeatable process that can come close to meeting the needs of modern enterprise environments.

Managing haystacks of needles

We all know the saying “finding a needle in a haystack.” But in today’s threat landscape, given the data volumes with which analysts are burdened, it’s more like finding the sharpest needle in a haystack of needles. Following the decades-old mantra of “assume breach,” we need to turn our focus to the threats that matter most — the sharpest needles. This requires operationalizing the hunt, triage, investigation and response by removing humans from being “artistic” speed bumps and instead empowering them with the science of protection embedded in security analytics.

Adopting the science of security analytics that leverages automation built on machine learning and AI enables repeatable, reliable streaming investigations of threats across all the data, at all times. Applying this method will reveal orders of magnitude more threats and incidents — across a broad spectrum of risk — occurring continuously within the enterprise. We’ve reached the tipping point where threat volumes have far exceeded what any number of human analysts could reasonably hunt/triage, let alone respond to. This means enterprise security teams must increasingly apply AI and ML to the management of the threats they find (i.e., managing those stacks of needles) as well as the mitigations and responses.

Reprieve begins with automation

Building processes that are autonomous is the critical element to embracing a scientific approach to protection. While past security solutions focused on automation, they were largely unsuccessful due to inflexibility and reliance upon humans to choose the right automation steps in advance of applying them for every exception. This is not the role people should be playing when it comes to successfully implementing autonomous solutions, and it doesn’t do anything to lighten their load. Instead, autonomous solutions should deploy system “smartness” to fill in the blanks and know to ask for human guidance when it’s actually needed.

If we continue with the mantra of “assume breach,” and operationalize security as described above, we also must completely rethink the human-focused SOC solution of filtering alerts. With people having been swamped to the point of (and beyond) alert fatigue, the solution has been to drastically manage the funnel of events and alerts, thus reducing the aperture of enterprise threat visibility and response — none of which sounds like a solution to me.

It begs the question: Why bother collecting alerts and events in the first place if you’re only going to do something with 1% of the top 1% most critical alerts? My response: Filtering is the worst way to manage security.

Instead, let’s do this:

With modern AI and autonomous hunting and triaging solutions, the system can look at every event and alert as it streams by and correlate, question and enrich them all in real time — all the time. The more data collected the more accurate and useful the autonomous system becomes, improving its ability to identify the collective stories and present them to the business and the analysts. To take it a step further, the autonomous system can then, in most cases, perform autonomous responses to the threats being found.

Human and machine harmony

Anytime automation in security is discussed it brings up the fear of automating away the analyst. But with a science-first approach, they aren’t going anywhere. The human analyst role is transforming, which will be a huge benefit to the people who work in SOCs. By adopting a scientific method for security analytics, the analyst will influence and guide the autonomous system to ensure it delivers business impact and value:

• For exceptions when the AI doesn’t have enough information or confidence to provide an autonomous response, it watches and learns how the human analyst does or did it, thus building and establishing a scientific methodology.
• At the cloud-SaaS level, those learnings may come from hundreds of enterprise SOC teams and thousands of expert security analysts, from which the AI systems can take collective intelligence and apply those learnings and methodology refinements back into the hands of the individual analyst.

The final result? The loop gets closed. The analyst is augmented.

The autonomous system deals with the daily grind, identifies the gaps that require human expertise, learns by watching how humans fill in the methodology gaps, and reapplies those learnings collectively. For instance, assume that a security team is capable of performing 100 manual investigations per day. An autonomous system could ask millions of forensic questions in a day. Time to resolution is shortened by augmenting the work the analyst does. The autonomous system performs repetitive, data-intensive work, it can quickly go back in time and ask an infinite number of questions, and the efficiency benefits just go on and on.

Leading with science will equip security analysts with actionable data across use cases ranging from threat detection, threat investigation, and threat hunting to ransomware investigation and incident response. It helps security teams work smarter and respond faster while boosting productivity and strengthening security.

-- Gunter Ollmann

First Published: Medium - September 8, 2022

Why the SOC Needs to Learn from the Aviation Industry

The cybersecurity industry has spent a lot of time talking about improving the analyst experience while not making significant improvements, as much of the efforts have been too focused on finding a silver bullet solution. Combine that with a global pandemic and now things are just getting worse. A recent study published by Devo, the 2021 SOC Performance Report, found that on a 10-point scale, where 10 indicates SOC staff have a “very painful” experience performing their jobs, 72% of respondents rated the pain of SOC analysts at a 7 or above.

Instead of thinking about the aforementioned silver bullet to alleviating SOC pain, I wanted to focus on one of the top sources, alert fatigue, and how the cybersecurity industry might be able to take a page out of another field to find a solution.

In the SOC Performance Report, a whopping 61% said a cause of SOC pain was that there are too many alerts to chase. I think it’s safe to draw the connection that “alert fatigue” will expand to “posture fatigue” and “policy fatigue,” as it adversely affects both recruitment and all too critical retention of experienced SOC professionals.

Alert fatigue may exit the aircraft

So, if we can’t figure out within the security industry, let’s learn from others. There are many non-cyber industries and professions that suffer similarly with alert fatigue, and perhaps the cybersecurity industry can reapply some of those learnings. Across these compatriots of alert fatigue, if we ask the question “how do alarms, warnings, and alerts differ?” I think we’ll find much similarity and overlap in answers — in both the theory and practice of how human operators are supposed to respond and how they do so in reality.

For the purpose of this article, I want to take a look at the aviation industry as our example to the SOC. They have navigated many of the problems SOC operators face today and have made the most progress in governing and managing the ergonomics of sensory overload and automation. Picture this: the inside of an airplane cockpit with all its knobs, buttons, lights, and alerts isn’t too dissimilar to the combined dashboards SOC analysts have to navigate when triaging, investigating, and responding to threats.

In 1988, The Washington Post reported on a “glass cockpit” syndrome in the aviation industry, that reads eerily similar to what many say or think about the SOC today. Researchers from the American Psychological Association noted that pilots would “fall victim to information overload and ignore the many bits of data pouring from myriad technical systems,” and that in airline crashes they studied it was found that “black box recordings showed that the crews talked about ‘how the systems sure were screwed up’ but did not verify what was wrong. In both cases, the systems worked but crews failed to check the information and crashed.”

Similarly, research published in 2001 by the Royal Institute of Technology examined “the alarm problem” in aviation, meaning, “in the most critical situations with the highest cognitive load for the pilots, the technology lets you down.” The reports noted that “the warning system of the modern cockpits are not always easy to use and understand. The tendency is to overload the display with warnings, cautions and inoperative system information accompanied by various audio warnings.” It went on to identify one of the main problems as a result of this overload as “a cognitive problem of understanding and evaluating from the displayed information which is the original fault and which are the consecutive faults.” Sound familiar? You would likely hear something extremely similar from someone working in today’s SOC.

In the decades that followed, aircraft cockpit design has progressively applied new learnings and automation to dynamically manage alert volume and the attention of the pilot to priorities. In the Royal Institute of Technology’s report, researchers identified accident simulation as an effective tool for improving cockpit alert systems, finding more associable ways to present alerts such as differentiating sounds and the introduction of context, which would allow pilots to “immediately understand what part or function of the aircraft is suffering a malfunction.” More context would also include guidance on what to do next. In its conclusion the study noted:

Such simulations would hopefully result in less cognitive stress on behalf of the pilots: they would know that they have started to solve the right problem. They would not have to worry that they have entered the checklist at the wrong place. With a less stressful situation even during malfunctions there is greater hope for correct actions being taken, leading to increased flight safety.

SOC systems need to embrace and apply many of these same learnings that have spanned decades for aviation. The majority of the cybersecurity industry seems to have only gotten as far as color coding alert and warning significance, leaving the analyst faced with a hundred flashing red priorities, even after triaging it. It’s no surprise that they’re both overwhelmed and unable to respond to complex threats across a broadening attack surface.

Beware of Autopilot

When it comes to solving the issue of alert fatigue, automation is typically one of the first things to come to mind. The same went for aviation in 1988, where the previously mentioned Washington Post report quoted researchers saying what could have been taken right from a security trade publication in 2022:

Research is badly needed to understand just how much automation to introduce — and when to introduce it — in situations where the ultimate control and responsibility must rest with human operators, said psychologist Richard Pew, manager of the experimental psychology department at BBN Systems and Technologies Corp. in Cambridge, Mass.

“Everywhere we look we see the increasing use of technology,” Pew said. “In those situations where the operator has to remain in control, I think that we have to be very careful about how much automation we add.”

The growing use of high-tech devices in the cockpit or on ships can have two seemingly contradictory effects. One response is to lull crew members into a false sense of security. They “regard the computer’s recommendation as more authoritative than is warranted,” Pew said. “They tend to rely on the system and take a less active role in control.” Sometimes crews are so mesmerized by technological hardware that they are lulled into what University of Texas psychologist Robert Helmreich calls “automation complacency.”

And while automation of course has an important part to play in incident response and investigation — just as it does in modern aircraft cockpit design — it comes with some key warnings:

1. Situational awareness is lost. Automation is often brittle, unable to operate outside of the situations it is programmed for, and subject to inappropriate performance due to faulty sensors or limited knowledge about a situation.
2. Automation creates high workload spikes (such as when routine changes or a problem occurs) and long periods of boredom (in which attention wavers and response to exceptions may be missed). If you’re staffing for automation-level activities, how do you manage capacity for spikes?

The SOC Earns its Wings

As an industry we have to take a page from the aircraft handbook and avoid increasing cognitive demands, workload and distractions, and make tasks easier to perform. But we must also understand how to manage automation failure and exceptions better.

• Embrace AI and autocomplete: Like the more advanced sentence autocomplete functions appearing in email and word processing applications, SOC analysts are still in charge of managing an incident, but there is an opportunity to further guide and preemptively enrich a threat investigation, thereby increasing the speed and robustness of response.
• Distill and prioritize at the incident level, not the alert level: It’s not about filtering/correlating/aggregating alerts, it’s about contextualizing both events and alerts in the background and only articulating an incident in plain single-sentence language. Analysts can double-click down from there.
• Leverage a community of experts: As attack surfaces increase and vertical technology specialization becomes tougher for in-house SOCs to cover (particularly in times of competing incident prioritization), it becomes increasingly important to be able to “phone-a-friend” and access an on-demand global pool of expert talent. It’s like having several Boeing engineers sitting in the cockpit with the pilot to troubleshoot a problem with the plane.

-- Gunter Ollmann

First Published: Medium - March 29, 2022