Measuring and Increasing Code Quality

At some point in a CTO’s career questions will be raised about “code quality” under their watch. Engineering teams will typically associate code quality with bugs and feature release velocity, while Product Management and Customer Success organizations will often define it in terms of platform stability and reliability or customer-derived metrics. 

How do you quantify code quality? What strategies are there for increasing code quality? How will you know if code quality is improving? – These are the questions myself and every other product or engineering leader ends up facing at some point as product development teams become larger and  products mature.

Key Metrics to Gather

From a CTO perspective, key go-to metrics typically include a subset or all of the following:

1. Cyclomatic Complexity – Measures the number of linearly independent paths through the code.
1. Benefits: Helps identify complex and potentially error-prone code that is difficult to maintain and test.
2. Change Over Time: Should decrease as code is refactored and simplified.
2. Code Churn – Measures the frequency of code changes.
1. Benefits: High churn can indicate unstable code or unclear requirements.
2. Change Over Time: Should stabilize as the codebase matures and becomes more stable.
3. Code Coverage - Percentage of code covered by automated tests.
1. Benefits: Higher coverage indicates better-tested code, reducing the likelihood of defects.
2. Change Over Time: Should increase as more tests are added, aiming for 80% or higher.
4. Technical Debt - Represents the cost of additional rework caused by choosing an easy solution now instead of a better approach that would take longer.
1. Benefits: Helps prioritize refactoring efforts and manage long-term code quality.
2. Change Over Time: Should decrease as technical debt is addressed and reduced.
5. Defect Density - Number of defects per unit of code (e.g., per 1,000 lines of code).
1. Benefits: Indicates the quality of the codebase and effectiveness of testing.
2. Change Over Time: Should decrease as code quality improves and defects are fixed.
6. Maintainability Index - A composite metric that includes cyclomatic complexity, lines of code, and Halstead volume.
1. Benefits: Provides an overall measure of how easy the code is to maintain.
2. Change Over Time: Should improve as code is refactored and simplified.

What does “success” look like?

• Reduced Cyclomatic Complexity - Functions and methods become simpler and easier to understand, leading to fewer bugs and easier maintenance.
• Stabilized Code Churn - Indicates that the codebase is stable, with fewer frequent changes, suggesting better initial design and clearer requirements.
• Increased Code Coverage - Higher test coverage means more of the code is tested, reducing the likelihood of defects and increasing confidence in the codebase.
• Decreased Technical Debt - Lower technical debt means the codebase is cleaner and more maintainable, reducing long-term costs and improving developer productivity.
• Lower Defect Density - Fewer defects per unit of code indicate higher code quality and more effective testing processes.
• Improved Maintainability Index - A higher maintainability index means the code is easier to understand, modify, and extend, leading to more efficient development processes.

By continuously monitoring these metrics and making data-driven decisions, you can systematically improve code quality, leading to more reliable, maintainable, and scalable software products.

From Metrics to Strategy

Modern software development pipelines and tooling are pretty efficient at generating these important metrics. I’ve found the following four strategies most efficient.

• Static Code Analysis
• Common Tools: SonarQube, Checkstyle, PMD, SpotBugs
• Key Metrics:
• Code Coverage: Measures the extent to which code is covered by automated tests.
• Technical Debt: Quantifies the effort required to fix issues and improve code quality.
• Security Vulnerabilities: Identifies potential security risks.
• Code Duplication: Detects redundant code sections.
• Complexity: Assesses the complexity of code, including cyclomatic complexity and nesting depth.
• Strategies for Improvement:
• Regular Static Analysis: Schedule regular static analysis runs to identify and address issues early.
• Code Review: Encourage developers to review code for potential issues and suggest improvements.
• Refactoring: Refactor code to improve readability, maintainability, and performance.
• Dynamic Code Analysis
• Common Tools: JUnit, TestNG, Selenium
• Key Metrics:
• Test Coverage: Measures the extent to which code is covered by tests.
• Test Failure Rate: Tracks the frequency of test failures.
• Test Execution Time: Monitors the time taken to run test suites.
• Strategies for Improvement:
• Write Comprehensive Tests: Develop thorough unit, integration, and end-to-end tests.
• Test Automation: Automate tests to increase efficiency and reduce manual effort.
• Test-Driven Development (TDD): Write tests before writing code to ensure quality and functionality.
• Code Reviews
• Common Tools: GitHub, GitLab, Bitbucket
• Key Metrics:
• Review Time: Measures the average time taken to review code changes.
• Review Comments: Tracks the number of comments and suggestions made during reviews.
• Defect Density: Calculates the number of defects found per line of code.
• Strategies for Improvement:
• Establish Clear Guidelines: Define clear guidelines for code reviews, including formatting, commenting, and testing standards.
• Encourage Timely Reviews: Promote timely reviews to avoid bottlenecks and delays.
• Provide Constructive Feedback: Provide constructive feedback to improve code quality and foster a positive review culture.
• Continuous Integration and Continuous Delivery (CI/CD)
• Common Tools: Jenkins, CircleCI, GitLab CI/CD
• Key Metrics:
• Build Success Rate: Measures the percentage of successful builds.
• Deployment Frequency: Tracks the frequency of deployments to production.
• Mean Time to Recovery (MTTR): Measures the time taken to recover from failures.
• Strategies for Improvement:
• Automate the Build Process: Automate the build, test, and deployment processes to reduce manual effort and errors.
• Implement Automated Testing: Integrate automated tests into the CI/CD pipeline to catch issues early.
• Monitor Deployment Metrics: Monitor key metrics to identify and address performance bottlenecks and failures.

Final Thoughts

In addition to measuring code quality, it's essential to implement strategies to continuously help improve the scores you're tracking.

• Foster a Culture of Quality - Encourage developers to prioritize code quality and take ownership of their work.
• Provide Training and Development Opportunities - Invest in training and development to improve developers' skills and knowledge.
• Use Code Quality Tools Effectively - Utilize code quality tools to identify and address issues proactively.
• Regularly Review and Refactor Code - Schedule regular code reviews and refactoring sessions to improve code quality and maintainability.
• Encourage Pair Programming - Pair programming can help improve code quality, knowledge sharing, and collaboration.
• Establish Coding Standards and Guidelines - Define clear coding standards and guidelines to ensure consistency and maintainability.

-- Gunter Ollmann

Getting Your SOC SOARing Despite AI

It’s a fact: enterprise security operations centers (SOCs) that are most satisfied with their investments in Security Information and Event Management (SIEM) and Security Orchestration, Automation, and Response (SOAR) operate and maintain less than a dozen playbooks. This is something I’ve uncovered in recent years whilst building SIEM+SOAR and autonomous SOC solutions – and it perhaps runs counterintuitive to many security leaders’ visions for SOAR use and value.

SOAR technology is one of those much-touted security silver bullets that have tarnished over time and been subsumed into broader categories of threat detection and incident response (TDIR) solutions, yet it continues to remain a distinct must-have modern SOC capability.

Why do satisfied SOCs run so few playbooks? After all, a core premise of SOAR was that you can automate any (perhaps all) security responses, and your SOC analysts would spend less time on ambient security noise – giving them more time to focus on higher-priority incidents. Surely “success” would include automating as many responses as possible?

Beyond the fact that “security is hard,” the reality is that threat detection and response is as dynamic as the organization you’re trying to protect. New systems, new policies, new business owners, new tools, and a sea of changing security products and API connector updates mean that playbooks must be dynamic and vigilantly maintained, or they become stale, broken, and ineffective.

Every SOC team has at least one playbook covering their phishing response. It’s one of the most common and frequently encountered threats within the enterprise, yet “phishing” covers an amazingly broad range of threats and possible responses, so a playbook-based response to the threat is programmatically very complex and brittle to environmental changes.

From a SOC perspective of automating and orchestrating a response, you would either build a lengthy single if/then/else-stye playbook or craft individual playbooks for each permutation of the threat. Smart SOC operators quickly learn that the former is more maintainable and scalable than the latter. A consequence of this is that you need analysts with more experience to maintain and operate the playbook. Any analyst can knock-up a playbook for a simple or infrequently encountered threat vector, but it takes business knowledge and vigilance to maintain each playbook’s efficacy beyond the short term.

Surely AI and a sprinkle of LLM magic will save the day though, right?

I can’t count the number of security vendors and startups that have materialized over the last couple of years with AI and LLM SOAR capabilities, features, or solutions – all with pitches that suggest playbooks are dead, dying, replaced, accelerated, automatically maintained, dynamically created, managed, open sourced, etc., so the human SOC analyst does less work going forward. I remain hopeful that 10% of any of that eventually becomes true.

For the immediate future, SOC teams should continue to be wary of any AI stories that emphasize making it easier to create playbooks (or their product-specific equivalent of a playbook). More is NOT better. It’s too easy (and all too common) to fall down the rathole of creating a new playbook for an existing threat because it’s too hard to find and maintain an earlier iteration of that threat’s playbook. Instead, focus on a subset of the most common and time-consuming threats that your SOC already faces daily, and nail them using the smallest number of playbooks you can get away with.

With the Rolling Stones’ “(I Can’t Get No) Satisfaction” playing in the background, perhaps you’ll get some modicum of SIEM+SOAR satisfaction by keeping your playbook playlist under a dozen.

-- Gunter Ollmann

First Published: IOActive Blog - October 15, 2024

Get Strategic About Cyber Risk Management

With global cybercrime damage costs exceeding $11 trillion last year and moving toward an estimated $20 trillion by 2026, robust cybersecurity risk management has never been more imperative.

The interconnected nature of modern technology means that, by default, even small vulnerabilities can lead to catastrophic losses. And it’s not just about finances. Unmitigated risk raises the specter of eroded customer confidence and tainted brand reputation. In this comprehensive guide, we’ll give enterprise defenders a holistic, methodical, checklist-style approach to cybersecurity risk management. We’ll focus on practical applications, best practices, and ready-to-implement strategies designed to mitigate risks and safeguard digital assets against ever-more numerous—and increasingly capable—threats and adversaries.

What is Cybersecurity Risk Management?

This subspecialty of enterprise risk management describes a systematic approach to identifying, analyzing, evaluating, and addressing cyber threats to an organization’s assets and operations. At its core, it involves a continuous cycle of risk assessment, risk decision-making, and the implementation of risk controls intended to minimize the negative impact of cyber incidents.

A proactive cyber risk mitigation approach helps organizations protect critical digital assets and bolster business continuity, legal compliance, and customer trust. By integrating risk management with the organization’s overall strategic planning, cybersecurity teams can prioritize resources efficiently and align their efforts with the business’s risk appetite and objectives.

Why Has Cyber Risk Management Become So Critical?

Getting control over cyber risk is quickly becoming a core requirement for businesses operating in today’s digital ubiquity. The proliferation of digital information and internet connectivity have paved the way for sophisticated cyber threats that can penetrate many of our most robust defenses. With the digital footprint of businesses expanding exponentially, the potential for data breaches, ransomware attacks, and other forms of cybercrime has escalated dramatically.

These incidents can result in devastating financial losses, legal repercussions, and irreparable damage to an organization’s reputation. Furthermore, as regulatory frameworks around data protection become more stringent, failure to comply can lead to significant penalties. Given these conditions, an aggressive and comprehensive approach to managing cybersecurity risks is crucial for safeguarding an organization’s assets, ensuring operational continuity, and maintaining trust with customers and stakeholders.

Effective Cyber Risk Management: A Framework-Based Approach

Adopting a structured, framework-based approach to cybersecurity risk management lets security teams corral the complexity of digital environments with a methodical, strategic mitigation methodology. For most enterprise applications, there’s no need to reinvent the wheel. There are a myriad of established frameworks that can be modified and customized for effective use in nearly any environment.

Perhaps the best known is the National Institute of Standards and Technology (NIST) Risk Management Framework (RMF), a companion to NIST’s well-tested and widely implemented Cybersecurity Framework (CSF). The NIST RMF offers a structured and systematic approach for integrating security, privacy, and risk management processes into an organization’s system development life cycle.

Such frameworks provide a comprehensive set of guidelines that help identify and assess cyber threats and facilitate the development of effective strategies to mitigate these risks. By standardizing cybersecurity practices, organizations can ensure a consistent and disciplined application of security measures across all departments and operations.

This coherence and uniformity are crucial for effectively addressing vulnerabilities and responding to incidents promptly. Equally important, frameworks incorporate best practices and benchmarks that help guide organizations toward achieving compliance with regulatory requirements, thus minimizing legal risks and enhancing the safeguarding of customer data. In essence, a framework-based approach offers a clear roadmap for managing cyber risk in a way that’s aligned with organizational strategic objectives and industry standards.

What follows is a checklist based on the 7-step RMF process. This is just a starting point. A framework to-do list like this can and should be tweaked to aid in reducing and managing specific cyber risks in your unique enterprise environment.

1. Preparation

In this initial phase, organizations focus on establishing the context and priorities for the Risk Management Framework process. This involves identifying critical assets, defining the boundaries, and codifying a risk management strategy that aligns with the organization’s objectives and resources. This is the foundation upon which a tailored approach to managing cybersecurity risk will ultimately be built throughout the system’s lifecycle.

• Establish the context for risk management and create a risk management strategy.
• Define roles and responsibilities across the organization.
• Develop a taxonomy for categorizing information and information systems.
• Determine the legal, regulatory, and contractual obligations.
• Prepare an inventory of system elements, including software and hardware.

2. Systems Categorization

Expanding on the categorization step (above), this phase involves identifying the types of information processed, stored, and transmitted to determine potential impact as measured against the information security CIA triad (confidentiality, integrity, and availability). Organizations can assign appropriate security categories to their systems by leveraging a categorization standard such as the Federal Information Processing Standard (FIPS) 199, ensuring that the protective measures taken are tailored to the specific needs and risks associated with the information being handled. This step is crucial as it lays the groundwork for selecting suitable security controls in the later stages of the risk management process.

• Identify the types of information processed, stored, and transmitted by the system.
• Assess the potential impact of loss of Confidentiality, Integrity, and Availability (CIA) associated with each type.
• Document findings in a formal security categorization statement.

3. Selecting Appropriate Security Controls

This critical step begins the safeguarding of information systems against potential threats and vulnerabilities in earnest. Based on the categorization of the information system, organizations select a baseline of security and privacy controls (NIST Special Publication 800-53 or some equivalent controls standard is a good starting point here), corresponding to the system’s impact level. This baseline acts as the jumping-off point for the security controls, which can be tailored to address the specific risks identified throughout the risk assessment process. Customization involves adding, removing, or modifying controls to ensure a robust defense tailored to the unique requirements and challenges of the organization.

• Select an appropriate baseline of security controls (NIST SP 800-53 or equivalent).
• Tailor the baseline controls to address specific organizational needs and identified risks.
• Document the selected security controls in the system security plan.
• Develop a strategy for continuously monitoring and maintaining the effectiveness of security controls.

4. Implementing the Selected Controls

Implementing security controls involves the physical and technical application of measures chosen during the previous selection phase. This step requires careful execution to ensure all controls are integrated effectively within the environment, aligning with its architecture and operational practices. Documenting the implementation details is crucial to provide a reference for future assessments and maintenance activities.

• Implement the security controls as documented in Step 3.
• Document the security controls and the responsible entities in place.
• Test thoroughly to ensure compatibility and uninterrupted functionality.
• Prepare for security assessment by documenting the implementation details.

5. Assessing Controls Performance

Assessing security controls involves evaluating effectiveness and adherence to the security requirements outlined in the overall security plan. This phase is critical for identifying any control deficiencies or weaknesses that could leave the information system vulnerable. Independent reviewers or auditors typically conduct assessments to ensure objectivity and a comprehensive analysis.

• Develop and implement a plan to assess the security controls.
• Perform security control assessments as per the plan.
• Prepare a Security Assessment Report (SAR) detailing the effectiveness of the controls.
• Determine if additional controls are needed and append the master security plan accordingly.

6. Authorizing the Risk Management Program

The authorization phase is a vital decision-making interval where one or more senior executives evaluate the security controls’ assessment results and decide whether the remaining risks to the information systems are acceptable to the organization. Upon acceptance, authorization is granted to operate the mitigation program for a specific time period, during which its compliance and security posture are continuously monitored. This authorization is formalized through the issuance of what is known as an Authorization to Operate (ATO) in some organizations, particularly in the public sector.

• Compile the required authorization package, including the master plan, the SAR, and the so-called Plan of Action and Milestones (POA&M).
• Assess the residual risk against the organizational risk tolerance.
• Document the authorization decision in an Authorization Decision Document.

7. Monitoring and Measuring Against Performance Metrics

The monitoring phase ensures that all implemented security controls remain effective and compliant over time. Continuous surveillance, reporting, and analysis can promptly address any identified vulnerabilities or changes in the operational environment. This ongoing process supports the kind of flexible, adaptive security posture necessary for dealing with evolving threats while steadfastly maintaining the integrity and availability of the information system.

• Implement the plan for ongoing monitoring of security controls.
• Report the system’s security state to designated leaders in the organization.
• Perform ongoing risk assessments and response actions, updating documentation as necessary.
• Conduct reviews and updates regularly, in accordance with the organizational timelines, or as significant changes occur.

Conclusion: Formalizing Cyber Risk Mitigation

A solid risk management framework provides a comprehensive guide for enhancing the security and resilience of information systems through a structured process of identifying, implementing, and monitoring security controls.

Sticking to a framework checklist helps ensure a successful, systematic adoption. As noted throughout, engaging stakeholders from across the organization, including IT, security, operations, and compliance, is critical to ensuring a truly comprehensive risk management program. Additionally, periodic training and awareness for team members involved in various phases of the risk management project will contribute to the resilience and security of the organization’s digital assets.

Organizations can effectively safeguard their digital assets and mitigate unacceptable risks by following the outlined steps, tailoring the program to fit specific organizational needs, involving stakeholders, conducting regular training, and adapting to the evolving cybersecurity landscape. Ultimately, this kind of formal, structured cyber risk management fosters a culture of continuous improvement and vigilance in an enterprise, contributing to the overall security posture and the success of the organization.

-- Gunter Ollmann

First Published: IOActive Blog - August 20, 2024

The Security Imperative in Artificial Intelligence

Artificial Intelligence (AI) is transforming industries and everyday life, driving innovations once relegated to the realm of science fiction into modern reality. As AI technologies grow more integral to complex systems like autonomous vehicles, healthcare diagnostics, and automated financial trading platforms, the imperative for robust security measures increases exponentially.

Securing AI is not only about safeguarding data but also about ensuring the core systems — in particular, the trained models that really put the “intelligence” in AI — function as intended without malicious interference. Historical lessons from earlier technologies offer some guidance and can be used to inform today’s strategies for securing AI systems. Here, we’ll explore the evolution, current state, and future direction of AI security, with a focus on why it’s essential to learn from the past, secure the present, and plan for a resilient future.

AI: The Newest Crown Jewel

Security in the context of AI is paramount precisely because AI systems increasingly handle sensitive data, make important, autonomous decisions, and operate with limited supervision in critical environments where safety and confidentiality are key. As AI technologies burrow further into sectors like healthcare, finance, and national security, the potential for misuse or harmful consequences due to security shortcomings rises to concerning levels. Several factors drive the criticality of AI security:

• Data Sensitivity: AI systems process and learn from large volumes of data, including personally identifiable information, proprietary business information, and other sensitive data types. Ensuring the security of enterprise training data as it passes to and through AI models is crucial to maintaining privacy, regulatory compliance, and the integrity of intellectual property.
• System Integrity: The integrity of AI systems themselves must be well defended in order to prevent malicious alterations or tampering that could lead to bogus outputs and incorrect decisions. In autonomous vehicles or medical diagnosis systems, for example, instructions issued by compromised AI platforms could have life-threatening consequences.
• Operational Reliability: AI is increasingly finding its way into critical infrastructure and essential services. Therefore, ensuring these systems are secure from attacks is vital for maintaining their reliability and functionality in critical operations.
• Matters of Trust: For AI to be widely adopted, users and stakeholders must trust that the systems are secure and will function as intended without causing unintended harm. Security breaches or failures can undermine public confidence and hinder the broader adoption of emerging AI technologies over the long haul.
• Adversarial Activity: AI systems are uniquely susceptible to certain attacks, whereby slight manipulations in inputs — sometimes called prompt hacking — can deceive an AI system into making incorrect decisions or spewing malicious output. Understanding the capabilities of malicious actors and building robust defenses against such prompt-based attacks is crucial for the secure deployment of AI technologies.

In short, security in AI isn’t just about protecting data. It’s also about ensuring safe, reliable, and ethical use of AI technologies across all applications. These inexorably nested requirements continue to drive research and ongoing development of advanced security measures tailored to the unique challenges posed by AI.

Looking Back: Historical Security Pitfalls

We don’t have to turn the clock back very far to witness new, vigorously hyped technology solutions wreaking havoc on the global cybersecurity risk register. Consider the peer-to-peer recordkeeping database mechanism known as blockchain.  When blockchain exploded into the zeitgeist circa 2008 — alongside the equally disruptive concept of cryptocurrency — its introduction brought great excitement thanks to its potential for both decentralization of data management and the promise of enhanced data security. In short order, however, events such as the DAO hack —an exploitation of smart contract vulnerabilities that led to substantial, if temporary, financial losses — demonstrated the risk of adopting new technologies without diligent security vetting.

As a teaching moment, the DAO incident highlights several issues: the complex interplay of software immutability and coding mistakes; and the disastrous consequences of security oversights in decentralized systems. The case study teaches us that with every innovative leap, a thorough understanding of the new security landscape is crucial, especially as we integrate similar technologies into AI-enabled systems.

Historical analysis of other emerging technology failures over the years reveals other common themes, such as overreliance on untested technologies, misjudgment of the security landscape, and underestimation of cyber threats. These pitfalls are exacerbated by hype-cycle-powered rapid adoption that often outstrips current security capacity and capabilities. For AI, these themes underscore the need for a security-first approach in development phases, continuous vulnerability assessments, and the integration of robust security frameworks from the outset.

Current State of AI Security

With AI solutions now pervasive, each use case introduces unique security challenges. Be it predictive analytics in finance, real-time decision-making systems in manufacturing systems, or something else entirely,  each application requires a tailored security approach that takes into account the specific data types and operational environments involved. It’s a complex landscape where rapid technological advancements run headlong into evolving security concerns. Key features of this challenging  infosec environment include:

• Advanced Threats: AI systems face a range of sophisticated threats, including data poisoning, which can skew an AI’s learning and reinforcement processes, leading to flawed outputs; model theft, in which proprietary intellectual property is exposed; and other adversarial actions that can manipulate AI perceptions and decisions in unexpected and harmful ways. These threats are unique to AI and demand specialized security responses that go beyond traditional cybersecurity controls.
• Regulatory and Compliance Issues: With statutes such as GDPR in Europe, CCPA in the U.S., and similar data security and privacy mandates worldwide, technology purveyors and end users alike are under increased pressure to prioritize safe data handling and processing. On top of existing privacy rules, the Biden administration in the U.S. issued a comprehensive executive order last October establishing new standards for AI safety and security. In Europe, meanwhile, the EU’s newly adopted Artificial Intelligence Act provides granular guidelines for dealing with AI-related risk. This spate of new rules can often clash with AI-enabled applications that demand more and more access to data without much regard for its origin or sensitivity.
• Integration Challenges: As AI becomes more integrated into critical systems across a wide swath of vertical industries, ensuring security coherence across different platforms and blended technologies remains a significant challenge. Rapid adoption and integration expose modern AI systems to traditional threats and legacy network vulnerabilities, compounding the risk landscape.
• Explainability: As adoption grows, the matter of AI explainability  — or the ability to understand and interpret the decisions made by AI systems — becomes increasingly important. This concept is crucial in building trust, particularly in sensitive fields like healthcare where decisions can have profound impacts on human lives.Consider an AI system used to diagnose disease from medical imaging. If such a system identifies potential tumors in a scan, clinicians and patients must be able to understand the basis of these conclusions to trust in their reliability and accuracy. Without clear explanations, hesitation to accept the AI’s recommendations ensues, leading to delays in treatment or disregard of useful AI-driven insights. Explainability not only enhances trust, it also ensures AI tools can be effectively integrated into clinical workflows, providing clear guidance that healthcare professionals can evaluate alongside their own expertise.

Addressing such risks requires a deep understanding of AI operations and the development of specialized security techniques such as differential privacy, federated learning, and robust adversarial training methods. The good news here: In response to AI’s risk profile, the field of AI security research and development is on a steady growth trajectory. Over the past 18 months the industry has witnessed  increased investment aimed at developing new methods to secure AI systems, such as encryption of AI models, robustness testing, and intrusion detection tailored to AI-specific operations.

At the same time, there’s also rising awareness of AI security needs beyond the boundaries of cybersecurity organizations and infosec teams. That’s led to better education and training for application developers and users, for example, on the potential risks and best practices for securing A-powered systems.

Overall,  enterprises at large have made substantial progress in identifying and addressing AI-specific risk, but significant challenges remain, requiring ongoing vigilance, innovation, and adaptation in AI defensive strategies.

Data Classification and AI Security

One area getting a fair bit of attention in the context of safeguarding AI-capable environments is effective data classification. The ability to earmark data (public, proprietary, confidential, etc.) is essential for good AI security practice. Data classification ensures that sensitive information is handled appropriately within AI systems. Proper classification aids in compliance with regulations and prevents sensitive data from being used — intentionally or unintentionally — in training datasets that can be targets for attack and compromise.

The inadvertent inclusion of personally identifiable information (PII) in model training data, for example, is a hallmark of poor data management in an AI environment. A breach in such systems not only compromises privacy but exposes organizations to profound legal and reputational damage as well. Organizations in the business of adopting AI to further their business strategies must be ever aware of the need for stringent data management protocols and advanced data anonymization techniques before data enters the AI processing pipeline.

The Future of AI Security: Navigating New Horizons

As AI continues to evolve and tunnel its way further into every facet of human existence, securing these systems from potential threats, both current and future, becomes increasingly critical. Peering into AI’s future, it’s clear that any promising new developments in AI capabilities must be accompanied by robust strategies to safeguard systems and data against the sophisticated threats of tomorrow.

The future of AI security will depend heavily on our ability to anticipate potential security issues and tackle them proactively before they escalate. Here are some ways security practitioners can prevent future AI-related security shortcomings:

• Continuous Learning and Adaptation: AI systems can be designed to learn from past attacks and adapt to prevent similar vulnerabilities in the future. This involves using machine learning algorithms that evolve continuously, enhancing their detection capabilities over time.
• Enhanced Data Privacy Techniques: As data is the lifeblood of AI, employing advanced and emerging data privacy technologies such as differential privacy and homomorphic encryption will ensure that data can be used for training without exposing sensitive information.
• Robust Security Protocols: Establishing rigorous security standards and protocols from the initial phases of AI development will be crucial. This includes implementing secure coding practices, regular security audits, and vulnerability assessments throughout the AI lifecycle.
• Cross-Domain Collaboration: Sharing knowledge and strategies across industries and domains can lead to a more robust understanding of AI threats and mitigation strategies, fostering a community approach to AI security.

Looking Further Ahead

Beyond the immediate horizon, the field of AI security is set to witness several meaningful advancements:

• Autonomous Security: AI systems capable of self-monitoring and self-defending against potential threats will soon become a reality. These systems will autonomously detect, analyze, and respond to threats in real time, greatly reducing the window for attacks.
• Predictive Security Models: Leveraging big data and predictive analytics, AI can forecast potential security threats before they manifest. This proactive approach will allow organizations to implement defensive measures in advance.
• AI in Cybersecurity Operations: AI will increasingly become both weapon and shield. AI is already being used to enhance cybersecurity operations, providing the ability to sift through massive amounts of data for threat detection and response at a speed and accuracy unmatchable by humans. The technology and its underlying methodologies will only get better with time. This ability for AI to remove the so-called “human speed bump” in incident detection and response will take on greater importance as the adversaries themselves increasingly leverage AI to generate malicious attacks that are at once faster, deeper, and potentially more damaging than ever before.
• Decentralized AI Security Frameworks: With the rise of blockchain technology, decentralized approaches to AI security will likely develop. These frameworks can provide transparent and tamper-proof systems for managing AI operations securely.
• Ethical AI Development: As part of securing AI, strong initiatives are gaining momentum to ensure that AI systems are developed with ethical considerations in mind will prevent biases and ensure fairness, thus enhancing security by aligning AI operations with human values.

As with any rapidly evolving technology, the journey toward a secure AI-driven future is complex and fraught with challenges. But with concerted effort and prudent innovation, it’s entirely within our grasp to anticipate and mitigate these risks effectively. As we advance, the integration of sophisticated AI security controls will not only protect against potential threats, it will foster trust and promote broader adoption of this transformative technology. The future of AI security is not just about defense but about creating a resilient, reliable foundation for the growth of AI across all sectors.

Charting a Path Forward in AI Security

Few technologies in the past generation have held the promise for world-altering innovation in the way AI has. Few would quibble with AI’s immense potential to disrupt and benefit human pursuits from healthcare to finance, from manufacturing to national security and beyond. Yes, Artificial Intelligence is revolutionary. But it’s not without cost. AI comes with its own inherent collection of vulnerabilities that require vigilant, innovative defenses tailored to their unique operational contexts.

As we’ve discussed, embracing sophisticated, proactive, ethical, collaborative AI security and privacy measures is the only way to ensure we’re not only safeguarding against potential threats but also fostering trust to promote the broader adoption of what most believe is a brilliantly transformative technology.

The journey towards a secure AI-driven future is indeed complex and fraught with obstacles. However, with concerted effort, continuous innovation, and a commitment to ethical practices, successfully navigating these impediments is well within our grasp. As AI continues to evolve, so too must our strategies for defending it. 

-- Gunter Ollmann

First Published: IOActive Blog - May 30, 2024

GenAI Influencing the Secure Code Review Profession

 It’s tough to be a secure code reviewer. There are already over 700 programming languages according to Wikipedia, and seemingly more languages materializing every year. Expectations are high that rapid developments in Generative Artificial Intelligence (GenAI) will bring a new suite of languages and security issues that’ll have an oversized impact on software development. Consequently, secure software development lifecycle (SDL) processes and security code review are having to evolve rapidly.

I’m both excited and nervous about GenAI advancements in the world of software development and secure application design. It’s exciting to see how prompt engineering of Large Language Models (LLM) and adoption of AI augmentation in the form of copilots and chatbots are increasing the pace of ideation into new products. I’m nervous about the hallucinations and code quality being generated in response though.

English as a Programming Language

2023 was the breakthrough year for AI, with LLM and GenAI permeating every industry, technology, and product. Today, the most in-demand languages currently are Python, C, and C++ but, controversially, the future star programming language may in fact be English; something that’ll take some time to adjust to.

For over a decade we’ve been told that the supply of experienced cybersecurity professionals has trailed the market’s requirements, with a deficit growing year-on-year, and a casual scan across office desks and cubicles will highlight a more significant gender gap across the cybersecurity (and software development) industry. I think AGI and emergence of English as a critical programming language are fundamental to correcting both industry problems.

GenAI, particularly those based upon LLM advancements, are increasingly sophisticated language machines – and women may have an advantage over men in maximizing utility and productivity from them.

Multiple studies over the last 30 years have constantly highlighted that women are better communicators than men. “Better” is obviously an explosive and controversial term even amongst the academics who published the studies, but in general women have more expansive vocabularies and stronger interpretative communication skills. Modern neuroscience and studies in children and adolescents identify girls as more garrulous than boys, with greater complexity and sophistication of language, and tend to develop more in the realm of listening with greater focus and concentration as they age. This historically translates into women being better coders than men (once you remove the bias in the system).

As I look to GenAI and the expanding world of prompt engineering, I anticipate that women will have an advantage over their male developer counterparts. Strong and well-developed communication skills (and the reasoning and understanding that underlays those polished skills) are prerequisites for maximizing efficiency of GenAI-returned results and tuning responses – both now and for the immediate future.

Starter-job Experience

But what about experience? The “experience gap” is often called out as a chasm for newly minted degree-holding graduates and landing a starter-job in cybersecurity.

It’s rare to find an entry-level job in our industry that doesn’t require multiple years of hands-on security experience nowadays as many of those traditional starter roles – network scanning, alert triage, playbook maintenance, patch management – have been automated away, with many more projected to disappear as AI adoption increases.

Most successful new entrants into the cybersecurity profession come from adjacent technical industries making a career jump rather than direct from a college or university. Armed with transferable skills and technical experience, they’re capable of crossing the chasm left in the wake of cyber automation. However, the security knowledge gap between a cybersecurity veteran and a recent transfer remains large and a growing concern for the industry.

I’m excited to think AI augmentation and copilot technologies will have one of the largest impacts on our industry – removing much of the security knowledge gap and reducing the overall impact of the experience gap – like what is happening in other industries, such as the medical field. For example, AI use in patient triage, predictive analytics, and virtual assistants are augmenting generalist regional nurses (two-year qualification) and Bachelor of Science in Nursing (four-year qualification) graduates, and allowing them to perform many of the roles and responsibilities traditionally associated with a completed medical doctor degree (10 to 12 years).

Secure Code Reviews

It’s tough to be a secure code reviewer. There aren’t enough of them. The job requires tremendous amounts of experience and advanced security knowledge, and it’s tiring and hard work.

GenAI is going to have a huge impact on their job.

On the positive side, English as a programming language and AI augmentation and copilots is going to help increase both the breadth and depth of the cybersecurity talent pool available to perform this critical job. The tools available to code reviewers to automatically review and assess code security are advancing quickly and, while still in their first generation of AI adoption, are anticipated to mature rapidly and identify vulnerabilities and logic flaws with higher fidelity and trust. I’m sure there’ll be a gap between the best that a tool can achieve versus the best-of-the-best human expert though – especially when that expert is augmented and using similar tools themselves.

Meanwhile, GenAI is spearheading prompt engineering of new software applications. A new generation of product developers may have little to no influence over the code that powers the application. Indeed, I’ve previously argued that the role of product manager will change greatly in the coming years as their skills in product design and requirement setting pivot from being directed to engineering teams and into GenAI prompts instead.

What does an GenAI-generated application look like under the covers? Time will tell. We anticipate it’ll increasingly become more secure – using best security practices and recycling pretested code behind the scenes – and that it’ll constantly learn, optimize, and apply best and better security practices – but we’ll still need those human secure code reviewers for some time to come, specially when it comes to high-impact applications and certification.

A concern though as GenAI does more of the application development software developers will have less direct influence over the underlying code is that powering the application and business logic. It would be a nightmare if the AGI produced entirely different code throughout the application each time it received new design criteria and re-optimized – making vulnerability triage, reporting, reconciliation, and tracking near impossible, and code reviews and code certifications (human or tool led) largely nonsensical.

Watch this space!

I think we’re still some years away from having to worry about continuously reimagined code generated by GenAI without human software developers tweaking and refining the underlaying application code, but it is tremendously exciting to see the rapid advances in prompt engineering and how LLM’s are being incorporated into both old and new products.

Our industry has consistently struggled to attract and retain women. GenAI has the potential to not only level the field and make it easier to join the game, but to also leverage previously poorly-tapped communication skills for the betterment of both application development and security. There’s a lot of work ahead. There’s a lot of research to be done. There’s a lot of opportunities to make code more secure!

-- Gunter Ollmann

First Published: IOActive Blog - March 1, 2024