Measuring and Increasing Code Quality

At some point in a CTO’s career questions will be raised about “code quality” under their watch. Engineering teams will typically associate code quality with bugs and feature release velocity, while Product Management and Customer Success organizations will often define it in terms of platform stability and reliability or customer-derived metrics. 

How do you quantify code quality? What strategies are there for increasing code quality? How will you know if code quality is improving? – These are the questions myself and every other product or engineering leader ends up facing at some point as product development teams become larger and  products mature.

Key Metrics to Gather

From a CTO perspective, key go-to metrics typically include a subset or all of the following:

1. Cyclomatic Complexity – Measures the number of linearly independent paths through the code.
1. Benefits: Helps identify complex and potentially error-prone code that is difficult to maintain and test.
2. Change Over Time: Should decrease as code is refactored and simplified.
2. Code Churn – Measures the frequency of code changes.
1. Benefits: High churn can indicate unstable code or unclear requirements.
2. Change Over Time: Should stabilize as the codebase matures and becomes more stable.
3. Code Coverage - Percentage of code covered by automated tests.
1. Benefits: Higher coverage indicates better-tested code, reducing the likelihood of defects.
2. Change Over Time: Should increase as more tests are added, aiming for 80% or higher.
4. Technical Debt - Represents the cost of additional rework caused by choosing an easy solution now instead of a better approach that would take longer.
1. Benefits: Helps prioritize refactoring efforts and manage long-term code quality.
2. Change Over Time: Should decrease as technical debt is addressed and reduced.
5. Defect Density - Number of defects per unit of code (e.g., per 1,000 lines of code).
1. Benefits: Indicates the quality of the codebase and effectiveness of testing.
2. Change Over Time: Should decrease as code quality improves and defects are fixed.
6. Maintainability Index - A composite metric that includes cyclomatic complexity, lines of code, and Halstead volume.
1. Benefits: Provides an overall measure of how easy the code is to maintain.
2. Change Over Time: Should improve as code is refactored and simplified.

What does “success” look like?

• Reduced Cyclomatic Complexity - Functions and methods become simpler and easier to understand, leading to fewer bugs and easier maintenance.
• Stabilized Code Churn - Indicates that the codebase is stable, with fewer frequent changes, suggesting better initial design and clearer requirements.
• Increased Code Coverage - Higher test coverage means more of the code is tested, reducing the likelihood of defects and increasing confidence in the codebase.
• Decreased Technical Debt - Lower technical debt means the codebase is cleaner and more maintainable, reducing long-term costs and improving developer productivity.
• Lower Defect Density - Fewer defects per unit of code indicate higher code quality and more effective testing processes.
• Improved Maintainability Index - A higher maintainability index means the code is easier to understand, modify, and extend, leading to more efficient development processes.

By continuously monitoring these metrics and making data-driven decisions, you can systematically improve code quality, leading to more reliable, maintainable, and scalable software products.

From Metrics to Strategy

Modern software development pipelines and tooling are pretty efficient at generating these important metrics. I’ve found the following four strategies most efficient.

• Static Code Analysis
• Common Tools: SonarQube, Checkstyle, PMD, SpotBugs
• Key Metrics:
• Code Coverage: Measures the extent to which code is covered by automated tests.
• Technical Debt: Quantifies the effort required to fix issues and improve code quality.
• Security Vulnerabilities: Identifies potential security risks.
• Code Duplication: Detects redundant code sections.
• Complexity: Assesses the complexity of code, including cyclomatic complexity and nesting depth.
• Strategies for Improvement:
• Regular Static Analysis: Schedule regular static analysis runs to identify and address issues early.
• Code Review: Encourage developers to review code for potential issues and suggest improvements.
• Refactoring: Refactor code to improve readability, maintainability, and performance.
• Dynamic Code Analysis
• Common Tools: JUnit, TestNG, Selenium
• Key Metrics:
• Test Coverage: Measures the extent to which code is covered by tests.
• Test Failure Rate: Tracks the frequency of test failures.
• Test Execution Time: Monitors the time taken to run test suites.
• Strategies for Improvement:
• Write Comprehensive Tests: Develop thorough unit, integration, and end-to-end tests.
• Test Automation: Automate tests to increase efficiency and reduce manual effort.
• Test-Driven Development (TDD): Write tests before writing code to ensure quality and functionality.
• Code Reviews
• Common Tools: GitHub, GitLab, Bitbucket
• Key Metrics:
• Review Time: Measures the average time taken to review code changes.
• Review Comments: Tracks the number of comments and suggestions made during reviews.
• Defect Density: Calculates the number of defects found per line of code.
• Strategies for Improvement:
• Establish Clear Guidelines: Define clear guidelines for code reviews, including formatting, commenting, and testing standards.
• Encourage Timely Reviews: Promote timely reviews to avoid bottlenecks and delays.
• Provide Constructive Feedback: Provide constructive feedback to improve code quality and foster a positive review culture.
• Continuous Integration and Continuous Delivery (CI/CD)
• Common Tools: Jenkins, CircleCI, GitLab CI/CD
• Key Metrics:
• Build Success Rate: Measures the percentage of successful builds.
• Deployment Frequency: Tracks the frequency of deployments to production.
• Mean Time to Recovery (MTTR): Measures the time taken to recover from failures.
• Strategies for Improvement:
• Automate the Build Process: Automate the build, test, and deployment processes to reduce manual effort and errors.
• Implement Automated Testing: Integrate automated tests into the CI/CD pipeline to catch issues early.
• Monitor Deployment Metrics: Monitor key metrics to identify and address performance bottlenecks and failures.

Final Thoughts

In addition to measuring code quality, it's essential to implement strategies to continuously help improve the scores you're tracking.

• Foster a Culture of Quality - Encourage developers to prioritize code quality and take ownership of their work.
• Provide Training and Development Opportunities - Invest in training and development to improve developers' skills and knowledge.
• Use Code Quality Tools Effectively - Utilize code quality tools to identify and address issues proactively.
• Regularly Review and Refactor Code - Schedule regular code reviews and refactoring sessions to improve code quality and maintainability.
• Encourage Pair Programming - Pair programming can help improve code quality, knowledge sharing, and collaboration.
• Establish Coding Standards and Guidelines - Define clear coding standards and guidelines to ensure consistency and maintainability.

-- Gunter Ollmann

Advancing DevSecOps Into the Future

If DevOps represents the union of people, process, and technology to continually provide value to customers, then DevSecOps represents the fusion of value and security provided to those same customers. The philosophy of integrating security practices within DevOps is obviously sensible (and necessary), but by attaching a different label perhaps we are likely admitting that, despite best efforts, this “fusion” is more of an emulsification.

DevSecOps incorporates discrete security elements and capabilities throughout the development process; “security as code” is the hymn recited by development and security operations teams alike. But when you look closer, the security elements of DevSecOps are discrete, like the tiny immiscible spheres of oil suspended within a tasty vinaigrette — incorporated rather than invisibly entwined within the fabric of DevOps.

Today’s DevSecOps can largely be divided into two core functions: the automated checking and gated prevention of known and potential security flaws throughout the continual integration and continual deployment (CI/CD) workflow, and the operational monitoring and response to security-imbued telemetry generated by the deployment and surrounding protection technologies.

Rightly, we cocoon the applications that flow from our CI/CD workflows with further layers of discrete security tooling to monitor, alert, and ideally protect against broad categories of threats — threats that may be more economically and reliably prevented from outside than within the workflows. Those layers of security almost always operate independently from the application they are defending. This needs to change if we’re to “level up” security and roll DevSecOps back into DevOps.

Although security operations (SecOps) teams are becoming vastly more efficient at managing and responding to the alerts generated by their perimeter, server, and behavioral defense systems, there is a need to incorporate this same telemetry, response workflows, and decision-making into both the CI/CD workflow and the application itself if businesses are to successfully battle advancing threats such as Adversarial AI, data lake tainting, and behavioral poisoning. 

Too many DevSecOps workflows depend upon humans being in them. They’re the “bump in the wire,” and when adversaries switch to newer automated or AI-enabled attack and exploitation modes, system compromise and data breaches will (repeatedly) occur before fixes can be created, defenses tweaked, and patches applied.

The future lies in moving beyond the independent operations of “secure the code” and “protect the app,” and into the realm of self-defending applications.

It sounds grandiose, but there are some core elements and opportunities to progress toward applications that can defend themselves.

• Telemetry from the security technologies that cocoon the application need to be available and consumable to the application and the CI/CD workflow.
• Applications must know when external security tools and monitors suspect or alert when attacked and be capable of responding if advantageous to do so. For example, an application may be capable of natively securely parsing a fund transfer request, but by knowing that a WAF had identified and blocked the previous 12 HTTP POST submissions due to malicious SQL injection payloads for the same session in the past 500 milliseconds, it could leverage the information in handling this 13th transfer and user session — perhaps by deceiving the attacker with a fake and evidentiary traceable response.
• Security technologies need to standardize on nomenclatures, severity, and impact for both threats and behaviors. The new generation of cloud-based SIEM, through normalization of data connectors and telemetry, is capable of providing a degree of (vendor-specific) standardization and is primed for being the source of real-time security telemetry for CI/CD and application consumption. Application development frameworks need to understand this nomenclature and, ideally, come pre-armed with libraries and functions to respond with best practices.
• Increased AI adoption and fusion within the CI/CD workflow can accelerate the pace at which workflows can respond to security telemetry. For example, a server-based security agent identifies a memory overflow and subsequent unwanted process startup, while the SIEM is able to reconstruct the session sequence to highlight the transaction string (0-day exploit). An intelligent and automated CI/CD process should be able to use that information to identify the vulnerable code and correct the logic flaw or bug, and proceed with an update to the live application with a fix — without developer involvement.

Security responsibility must, and will continue to, “shift left.” To enable that, security telemetry needs to be both accessible and incorporated into the application and the DevOps workflow, and the developers themselves must be comfortable and knowledgeable in integrating the information. Better developer tooling — such as secure coding languages and frameworks, accessible best-practice libraries and functions, and smart in-line developer guidance and correctors — will help close the gap.

Rapid advancement of AI and ML technologies and incorporation into the CI/CD workstream will be able to increase the pace of security integration and secure deployment. There is still much work to be done, and subsequently there are great opportunities for innovative companies to add significant value to the process. 

In the meantime, CISOs and DevOps leaders should press hard on technologies and processes that remove the human speed bumps from the CI/CD workflow. Adversaries are advancing at a fast pace in their development of fully automated and autonomous attack engines. Soon, defense and response will be measured in milliseconds, not in days and weeks as it is now.

-- Gunter Ollmann

First Published: SecurityWeek - March 3, 2020

Cloud is Creating Security and Network Convergence

Network Security Expertise is Needed More Than Ever Inside Security Operations Centers and on DevOps Teams

Digital transformation forces many changes to a business as it migrates to the public cloud. One of the most poorly examined is the convergence of network and security administration tasks and responsibilities in the public cloud.

On premises, the division between roles is pretty clear. The physical nature of networking infrastructure – the switches, routers, firewall appliances, network taps, WiFi hubs, and miles upon miles of cable – makes it easy to separate responsibilities. If it has power, stuff connects to it, it routes packets, and weighs more than 5 pounds, it probably belongs to the networking team.

In the cloud, where network connectivity features are defined by policies and code, the network is ephemeral. More importantly, the network is a security boundary – protecting services, applications, and data.

For many organizations, an early steppingstone in their digital transformation is virtualizing all their on-premises applications, infrastructure, and administrative and monitoring processes. Operating almost entirely within an Infrastructure-as-a-Service (IaaS) mode, previously favored network vendors provide virtual machine (VM) versions of their on-premises networking and security appliances – effectively making the transition to public cloud the equivalent of shifting to a new co-hosting datacenter.

This early stage takes very little advantage of public cloud. VMs remain implanted in statically defined networking architectures and old-style network monitoring remains largely the same. However, as organizations embrace continuous integration and continuous delivery (CI/CD), DevOps, serverless functions, and other cloud-native services, the roles of network and security administrator converge rapidly. At that point, network topology ceases to be the grid that servers and applications must snap to. Instead, leveraging the software defined network (SDN) nature of the cloud, the network becomes ephemeral – continuously defined, created, and disposed of in code.

With zero trust running core to modern CI/CD and DevOps security practices in the cloud, SDN has become a critical framework for protecting data, identities, and access controls.

Today, a cloud security architect, security analyst, or compliance officer cannot fulfill their security responsibilities without being a cloud network expert too. And, vice versa, a systems architect or network engineer cannot bring value to cloud operations without being comfortable wearing size 15 cloud security shoes.

For networking professionals transitioning to the cloud, I offer the following advice:

• Partner extensively with your peers on the security team – they too are a transformation and are destined to become network experts.
• Plan to transition from VM infested IaaS environments as fast as possible to cloud-native services which are easier to understand, manage, and deploy.
• Become familiar with the portal management experience of each new network (security) service, but plan on day-to-day management being at the command line.
• Brush up your scripting language expertise and get comfortable with code management tools. In a CI/CD workplace GitHub and its ilk are where the real action happens.
• Throw out the old inhibitions of consuming valuable network bandwidth with event logs and streaming service health telemetry. In the age of cloud SIEM, data is king and storage is cheap, and trouble-shooting ephemeral network problems requires both in abundance.
• Forget thumbing through network security books to learn. Training is all online. Watch the cloud provider’s workshop videos and test the lessons in real-time online.

With so many cloud critical controls existing at the network layer, network security expertise is needed more than ever inside security operations centers and on DevOps teams.

The faster in-house network administrators can transition to becoming public cloud network security engineers, architects, or analysts, the faster their organizations can implement digital transformation.

-- Gunter Ollmann

First Published: SecurityWeek - October 8, 2019