Measuring and Increasing Code Quality

At some point in a CTO’s career questions will be raised about “code quality” under their watch. Engineering teams will typically associate code quality with bugs and feature release velocity, while Product Management and Customer Success organizations will often define it in terms of platform stability and reliability or customer-derived metrics. 

How do you quantify code quality? What strategies are there for increasing code quality? How will you know if code quality is improving? – These are the questions myself and every other product or engineering leader ends up facing at some point as product development teams become larger and  products mature.

Key Metrics to Gather

From a CTO perspective, key go-to metrics typically include a subset or all of the following:

1. Cyclomatic Complexity – Measures the number of linearly independent paths through the code.
1. Benefits: Helps identify complex and potentially error-prone code that is difficult to maintain and test.
2. Change Over Time: Should decrease as code is refactored and simplified.
2. Code Churn – Measures the frequency of code changes.
1. Benefits: High churn can indicate unstable code or unclear requirements.
2. Change Over Time: Should stabilize as the codebase matures and becomes more stable.
3. Code Coverage - Percentage of code covered by automated tests.
1. Benefits: Higher coverage indicates better-tested code, reducing the likelihood of defects.
2. Change Over Time: Should increase as more tests are added, aiming for 80% or higher.
4. Technical Debt - Represents the cost of additional rework caused by choosing an easy solution now instead of a better approach that would take longer.
1. Benefits: Helps prioritize refactoring efforts and manage long-term code quality.
2. Change Over Time: Should decrease as technical debt is addressed and reduced.
5. Defect Density - Number of defects per unit of code (e.g., per 1,000 lines of code).
1. Benefits: Indicates the quality of the codebase and effectiveness of testing.
2. Change Over Time: Should decrease as code quality improves and defects are fixed.
6. Maintainability Index - A composite metric that includes cyclomatic complexity, lines of code, and Halstead volume.
1. Benefits: Provides an overall measure of how easy the code is to maintain.
2. Change Over Time: Should improve as code is refactored and simplified.

What does “success” look like?

• Reduced Cyclomatic Complexity - Functions and methods become simpler and easier to understand, leading to fewer bugs and easier maintenance.
• Stabilized Code Churn - Indicates that the codebase is stable, with fewer frequent changes, suggesting better initial design and clearer requirements.
• Increased Code Coverage - Higher test coverage means more of the code is tested, reducing the likelihood of defects and increasing confidence in the codebase.
• Decreased Technical Debt - Lower technical debt means the codebase is cleaner and more maintainable, reducing long-term costs and improving developer productivity.
• Lower Defect Density - Fewer defects per unit of code indicate higher code quality and more effective testing processes.
• Improved Maintainability Index - A higher maintainability index means the code is easier to understand, modify, and extend, leading to more efficient development processes.

By continuously monitoring these metrics and making data-driven decisions, you can systematically improve code quality, leading to more reliable, maintainable, and scalable software products.

From Metrics to Strategy

Modern software development pipelines and tooling are pretty efficient at generating these important metrics. I’ve found the following four strategies most efficient.

• Static Code Analysis
• Common Tools: SonarQube, Checkstyle, PMD, SpotBugs
• Key Metrics:
• Code Coverage: Measures the extent to which code is covered by automated tests.
• Technical Debt: Quantifies the effort required to fix issues and improve code quality.
• Security Vulnerabilities: Identifies potential security risks.
• Code Duplication: Detects redundant code sections.
• Complexity: Assesses the complexity of code, including cyclomatic complexity and nesting depth.
• Strategies for Improvement:
• Regular Static Analysis: Schedule regular static analysis runs to identify and address issues early.
• Code Review: Encourage developers to review code for potential issues and suggest improvements.
• Refactoring: Refactor code to improve readability, maintainability, and performance.
• Dynamic Code Analysis
• Common Tools: JUnit, TestNG, Selenium
• Key Metrics:
• Test Coverage: Measures the extent to which code is covered by tests.
• Test Failure Rate: Tracks the frequency of test failures.
• Test Execution Time: Monitors the time taken to run test suites.
• Strategies for Improvement:
• Write Comprehensive Tests: Develop thorough unit, integration, and end-to-end tests.
• Test Automation: Automate tests to increase efficiency and reduce manual effort.
• Test-Driven Development (TDD): Write tests before writing code to ensure quality and functionality.
• Code Reviews
• Common Tools: GitHub, GitLab, Bitbucket
• Key Metrics:
• Review Time: Measures the average time taken to review code changes.
• Review Comments: Tracks the number of comments and suggestions made during reviews.
• Defect Density: Calculates the number of defects found per line of code.
• Strategies for Improvement:
• Establish Clear Guidelines: Define clear guidelines for code reviews, including formatting, commenting, and testing standards.
• Encourage Timely Reviews: Promote timely reviews to avoid bottlenecks and delays.
• Provide Constructive Feedback: Provide constructive feedback to improve code quality and foster a positive review culture.
• Continuous Integration and Continuous Delivery (CI/CD)
• Common Tools: Jenkins, CircleCI, GitLab CI/CD
• Key Metrics:
• Build Success Rate: Measures the percentage of successful builds.
• Deployment Frequency: Tracks the frequency of deployments to production.
• Mean Time to Recovery (MTTR): Measures the time taken to recover from failures.
• Strategies for Improvement:
• Automate the Build Process: Automate the build, test, and deployment processes to reduce manual effort and errors.
• Implement Automated Testing: Integrate automated tests into the CI/CD pipeline to catch issues early.
• Monitor Deployment Metrics: Monitor key metrics to identify and address performance bottlenecks and failures.

Final Thoughts

In addition to measuring code quality, it's essential to implement strategies to continuously help improve the scores you're tracking.

• Foster a Culture of Quality - Encourage developers to prioritize code quality and take ownership of their work.
• Provide Training and Development Opportunities - Invest in training and development to improve developers' skills and knowledge.
• Use Code Quality Tools Effectively - Utilize code quality tools to identify and address issues proactively.
• Regularly Review and Refactor Code - Schedule regular code reviews and refactoring sessions to improve code quality and maintainability.
• Encourage Pair Programming - Pair programming can help improve code quality, knowledge sharing, and collaboration.
• Establish Coding Standards and Guidelines - Define clear coding standards and guidelines to ensure consistency and maintainability.

-- Gunter Ollmann

Navigating the Rapid Digital Shift: Ticket on the Bus, Not the Whole Bus

Global Companies’ Evaluation of Cybersecurity Solutions Selection Has Been Steadily Changing 

If it wasn’t already obvious to cybersecurity sales teams, there’s been a sea change for large organizations evaluating and buying new security products to protect their businesses. Responding to COVID-19, transformation plans that enable “work from home” such as Zero Trust identity and access management have been greatly accelerated, while technology refreshes and other capital-intensive plans are being pushed back.

Now, several months into this new operations paradigm, there may be added credence to the adage “in for a penny, in for a pound.” 

Many large companies have successfully navigated the digital shift to most of their workforce working remotely, finding the transition less difficult than first envisaged and achieving higher productivity than anticipated. Because such companies have resolved long-held internal conflicts over the security and integrity of cloud-based business operations, many of those postponed capital-intensive projects are being reviewed with a cloud-enabled, subscription-based lens.

This has several ramifications for cybersecurity vendors—particularly the specialized boutiques and innovative startups looking to quickly capitalize on new security opportunities.

Global companies’ evaluation of cybersecurity solutions selection has been steadily changing over the past couple years. The rapid digital shift of recent months has reinforced the need for change. 

I’d like to offer advice to vendors attempting to reach out and position their new cybersecurity products.

1. “I’ll buy a ticket, not the whole bus.” For decades, startups have looked to the largest companies as the Golden Goose and focused great energies in selling into them. The premise being that by solving a critical problem for them at a very high premium, that will cover the costs of developing an actual solution that can be sold broadly—e.g., the sale will fund my company’s product development. Although there may be a few cases where only a custom-tuned solution is required, many large businesses now prefer to buy a close-enough solution off the rack and work with the vendor as an advisor—not an investor. CISOs are looking at the sustainable list price of the solution and will purchase at a discounted level proportional to their deployment’s scale.
2. “Cost projection is critical.” Although highly versatile and scalable, cloud-based services billing can be difficult to predict—especially if the cybersecurity solution requires multiple third-party and cloud-provider SaaS dependencies. Security owners and budget holders are requiring vendors to provide accurate billing forecast and tiered discount models for the complete solution—models that include all dependent service costs (e.g., log storage analytics, container management). Vendors need to remove as much calculus from the pricing as possible and be prepared for billed services to be pared back if overly optimistic projections exceed the planned budget. Cost discussions have replaced those about cloud solutions prices.
3. “Features must be pre-integrated.” If the product is a feature (which, let’s face it, almost all new startup products are!), recognize it as a feature and don’t position it as a partial solution. As a feature product, integration with the solutions businesses already use is a prerequisite, and sales representatives should lead with the integration and interoperability first. CISOs are looking to shrink their attack surface and simplify the portfolio of products and vendors they rely on, and are increasingly reluctant to take on the task of brokering partnership between vendors as a prerequisite for extracting new protection value. Feature products benefit greatly by being enabled from within a solution provider’s product or marketplace.

On a related note, with the surge to execute day-to-day business operations remotely with a diverse and globally distributed workforce, cybersecurity buying decisions will increasingly factor accessibility, usability, and inclusiveness in solution design and operability. Vendors will be steered toward cloud-standardized accessibility interfaces—enabling visually impaired employees to use screen readers or dexterity-limited users to employ voice-to-text controls—to perform their analysis.

These changes are not unique to the largest enterprise businesses and are trickling down to other educated cybersecurity buyers feeling the same buying pain. Forewarned is forearmed.

-- Gunter Ollmann

First Published: SecurityWeek - June 9, 2020