Showing posts with label Code Reviews. Show all posts
Showing posts with label Code Reviews. Show all posts

Tuesday, November 26, 2024

Measuring and Increasing Code Quality

At some point in a CTO’s career questions will be raised about “code quality” under their watch. Engineering teams will typically associate code quality with bugs and feature release velocity, while Product Management and Customer Success organizations will often define it in terms of platform stability and reliability or customer-derived metrics. 

How do you quantify code quality? What strategies are there for increasing code quality? How will you know if code quality is improving? – These are the questions myself and every other product or engineering leader ends up facing at some point as product development teams become larger and  products mature.

Key Metrics to Gather

From a CTO perspective, key go-to metrics typically include a subset or all of the following:

  1. Cyclomatic Complexity – Measures the number of linearly independent paths through the code.
    1. Benefits: Helps identify complex and potentially error-prone code that is difficult to maintain and test.
    2. Change Over Time: Should decrease as code is refactored and simplified.
  2. Code Churn – Measures the frequency of code changes.
    1. Benefits: High churn can indicate unstable code or unclear requirements.
    2. Change Over Time: Should stabilize as the codebase matures and becomes more stable.
  3. Code Coverage - Percentage of code covered by automated tests.
    1. Benefits: Higher coverage indicates better-tested code, reducing the likelihood of defects.
    2. Change Over Time: Should increase as more tests are added, aiming for 80% or higher.
  4. Technical Debt - Represents the cost of additional rework caused by choosing an easy solution now instead of a better approach that would take longer.
    1. Benefits: Helps prioritize refactoring efforts and manage long-term code quality.
    2. Change Over Time: Should decrease as technical debt is addressed and reduced.
  5. Defect Density - Number of defects per unit of code (e.g., per 1,000 lines of code).
    1. Benefits: Indicates the quality of the codebase and effectiveness of testing.
    2. Change Over Time: Should decrease as code quality improves and defects are fixed.
  6. Maintainability Index - A composite metric that includes cyclomatic complexity, lines of code, and Halstead volume.
    1. Benefits: Provides an overall measure of how easy the code is to maintain.
    2. Change Over Time: Should improve as code is refactored and simplified.

What does “success” look like?

  • Reduced Cyclomatic Complexity - Functions and methods become simpler and easier to understand, leading to fewer bugs and easier maintenance.
  • Stabilized Code Churn - Indicates that the codebase is stable, with fewer frequent changes, suggesting better initial design and clearer requirements.
  • Increased Code Coverage - Higher test coverage means more of the code is tested, reducing the likelihood of defects and increasing confidence in the codebase.
  • Decreased Technical Debt - Lower technical debt means the codebase is cleaner and more maintainable, reducing long-term costs and improving developer productivity.
  • Lower Defect Density - Fewer defects per unit of code indicate higher code quality and more effective testing processes.
  • Improved Maintainability Index - A higher maintainability index means the code is easier to understand, modify, and extend, leading to more efficient development processes.

By continuously monitoring these metrics and making data-driven decisions, you can systematically improve code quality, leading to more reliable, maintainable, and scalable software products.

From Metrics to Strategy

Modern software development pipelines and tooling are pretty efficient at generating these important metrics. I’ve found the following four strategies most efficient.

  • Static Code Analysis
    • Common Tools: SonarQube, Checkstyle, PMD, SpotBugs
    • Key Metrics:
      • Code Coverage: Measures the extent to which code is covered by automated tests.
      • Technical Debt: Quantifies the effort required to fix issues and improve code quality.
      • Security Vulnerabilities: Identifies potential security risks.
      • Code Duplication: Detects redundant code sections.
      • Complexity: Assesses the complexity of code, including cyclomatic complexity and nesting depth.
    • Strategies for Improvement:
      • Regular Static Analysis: Schedule regular static analysis runs to identify and address issues early.
      • Code Review: Encourage developers to review code for potential issues and suggest improvements.
      • Refactoring: Refactor code to improve readability, maintainability, and performance.
  • Dynamic Code Analysis
    • Common Tools: JUnit, TestNG, Selenium
    • Key Metrics:
      • Test Coverage: Measures the extent to which code is covered by tests.
      • Test Failure Rate: Tracks the frequency of test failures.
      • Test Execution Time: Monitors the time taken to run test suites.
    • Strategies for Improvement:
      • Write Comprehensive Tests: Develop thorough unit, integration, and end-to-end tests.
      • Test Automation: Automate tests to increase efficiency and reduce manual effort.
      • Test-Driven Development (TDD): Write tests before writing code to ensure quality and functionality.
  • Code Reviews
    • Common Tools: GitHub, GitLab, Bitbucket
    • Key Metrics:
      • Review Time: Measures the average time taken to review code changes.
      • Review Comments: Tracks the number of comments and suggestions made during reviews.
      • Defect Density: Calculates the number of defects found per line of code.
    • Strategies for Improvement:
      • Establish Clear Guidelines: Define clear guidelines for code reviews, including formatting, commenting, and testing standards.
      • Encourage Timely Reviews: Promote timely reviews to avoid bottlenecks and delays.
      • Provide Constructive Feedback: Provide constructive feedback to improve code quality and foster a positive review culture.
  • Continuous Integration and Continuous Delivery (CI/CD)
    • Common Tools: Jenkins, CircleCI, GitLab CI/CD
    • Key Metrics:
      • Build Success Rate: Measures the percentage of successful builds.
      • Deployment Frequency: Tracks the frequency of deployments to production.
      • Mean Time to Recovery (MTTR): Measures the time taken to recover from failures.
    • Strategies for Improvement:
      • Automate the Build Process: Automate the build, test, and deployment processes to reduce manual effort and errors.
      • Implement Automated Testing: Integrate automated tests into the CI/CD pipeline to catch issues early.
      • Monitor Deployment Metrics: Monitor key metrics to identify and address performance bottlenecks and failures.

Final Thoughts

In addition to measuring code quality, it's essential to implement strategies to continuously help improve the scores you're tracking.

  • Foster a Culture of Quality - Encourage developers to prioritize code quality and take ownership of their work.
  • Provide Training and Development Opportunities - Invest in training and development to improve developers' skills and knowledge.
  • Use Code Quality Tools Effectively - Utilize code quality tools to identify and address issues proactively.
  • Regularly Review and Refactor Code - Schedule regular code reviews and refactoring sessions to improve code quality and maintainability.
  • Encourage Pair Programming - Pair programming can help improve code quality, knowledge sharing, and collaboration.
  • Establish Coding Standards and Guidelines - Define clear coding standards and guidelines to ensure consistency and maintainability.
-- Gunter Ollmann

Thursday, October 19, 2023

A SAFE Journey to Selling Devices to Cloud and Datacenter Providers

Observations from the OCP Global Summit | San Jose, CA | October, 18, 2023

If you missed it, there was a significant launch of the Open Compute Project (OCP) Foundation’s new community-led security program for improving device security underpins a fundamental change in the way device vendors and manufacturers engage and sell their products to the worlds leading cloud and datacenter providers.


Beyond standing up a framework for driving continuous security conformance assurance, the Security Appraisal Framework and Enablement (S.A.F.E.) program redefines(and optimizes) the relationship between the device vendor, the device adopter, and the security review provider.

Back when I served as chief security officer for Microsoft’s Cloud and AI Security group, I encountered a scaling problem with the way Azure worked with device vendors (think in terms of CPU’s, GPU’s, SSD’s, network cards, etc.) in validating the integrity and security of their devices. The short story is that before any device or it’s firmware could be deployed within a datacenter a security assessment had to be undertaken – and only devices (and their firmware) that came back with “clean” security reports could be purchased and ultimately deployed. Through a combination of Microsoft’s in-house and trusted third-party penetration testing and code review teams, each key device would be tested and an iterative improvement cycle would continue until the security assurance report came back clean.

There were two obvious problems with that mode of security assurance and interaction. As a purchaser and adopter of devices, how do you scale an in-house security practices to assess and assure the thousands of devices from hundreds of vendors and their ever-changing firmware? And, as a device vendor, how do you meet and coordinate the overlapping yet distinct security requirements of every purchaser and adopter of your technology?

Which brings me back to S.A.F.E. While it has been a couple years since I moved on from Microsoft, it’s fantastic to see that those first meetings with the OCP team and like-minded cloud service providers to tackle those problems that culminated this week in the official launch of the solution – and doubly proud of IOActive’s key involvements in both the creation of the S.A.F.E. device security checklist and Security Review Provider (SRP) criteria, and for being one of the founding SRP providers.

For device vendors, I think S.A.F.E. is great news.

A few points to justify that statement:

  1. Just one set of device security requirements (complete with device checklists) that address all OCP member purchasers and adopters. Better yet, community-driven and publicly available security requirements (with input from purchasers, vendors, and SRP providers alike) that will evolve and adapt to the changing world
  2. Accredited and trusted OCP SRP providers that device vendors can engage with at anytime in their device and firmware’s development lifecycle – cost-effectively accelerating and simplifying a vendors journey to “clean” S.A.F.E.-Approved product status; ultimately ensuring a vendor’s devices can be purchased and deployed by cloud and datacenter providers quicker than ever before
  3. Devices with advanced and proprietary technologies can be security tested and assured to meet adopter security requirements without the vendor having to share proprietary source code with the adopter.

One last perspective to share – oriented specifically to device vendors: there’s fantastic innovation going on across the cloud and datacenter device ecosystem. As I navigate the OCP conference Expo hall, passing by liquid cooling-systems, quantum computing data connectors, petabyte storage arrays, and even specialized forklifts for moving data racks- I’m reminded that many vendors are only just now starting their security journey, and the prospect of having to meet the detailed requirements of S.A.F.E. may feel daunting.

In all my time at Microsoft (and IBM for that matter), I never encountered a device vendor that “nailed security” the first time, nor the third time. Reaching S.A.F.E. approval will be a journey that requires a trusted security partner. When you’re ready for that journey, you’ll not find a stronger and more accomplished or experienced partner than the team at IOActive.

-- Gunter Ollmann

First Published: IOActive Blog - October 19 2023