From APES to Bespoke Security Automated as a Service

Many of the most innovative security start-ups I come across share a common heritage – their core product evolved from a need to automate the delivery of an advanced service that had begun as a boutique or specialized consulting offering. Start-ups with this legacy tend to have bypassed the “feature looking for a problem” phase that many others struggle with and often launch their products on day-one alongside a parade of satisfied marque accounts.

While there isn’t a universal formula for success, over my years delivering boutique professional security services, I have been very lucky to encounter that product evolution several times, usually resulting from consultants intelligently automating the repetitive parts of their jobs away and creating a new class of product.

For example, around the turn of the millennium, when penetration testing came to the fore as the cutting edge in security consulting, the need for automating away the drudgery of port scans and vulnerability scanning was obvious. The first foray led to tooling that freed up consultants to focus on the “art” of bug hunting and recognition that some customers needs were satisfied with those basic capabilities. During my time at Internet Security Systems, that first automation came to be known as the “monkey scan” – because of how easy it was to run. Of course, once the marketing team got wind of customers purchasing the scanning service, a more sensible name was needed and so Automated Perimeter and Enterprise Scanner (APES) was born. From humble beginnings, that X-Force managed service line business grew and, through acquisition, its legacy continues today as part of IBM’s Managed Security Services Provider (MSSP) business. 

Automation of repetitive consulting tasks is an obvious and critical element, but so too is the need to ensure consistency and exhaustive completion of delivery. Along my own journey I’ve seen former colleagues spawn companies such as SPI Dynamics and PortSwigger Web Security (i.e. Burp Proxy) to bring to market new web application security testing tools, Continuum Security SL to solve SDLC-based threat modelling and risk management challenges, Endgame to kickstart the nation-state threat intelligence market, and AttackIQ to construct and define the Attack and Breach Simulation category – all springing from the imagination of talented consultants looking to make life just a little bit easier.

To understand what the next innovative security technology will be, we should look closely at the premium service offerings from specialized services boutique consulting companies and pay attention to those services that have a documented and repeatable methodology. While many young specialist service lines may present themselves as more “art” than “science” – the turning point comes with the development and enforcement of a standardized methodology.

A service methodology ensures consistency of delivery. Consistency means that the differentiated elements of the service can be, or must be, repeatable. If they are repeatable, then they almost always can be automated. If key elements of the service are automatable, then they can be productized. 

The depth of service that automation can deliver roughly defines whether the product will be most effectively delivered as a managed security service, a self-service SaaS offering, or a stand-alone product.

In the past, that evolution from boutique consulting service to top-right corner market leading product has taken a few years – typically three years for productization and market awareness, then another three to five before analysts label and assign a market segment. I anticipate that more consulting services will mature into products and the overall pace will increase over coming years because public cloud and AI are rapidly accelerating the gestation of these products. 

Just as many of the most innovative companies launch as cloud-native, security consultants have similarly embraced and applied their expertise in cloud environments. Consultants were often constraint bound by their clients’ hardware and physical locations. Now, when consultants need to automate repetitive tasks (e.g. enumerating APIs, fuzzing payloads, etc.) or to test a hypothesis, they already have the tools in front of them – with no energy lost in applying them. This greatly shortens the time needed to prototype new cross-client solution sets and capabilities.

But automation will only go so far. A successful product needs to capture and distill the expertise and experience that a specialist consultant applies when interpreting the output of all those automated tasks. This is where advances in AI are accelerating the product creation process and transforming managed services businesses.

Off-the-shelf AI libraries and cloud services are allowing innovators to move from linear content creation modes (e.g., each threat requires a unique signature) and decades-old if-then-else logic to training classifier systems capable of identifying and labeling swathes of the problem space they are seeking to solve, and teaching systems to learn new responses directly from the actions the consultants  are already undertaking to solve their customers’ problems.

In my time as a CISO for organizations that often required security consulting expertise, I’ve engaged in reviewing the methodology that consultants will be applying to my systems. Lack of a detailed methodology will inevitably lead to inconsistent results and lack of repeatability, the death knell of compliance. When reviewing a proposed methodology, a CISO should also ask about the automation process framework and whether those automated tasks can be separated from consultant billing. This could possibly reduce overall job costs, but also prompts your consulting partners to accelerate an important services transition into a more versatile product.

For my former consulting brethren, take a critical look at the innovative services you are delivering. Stop playing the “art” card and instead focus on the detailed methodology that’ll promote repeatability and confidence in your service. From there, invest time in applying the resources of public cloud to bring automation, scalability, and AI to solving the given problem as a platform for all customers – past, present, and future.

-- Gunter Ollmann

First Published: SecurityWeek - May 21, 2019

Navigating the "Pentest" World

The demand for penetration testing and security assessment services worldwide has been growing year-on-year. Driven largely by Governance, Risk, and Compliance (GRC) concerns, plus an evolving pressure to be observed taking information security and customer privacy seriously, most CIO/CSO/CISO’s can expect to conduct regular “pentests” as a means of validating their organizations or product’s security.

An unfortunate circumstance of two decades of professional service oriented delivery of pentests is that the very term “penetration testing” now covers a broad range of security services and risk attributes – with most consulting firms provide a smorgasbord of differentiated service offerings – intermixing terms such as security assessment and pentest, and constructing hybrid testing methodologies.

For those newly tasked with having to find and retain a team capable of delivering a pentest, the prospect of having to decipher the lingo and identify the right service is often daunting – as failure to get it right is not only financially costly, but may also be career-ending if later proven to be inadequate.

What does today’s landscape of pentesting look like?

All penetration testing methodologies and delivery approaches are designed to factor-in and illustrate a threat represented by an attack vector or exploitation. A key differentiator between many testing methodologies lies in whether the scope is to identify the presence of a vulnerability, or to exploit and subsequently propagate an attack through that vulnerability. The former is generally bucketed in the assessment and audit taxonomy, while the latter is more commonly a definition for penetration testing (or an ethical hack).

The penetration testing market and categorization of services is divided by two primary factors – the level of detail that will be provided by the client, and the range of “hacker” tools and techniques that will be allowed as part of the testing. Depending upon the business drivers behind the pentest (e.g. compliance, risk reduction, or attack simulation), there is often a graduated-scale of services. Some of the most common terms used are:

• Vulnerability Scanning
  The use of automated tools to identify hosts, devices, infrastructure, services, applications, and code snippets that may be vulnerable to known attack vectors or have a history of security issues and vulnerabilities.
• Black-box Pentest
  The application of common attack tools and methodologies against a client-defined target or range of targets in which the pentester is tasked with identifying all the important security vulnerabilities and configuration failures of the scoped engagement. Typically, the penetration scope is limited to approved systems and windows of exploitation to minimize the potential for collateral damage. The client provides little information beyond the scope and expects the consultant to replicate the discovery and attack phases of an attacker who has zero insider knowledge of the environment. 
• Gray-box Pentest
  Identical methodology to the Black-box Pentest, but with some degree of insider knowledge transfer. When an important vulnerability is uncovered the consultant will typically liaise with the client to obtain additional “insider information” which can be used to either establish an appropriate risk classification for the vulnerability, or initiate a transfer of additional information about the host or the data it contains (that could likely be gained by successfully exploiting the vulnerability), without having to risk collateral damage or downtime during the testing phase.
• White-box Pentest (also referred to as Crystal-box Pentest)
  Identical tools and methodology to the Black-box Pentest, but the consultants are supplied with all networking documentation and details ahead of time. Often, as part of a White-box Pentest, the client will provide network diagrams and the results of vulnerability scanning tools and past pentest reports. The objective of this type of pentest is to maximize the consultants time on identifying new and previously undocumented security vulnerabilities and issues.
• Architecture Review
  Armed with an understanding of common attack tools and exploitation vectors, the consultant reviews the underlying architecture of the environment. Methodologies often include active testing phases, such as network mapping and service identification, but may include third-party hosting and delivery capabilities (e.g. domain name registration, DNS, etc.) and resilience to business disruption attacks (e.g. DDoS, Ransomware, etc.). A sizable component of the methodology is often tied to the evaluation and configuration of existing network detection and protection technologies (e.g. firewall rules, network segmentation, etc.) – with configuration files and information being provided directly by the client.
• Redteam Pentest
  Closely related to the Black-box pentest, the Redteam pentest mostly closely resembles a real attack. The scope of the engagement (targets and tools that can be used) is often greater than a Black-box pentest, and typically conducted in a manner to not alert the client’s security operations and incident response teams. The consultant will try to exploit any vulnerabilities they reasonably believe will provide access to client systems and, from a compromised device, attempt to move laterally within a compromised network – seeking to gain access to a specific (hidden) target, or deliver proof of control of the entire client network.
• Code Review
  The consultant is provided access to all source code material and will use a mix of automated and manual code analysis processes to identify security issues, vulnerabilities, and weaknesses. Some methodologies will encompass the creation of proof-of-concept (PoC) exploitation code to manually confirm the exploitability of an uncovered vulnerability.
• Controls Audit
  Typically delivered on-site, the consultant is provided access to all necessary systems, logs, policy-derived configuration files, reporting infrastructure, and data repositories, and performs an audit of existing security controls against a defined list of attack scenarios. Depending upon the scope of the engagement, this may include validation against multiple compliance standards and use a mix of automated, manual, and questionnaire-based evaluation techniques.

The Hybrid Pentest Landscape

In recent years the pentest landscape has evolved further with the addition of hybrid services and community-sourcing solutions. 

Overlapping the field of pentesting, there are three important additions:

• Bug Bounty Programs
  Public bug bounty programs seek to crowdsource penetration testing skills and directly incentivize participants to identify vulnerabilities in the client’s online services or consumer products. The approach typically encompasses an amalgamation of Vulnerability Scanning and Black-box Pentest methodologies – but with very specific scope and limitations on exploitation depth. With (ideally) many crowdsourced testers, the majority of testing is repeated by each participant. The hope is that, over time, all low-hanging fruit vulnerabilities will be uncovered and later remediated. 
• Purple Team Pentest
  This hybrid pentest combines Redteam and Blueteam (i.e. the client’s defense or incident response team) activities in to a single coordinated testing effort. The Redteam employs all the tools and tricks of a Redteam Pentest methodology, but each test is watch and responded to in real-time by the client’s Blueteam. As a collaborative pentest, there is regular communication between the teams (typically end of day calls) and synching of events. The objectives of Purple Team pentesting is both assess the capabilities of the Blueteam and to reduce the time typically taken to conduct a Redteam Pentest – by quickly validating the success or failure of various attack and exploitation techniques, and limiting the possibility of downtime failures of targeted and exploited systems.
• Disaster Recovery Testing
  By combining a Whitebox Pentest with incident response preparedness testing and a scenario-based attack strategy, Disaster Recovery Testing is a hybrid pentest designed to review, assess, and actively test the organization's capability to respond and recover from common hacker-initiated threats and disaster scenarios.

Given the broad category of “pentest” and the different testing methodologies followed by security consulting groups around the globe, prospective clients of these services should ensure that they have a clear understanding of what their primary business objectives are. Compliance, risk reduction, and attack simulation are the most common defining characteristics driving the need for penetration testing – and can typically align with the breakdown of the various pentest service definitions.

[Update: First graph adapted from Patrick Thomas' tweet - https://twitter.com/coffeetocode/status/794593057282859008]

Panel Selection of Penetration Testing Vendors

Most large companies have settled into a repeatable model in the way they undertake penetration testing and engage with their penetration testing suppliers. A popular model for companies that need to have several dozen pentests performed per year is to have a “board” or “panel” of three or four vetted companies and to rotate one provider in and out of the scheme per year – meaning that there is potentially a total refresh of providers every few years.

As vendor performance models go there is a certain undeniable logic to the process. However, it is worth questioning if these “board” models actually deliver better results – in particular, are the full spectrum of vulnerabilities being examined and are the individual consultants capable of delivering the work? In general, I’d argue that such a model often fails to meet these core requirements.

Advanced companies (e.g. brand-name software manufacturers) that require access to the most skilled talent-pool of penetration testers and reverse engineers tend to select vendors based upon the skills and experience of the consultants they employ – often specifically calling out individual consultants by names within the terms of the contract. They also pay premium rates to have access to that exclusive talent pool. In turn, the vendors that employ those consultants market and position their own companies as advanced service providers. For these companies, talent is the critical buying decision and it is not uncommon for the client organization to engage with new vendors when highly skilled or specialized consultants move between service providers.

Most large companies are not as sophisticated in discerning the talent pool needed to review and secure their products – yet still have many of the same demands and needs from a penetration testing skills perspective. For them, vendor selection is often about the responsiveness of the service provider (e.g. can they have 6 penetration testers onsite within two weeks in Germany or Boston) and the negotiated hourly-rate for their services. The churn of vendors through the “board” model is typically a compromise effort as they try to balance the needs of negotiating more favorable contractual terms, overcoming a perception of skill gaps within their providers consulting pool, and serve a mechanism for tapping a larger pool of vetted consultants.

From past observations, there are several flaws to this model (although several elements are not unique to the model).

1. Today's automated vulnerability scanners (for infrastructure, web application, and code review) are capable of detecting up to 90% of the vulnerabilities an “average” penetration tester can uncover manually if they use their own scripts and tools. Managed vulnerability scanning services (e.g. delivered by managed security service providers (MSSP)) typically reach the same 90% level, but tend to provide the additional value of removing false positives and confirming true positives. If these automated tools and services already cover 90% of the vulnerability spectrum, organizations need to determine whether closing the gap on the remaining 10% is worth the consulting effort and price. Most often, the answer is “yes, but…” where the “but…” piece is assigned a discrete window of time and effort to uncover or solve – and hence value. Organizations who adopt the “board” approach often fail to get the balance between tools, MSSP, and consultant-led vulnerability discovery programs. There are significant cost savings to be had when the right balances have been struck.
2. Very few consultants share the same depth of skills and experience. If an organization is seeking to uncover vulnerabilities that lie out of reach of automated discovery tools, it is absolutely critical that the consultant(s) undertaking the work have the necessary skills and experience. There is little point throwing a 15 year veteran of Windows OS security at an Android mobile application served from the AWS cloud – and vice versa. To this end, clients must evaluate the skill sets of the consultants that are being offered up by the vendor and who are expected to do the work. The reality of the situation is that clients that don’t pay the necessary attention can almost guarantee that they’ll get the second-rung consultants (pending availability) to perform this important work. The exception being when a new vendor is being evaluated, and they’ll often try to throw their best at the engagement for a period of time in order to show their corporate value – but clients should not anticipate the same level of results in subsequent engagements unless they are specific about the consultants they need on the job.
3. Rotating a vendor in or out of a program based upon an annual schedule independent of evaluating the consultants employed by the company makes little sense. Many penetration testing companies will have a high churn of technical staff to begin with and their overall technical delivery capabilities and depth of skills specialization will flux though-out the year. By understanding what skill sets the client organization needs and the amount of experience in each skill area in advance, those organizations can better rationalize their service providers consulting capabilities – and negotiate better terms.
4. Because consultant skills and experience play such an important role in being able to uncover new vulnerabilities, client organizations should consider cross-vendor teams when seeking to penetration test and secure higher-priority infrastructure, applications, and products. Cherry-picking named consultants from multiple vendors to work on an important security requirement tends to yield the best and most comprehensive findings. Often there is the added advantage of those vendors choosing to compete to ensure that their consultants do the best team work on the joint project – hoping that more follow-on business will fall in their direction.

While “board” or “panel” approaches to penetration testing vendor management may have an appeal from a convenience perspective, the key to getting the best results (both economical and vulnerability discovery) lies with the consultants themselves. 

Treating the vendor companies as convenient payment shells for the consultants you want or need working on your security assignments is OK as long as you evaluate the consultants they employ and are specific on which consultants you want working to secure your infrastructure, applications, and products. To do otherwise is a disservice to your organization.

-- Gunter

Pentest Evolution: Malware Under Control

When I look back at the history of commercial consultancy-based pentesting I see two distinct forks in the road. The first happened around 2000, and the second happened around 2003. But I think another fork is about to crop up.

Prior to 2000, commercial pentesting was almost exclusively focused on the external hacking of an organizations Internet visible assets. Basically, professional full-time consulting teams (which can probably be tracked back to 1994 if you push hard enough) were following a loose pentest methodology (still mostly portrayed as a dark art and only "learnable" via an authoritative mentor) - plugging away with vulnerability scanners and exploiting anything that came up - where the goal was break in, plant a few flags, and then tell the client what patches and system hardening they needed to catch up on. This core area of pentesting (which is still a distinct suit of offerings and consulting skills today) focuses upon OS and network-level vulnerability discovery and careful exploitation.

The first fork
By 2000 though, simply hacking an IIS or Apache server through an unpatched vulnerability or permissions flaw and throwing up a command script to "root" the server wasn't really cutting it to anymore for all these new Web applications. So, the first real "specialist" services started appear - focused upon assessing the custom Web application itself - independent of the hosting platform. To my mind, that was the first forking of the pentest track. Sure, there were still (and are) security code reviews (dissecting lines of code and hunting for bugs and vulnerabilities) - but I don't class that as "pentesting" as such, thats either auditing or security assessment.

That first fork led to entirely new pentesting methodologies, training regimes and certifications. But, more importantly, it also led to distinct consulting teams - rather than a specialized subset of network skills learned as part of being a pentester. Today, there's so much to learn in the field of Web Application pentesting that to keep at the top of the game you'll never realistically have time to deep-dive more classical OS and network based pentesting.

The second fork
The next fork that altered the fundamentals of pentesting occurred around 2003 with the advent (and requirement) for specialized reverse engineering skills to "black-box" hack a brand-new commercial software product. Around this time major software vendors were struggling in their battles against blackhat hackers and the full disclosure movement - even the news media was keeping count of the vulnerabilities - and customers were scared.

The solution came from specialist pentesting consulting organizations that had established a name (and reputation) based upon their ability to discover/disclose new vulnerabilities. It was a simple business model - find new bugs in all the software that prospective customers use, tell the media you found some bugs, get recognized by prospective customers as being "elite" pentesters, and turn the "prospective" in to "loyal" customers.

I identify 2003 as the year that specialized bug-hunting and security reverse engineering services started to appear as commercial consulting offerings, and the first real wide-spread traction as software vendors began to procure this specialized consulting.

The skill-sets are (again) quite unique of any other arm of pentesting. While knowledge of the other two pentesting regimes is valuable (e.g. Network/OS pentesting and Web Application pentesting), it takes a different mind and training to excel in the area of security reverse engineering. While you could argue that some of the best "classical" pentesters had many of the skills to find and exploit any new bugs that stumbled across during a client engagement - it wasn't until 2003 that these services really became commercial offerings and sales teams started to sell them.

The impending fork?
Which all leads me to point out a probable new folk in the pentesting path - specialist malware and its employment in pentesting. Why?

It seems to me that we've reached a time where formalized methodologies and compliance mandates have pretty much defined the practical bounds of commercial pentesting (Network/OS, Web application and Reverse Engineering), and yet there is a sizable security gap. And that gap firmly lies within the "prove it" camp of pentesting.

What I mean by that is, as any savvy pentester will tell their customer, the pentest is only as good as the consultant and the tools they used, and is only valid for the configuration tested and the date/time of testing. No guarantees or warranties are inferred, and it's a point in time test. And, on top of all that the scope of the pentest has typically been narrowly defined - which means that you end up with phrases like "system was out of scope...", "...not all patches were applied", "...not allowed to install tools on the compromised host", etc., appearing in the final reports handed to the customer.

But, with the greater adoption/deployment (and availability) of technologies such as IPS, firewalls, ADS, Web filtering, mail gateways, host-based protection, DLP, NAC, etc. and the growing strictness (and relevance) of compliance regulations, those classic limitations of pentesting methodologies leave vacant the "prove it" - prove that those technologies are really working and that the formal emergency response systems really do work.

This is where I think a new skill set, mindset and pentesting methodology is developing - and is an area which I expect to see develop in to commercial offerings this year.

Pentesting with malware
What I envision is the requirement for specialised security pentesting offerings that focus upon developing new "malware" and "delivery systems" designed to not only test the perimeter defenses of an enterprise, but also every layer of their security system in one go.

I don't think it's enough to say "drive-by-downloads are a fact of life and all it takes is one unpatched host to browse a dangerous site to infect our network. but that's OK because we have anomaly detection systems and DLP, and we'll stop them that way". Prove it!

Given the widespread availability of DIY malware creation kits, and the staggering array of tools that can pack, crypt, armour, obfuscate and bind a custom malware sample - and make it completely invisible to any anti-virus technology deployed within an enterprise - I expect that there will be a demand for pentesting to evolve and encompass the use of "live" malware as a core pentest consultancy offering.

For example, does the customers enterprise prevent users from browsing key-munged web sites (e.g. www.gooogle.com, intranet.enterpriise.com, etc.)? Which browser plugings are installed and not fully patched? Can malicious URL's and zipped malware make it through the mail gateways? Can the host-based security package detect keyloggers and network sniffers? If a malware package starts to scan and enumerate the local network from an "infected" host, is it detected, and how fast? What types of data can be exported from an infected host? Does compression and encryption of exported data get detected by the DLP solution? Does the malware have to be "proxy-aware" and require user authentication? Is out-of-hours activity detected from an "infected" host? Is it possible to "worm" through the enterprise network and "infect" or enumerate shared file systems and servers?

All of these questions, and many more, can be answered through the deployment of specialized malware creations and focused delivery techniques. The problem though is that this is an untapped fork in the pentesting road, requiring new mindsets - particularly with enterpise security teams.

The bad guys are already exploiting enterprises with custom malware, yet its generally taboo for consultancies to test security using similar methods. To my mind, that means that new pentesting specialization is now required to deliver the expertiese needed by enterprise business to really test their security from today's threat spectrum.

Malware pentest anyone?

Pentesters and Beer

Over the years I've come to the inevitable conclusion that pentesters and beer are inseparable. It's as fundamental a pairing as salt & pepper, Internet & porn, Kebabs & chilli-sauce...

Most days of pentesting culminate in an evening down the pub (typically with the customers onsite technical authority), and yet the following day all concerned are as sprightly as they were on the first day (yes - there are reasons for that!).

On the other hand, gathering an onslaught of pentesters together is always a cause for concern. Taking the onslaught to a far-away land for a company kickoff tends to result in high medical expenses and some interesting legal fees - but, as the saying goes, you're not a real pentester if you get caught.

Nowadays pentesters go onsite armed with quad-core laptops, MP3 players and bottles of maximum strength paracetamol. But there's always been something missing - until now!

Pentesters of the world, I give you the preformatted Formal Apology...


Note: There have been many discussions about the naming convention for a group of pentesters. "Hustle", "swarm", "gaggle" and "pilgrimage" have all been proposed at some stage - I prefer "onslaught".