How Commercial Bug Hunting Changed the Boutique Security Consultancy Landscape

It’s been almost a decade since the first commercial “for-profit” bug bounty companies launched leveraging crowdsourced intelligence to uncover security vulnerabilities and simultaneously creating uncertainty for boutique security companies around the globe.

Not only could crowdsourced bug hunting drive down their consulting rates or result in their best bug hunters turning solo, it raised ethics questions, such as should a consultant previously engaged on a customer security assessment also pursue out-of-hours bug hunting against that same customer. What if she held back findings from the day-job to claim bounties at night?

With years of bug bounty programs now behind us, it is interesting to see how the information security sector transformed – or didn’t.

The fears of the boutique security consultancies – particularly those offering penetration testing and reverse engineering expertise – were proven unfounded. A handful of consultants did slip away and adopt full-time bug bounty pursuit lifestyles, but most didn’t. Nor did those companies feel a pinch on their hourly consulting rates. Instead, a few other things happened.

First, the boutiques upped the ante by repositioning their attack-based services – defining aggressive “red team” methodologies and doubling down on the value of combining black-box with white-box testing (or reverse engineering combined with code reviews) to uncover product and application bugs in a more efficient manner. Customers were (and are) encouraged to use bug bounties as a “first-pass filter” for finding common vulnerabilities – and then turn to dedicated experts to uncover (and help remediate) the truly nasty bugs.

Second, they began using bug bounty leaderboard tables as a recruitment vehicle for junior consultants. It was a subtle, but meaningful change. Previously, a lot of recruitment had been based off evaluating in-bound resumes by how many public disclosures or CVEs a security researcher or would-be consultant had made in the past. By leveraging the public leaderboards, suddenly there was a target list of candidates to go after. An interesting and obvious ramification was (and continues to be) that newly rising stars on public bug bounty leaderboards often disappear as they get hired as full-time consultants.

Third, bug bounty companies struggled with their business model. Taking a slice of the vendors payments to crowdsourced bug hunters sounded easier and less resource intensive than it turned out. The process of triaging the thousands of bug submissions – removing duplicates, validating proof-of-concept code, classifying criticality, and resolving disparities in hunter expectations – is tough work. It’s also something that tends to require a high degree of security research experience and costly expertise that doesn’t scale as rapidly as a crowdsource community can. The net result is that many of the bug bounty crowdsource vendors were forced to outsource sizable chunks of the triage work to boutique consultancies – as many in-house bug bounty programs also do.

A fourth (but not final) effect was that some consulting teams found contributing to public bug bounty programs an ideal way of cashing in on consulting “bench time” when a consultant is not directly engaged on a commercial project. Contributing to bug bounties has proven a nice supplement to what was previously lost productivity.

Over the last few years I’ve seen some pentesting companies also turn third-party bug bounty research and contribution into in-house training regimes, marketing campaigns, and an engagement model to secure new customers, e.g., find and submit bugs through the bug bounty program and then reach out directly to the customer with a bag full of more critical bugs.

Given the commercial pressures of on third-party bug bounty companies, it was not unexpected that they would seek to stretch their business model towards higher premium offerings, such as options for customers to engage with their best and most trusted bug hunters before opening up to the public or offering more traditional report-based “assessments” of the company’s product or website. More recently, some bug bounty vendors have expanded offerings to encompass community managed penetration testing and red team services.

The lines continue to blur between the boutique security consultancies and crowdsourcing bug bounty providers. It’ll be interesting to see what the landscape looks like in another decade. While there is a lot to be said and gained from crowdsourced security services, I must admit that the commercial realities of operating businesses that profit from managing or middle-manning their output strikes me as a difficult proposition in the long run.

I think the crowdsourcing of security research will continue to hold value for the businesses owning the product or web application, and I encourage businesses to take advantage of the public resource. But I would balance that with the reliability from engaging a dedicated consultancy for the tougher stuff.

-- Gunter Ollmann

First Published: SecurityWeek - December 11, 2019

Panel Selection of Penetration Testing Vendors

Most large companies have settled into a repeatable model in the way they undertake penetration testing and engage with their penetration testing suppliers. A popular model for companies that need to have several dozen pentests performed per year is to have a “board” or “panel” of three or four vetted companies and to rotate one provider in and out of the scheme per year – meaning that there is potentially a total refresh of providers every few years.

As vendor performance models go there is a certain undeniable logic to the process. However, it is worth questioning if these “board” models actually deliver better results – in particular, are the full spectrum of vulnerabilities being examined and are the individual consultants capable of delivering the work? In general, I’d argue that such a model often fails to meet these core requirements.

Advanced companies (e.g. brand-name software manufacturers) that require access to the most skilled talent-pool of penetration testers and reverse engineers tend to select vendors based upon the skills and experience of the consultants they employ – often specifically calling out individual consultants by names within the terms of the contract. They also pay premium rates to have access to that exclusive talent pool. In turn, the vendors that employ those consultants market and position their own companies as advanced service providers. For these companies, talent is the critical buying decision and it is not uncommon for the client organization to engage with new vendors when highly skilled or specialized consultants move between service providers.

Most large companies are not as sophisticated in discerning the talent pool needed to review and secure their products – yet still have many of the same demands and needs from a penetration testing skills perspective. For them, vendor selection is often about the responsiveness of the service provider (e.g. can they have 6 penetration testers onsite within two weeks in Germany or Boston) and the negotiated hourly-rate for their services. The churn of vendors through the “board” model is typically a compromise effort as they try to balance the needs of negotiating more favorable contractual terms, overcoming a perception of skill gaps within their providers consulting pool, and serve a mechanism for tapping a larger pool of vetted consultants.

From past observations, there are several flaws to this model (although several elements are not unique to the model).

1. Today's automated vulnerability scanners (for infrastructure, web application, and code review) are capable of detecting up to 90% of the vulnerabilities an “average” penetration tester can uncover manually if they use their own scripts and tools. Managed vulnerability scanning services (e.g. delivered by managed security service providers (MSSP)) typically reach the same 90% level, but tend to provide the additional value of removing false positives and confirming true positives. If these automated tools and services already cover 90% of the vulnerability spectrum, organizations need to determine whether closing the gap on the remaining 10% is worth the consulting effort and price. Most often, the answer is “yes, but…” where the “but…” piece is assigned a discrete window of time and effort to uncover or solve – and hence value. Organizations who adopt the “board” approach often fail to get the balance between tools, MSSP, and consultant-led vulnerability discovery programs. There are significant cost savings to be had when the right balances have been struck.
2. Very few consultants share the same depth of skills and experience. If an organization is seeking to uncover vulnerabilities that lie out of reach of automated discovery tools, it is absolutely critical that the consultant(s) undertaking the work have the necessary skills and experience. There is little point throwing a 15 year veteran of Windows OS security at an Android mobile application served from the AWS cloud – and vice versa. To this end, clients must evaluate the skill sets of the consultants that are being offered up by the vendor and who are expected to do the work. The reality of the situation is that clients that don’t pay the necessary attention can almost guarantee that they’ll get the second-rung consultants (pending availability) to perform this important work. The exception being when a new vendor is being evaluated, and they’ll often try to throw their best at the engagement for a period of time in order to show their corporate value – but clients should not anticipate the same level of results in subsequent engagements unless they are specific about the consultants they need on the job.
3. Rotating a vendor in or out of a program based upon an annual schedule independent of evaluating the consultants employed by the company makes little sense. Many penetration testing companies will have a high churn of technical staff to begin with and their overall technical delivery capabilities and depth of skills specialization will flux though-out the year. By understanding what skill sets the client organization needs and the amount of experience in each skill area in advance, those organizations can better rationalize their service providers consulting capabilities – and negotiate better terms.
4. Because consultant skills and experience play such an important role in being able to uncover new vulnerabilities, client organizations should consider cross-vendor teams when seeking to penetration test and secure higher-priority infrastructure, applications, and products. Cherry-picking named consultants from multiple vendors to work on an important security requirement tends to yield the best and most comprehensive findings. Often there is the added advantage of those vendors choosing to compete to ensure that their consultants do the best team work on the joint project – hoping that more follow-on business will fall in their direction.

While “board” or “panel” approaches to penetration testing vendor management may have an appeal from a convenience perspective, the key to getting the best results (both economical and vulnerability discovery) lies with the consultants themselves. 

Treating the vendor companies as convenient payment shells for the consultants you want or need working on your security assignments is OK as long as you evaluate the consultants they employ and are specific on which consultants you want working to secure your infrastructure, applications, and products. To do otherwise is a disservice to your organization.

-- Gunter

Breaking out of the consulting wave

There are certain thresholds in the life of a company that must be crossed and, in so doing, fundamentally alter the business. In the world of boutique security consulting companies, one such period of change (and resultant growth) is when the task of managing client relationships and securing the next project or client shifts from being part of a senior consultant’s role and transitions in to the waiting hands of a dedicated sales organization.

Over the years I’ve observed first-hand just how difficult this transition can be for both the senior consultants and the executive management.

A critical driver for this transition is the way consultants are forced to divide their time and attention. When the consultant isn’t on a paid engagement they spend time responding to clients and prospects – writing proposals, responding to RFI’s, and scoping engagements etc. When they’re working on a client project, it’s heads-down on delivery – meaning that there’s far less time to engage with other customers or prospects, and limited attention can be applied to lining up the next consulting job. Visually, the cyclical nature of this business mode resembles a graph of out-of-phase waveforms transposed upon one-another.

If the red line represents the effort the consultant applies to “project delivery” over time, and the blue line in turn represents “business development”, it should be clear that low periods of non-delivery are countered with high periods of hunting for new work, and vice versa.

The problem with this cyclical work pattern is that a company typically only makes money if the consulting is delivering on paid engagements – and ideally you’d want the red-line to be horizontal and as close to 100% delivery utilization as possible.

If that wasn’t already an obvious problem, its effect on the business is then multiplied – as the task of securing business and constructing new proposals typically falls upon the most senior consultants. This in turn means that the most expensive people in the consulting organization, who typically command the highest rates from clients, are the most absorbed in this perpetual sales-delivery cycle.

I’ve heard time and again that “it’s just the way it is” and arguments such as:

• As a technical consultancy, the client demands that they deal directly with the technical manager doing the delivery.
• Scoping a job and preparing a technical proposal requires an expert consultant.
• The onsite consultants know the customer the best. They’re always doing jobs for the client.
• Our consultants are managing consultants, and that’s what they do.

The list of “why things can’t change” could go on ad infinitum, but the reality is that a consulting company cannot grow and scale beyond its senior consultants until it breaks out of the cyclical pattern – which is why this particular threshold is both so important and difficult for a company to transition.

Some things I’ve learnt over the years in navigating this business transition (and hopefully serve as some useful advice to other businesses seeking to cross the threshold) include:

• The best security consultants, no matter how much they think of their skills at procuring and securing new business, are at best average farmers of an account (compared to a dedicated sales person). Yes, they typically understand the clients they do regular work for and are proficient at recognizing other opportunities within that client organization – however that pursuit and business development is limited to the client personnel they actually interact with during an engagement. The net result is that the client’s technical on-site folks love and adore the consultant and company, but most engagements are limited to a silo within the overall organization. For this reason the consulting company needs “hunters” – folks with the business development experience to identify other new people and opportunities in other parts of the same business.
• Dropping in a “sales guy” in to the organization and letting them figure it out because they have a track record selling things is unlikely to succeed. Security consulting (in particular) is a very technical sell, and those tasked with hunting and closing in on new clients and projects need to not only also be technical, but need to be backed by deeper technical expertise. Consider the physical differences between an Olympic high-jumper and an Olympic shot-putter. Both sports require unique attributes, and are unlikely to triumph in the others field of expertise. While an Olympic decathlon medalist may be able to do both, they’re also unlikely to win against someone who specialized in just one of those sports.
• Consulting managers are not sales people, they’re delivery coordinators and quality evangelists. Their role is often inglorious – as in-between chasing consultants for expenses and report deliverables, they spend much of their time apologizing to the client for things that didn’t go quite to plan and making the client happy again. Yes, they’re often the front line with existing customers and are core to delivering proposals to new clients, but their business development focus is (and should remain) blinkered to delivery.
• In many cases the role of a consulting manager can morph in to that of a sales engineer (just never call them that!). When a consulting manager has no direct reports, they can serve effectively as the technical backup to the sales team – scoping engagements, constructing technical proposals, and being the technical evangelist is new client and prospect meetings. This “sales engineer” (SE) role is often a critical component to building and supporting a successful consulting sales team. The stronger these technical experts are, and the more years under the belt consulting they have, the more respect they tend to garner from prospective clients, and the easier it is to close deals. In many ways they add the technical credibility to the sale organization for technical clients.
• Plan on building out a central team of technical authors. The technical author team provides the grease for easing a company through the transition period. By (slowly) removing some of the tedious consulting work – i.e. proposal generation, report proofing, and quality assurance on deliverables – the technical author team ensures a consistent quality of client-facing materials and eases the burden on the consulting and sales teams, and further frees up the time of valuable consultants. For global consulting companies or businesses that have consultants scattered around the world, the technical authorship team also helps overcome second-language frailties. Some caution needs to be maintained as these teams can be quickly overwhelmed with high workloads – which is why they should ideally report in to a senior consulting manager.
• Senior and managing consultants who have been “managing accounts” often have compensation plans linked to closing client deals. The incorporation of a dedicated sales team means that compensation plans need to be reevaluated for those consultants. Ideally this type of conversation happens prior to the hiring and buildout of a sales team – and that the consultants concerned are party to how the transition will occur and how compensation can be changed. Since the monies associated with managing an account are not often insignificant, it is vital that those consultants are offered alternative means of “making their number”. Luckily the company has several tools at their disposal. First of all, since the purpose of employing a dedicated sales team is to grow revenue and increase the billable hours of senior consultants, there is typically scope to increase the base salaries of those consultants and to create a bonus payment structure based upon utilization and customer satisfaction levels. Alternatively, that important role conversion in to a consulting manager (i.e. SE) can be useful in a hybrid compensation model, where factors such as new clients versus lateral growth in an existing client are bonused differently.

The business transformation from 100% consultants to a mix of consultants and dedicated sales personnel can be perilous if not managed carefully. The senior consultants need to be well informed and actively participate in the transition, and the sales team built gradually from a nucleus of experienced sales professionals that have come from consulting businesses that had already successfully transitioned.

Any transition will take time. The senior consultants in particular must be gradually weaned off their account management responsibilities, and replaced with ones that drive a higher utilization rate for them and any other consultants they may lead. The worst thing a leadership team can do is to expect the transition to happen overnight. Instead, they should anticipate the process being a 3-9 month transition; the end result is worth it though.