Kelihos' Voodoo Patronage

The King is dead. Long live the King! Or, given this week’s events, should the phrase now be “Kelihos is dead. Long live Kelihos”?

It is with a little amusement and a lot of cynicism that I’ve been watching the kerfuffle relating to the latest attempt to take down the Kelihos botnet. You may remember that a similar event (“Kelihos is dead”) occurred late last year after Microsoft and Kaspersky took it on themselves to shut down the botnet known as Kelihos (or sometimes as Waledac 2.0 or Hlux). Now, like a poor sequel to a TV docu-drama, Kaspersky and a number of other security vendors have attempted to slap down control of Kelihos Season Two – meanwhile Season Three of Kelihos has just begun to air. 

 

In the most recent attempt to interrupt the business operations of the criminal entity behind the Kelihos botnet, a bunch of threat researchers have managed to usurp command and control (C&C) of the Kelihos.B crimeware package by poisoning the peer-to-peer (P2P) relationships between all of the infected devices and install a surrogate control server. It’s good technical work by all those concerned, but has also proved to be ineffective if the objective was to actually takedown the botnet.

The good guys have set up what amounts to a sinkhole for a particular configuration of the Kelihos.B crimeware – with the Kaspersky blog initially identifying some 116,000 infected devices around the world. Like I’ve said many times before, botnets reliant upon P2P for transport of C&C information and stolen data propagation are vulnerable to this kind of takeover and victim enumeration. It’s one of the reasons we don’t see P2P being used much by sophisticated criminal groups – and almost never as a vehicle for attacks that target businesses.

Having said that, takeovers of portions of P2P botnets such as this most recent Kelihos.B example worry me quite a bit – it’s a reason why Damballa doesn’t offer to do this kind of work despite having excellent real-time visibility of the threat and the victims. There are two elements of P2P botnet takeovers that cause me the most concern:

1. To usurp control of the P2P botnet you have to initially join it in some shape or fashion, and then you have to send commands (via the P2P network) to all the other infected devices and redirect them to something you control.
   A victim of the Kelihos.B crimeware would be unable to differentiate the “good guys” from the “bad guys” – after all, their computer is still under someone else’s unauthorized control – and they could justly bring a legal case against those parties that seized control of their computer.
2. The use of sinkholes for victim data harvesting. It raises all kinds of questions about how you’re using someone’s stolen data – let alone the effect of sharing that victim information with other commercial entities. Obviously I have a strong opinion as to the ethics of selling these kinds of stolen information or using it for commercial purposes.

Despite the sinkholing attempt and the cries from the highest watchtowers that the Kelihos botnet is dead, it obviously isn’t. Within the security community there’s consequently a lot of discussion already over the semantics of what constitutes a botnet and what it means to have “killed” a botnet.

For example, the criminal operators behind the Kelihos.B botnet have been rolling out a new and improved variant of their crimeware – Kalihos.C – and it’s infecting a whole bunch of new victims (with some overlap with the Kelihos.B botnet victims). The fact that a new malware variant is being distributed to an overlapping group of victims seems to cause some degree of confusion to a few people.

Based upon my own observations, I’d be more inclined to take care when differentiating between the gang that operates botnets, botnets that share the same C&C infrastructure, and campaigns of crimeware updates and their installation. The claim of taking down the Kelihos botnet (twice now) is clearly false. It would be more precise to say that certain Kelihos campaigns have been disrupted. The criminals (and their core infrastructure) haven’t been significantly affected. In fact, the speed at which the Kelihos criminal gang was able to release an updated variant (Kelihos.C) reflects the futility of much of the current takedown effort.

Why go to all this effort? Why invest in Wac-a-mole style takedowns? While the efforts to takedown some Kelihos.A and Kelihos.B P2P botnets haven’t succeeded, they have enabled researcher to better understand the nature of the threat and hone their skills in the art of takedown. Knowing what doesn’t work (and why) is almost as valuable as knowing what does work.

I’m sure some group is going to try their hand at taking down Kelihos.C (and probably Kelihos.D) based botnets in the future. There’ll probably be the same claims of “Kelihos is dead” too. Unfortunately, if the Kelihos botnet controllers want to escape this bothersome cycle of losing a few thousand botnet victims each time, they already have the means available to them. As I discussed earlier this month, we’ve observed a growing number of criminal operators adding DGA’s to their malware families as a backup strategy should their P2P C&C fail for whatever reason. If the Kelihos operators add that feature to their next variant the wac-a-mole efforts of Kelihos-P2P-swatters truly become inconsequential.

Like I’ve said before, if you’re going to take down a botnet you have to take out the criminals at the top. It’s the only way. Taking out the infrastructure they depend upon for distributing new infectious material and C&C is a disruption technique – a delaying tactic if you will, and maybe an evidence building process if you’re lucky. In the case of P2P-based botnets, there’s very little infrastructure you can get your hands on – and you’ll probably end up having to issue commands to botnet victim devices – which is fraught with legal and ethical problems.

Oh, one last thing. Even if you’re lucky enough to be able to take out the C&C infrastructure or mechanism of communication, if you don’t take out the infection vector – the mechanisms of distributing new crimeware variants – you’ve achieved very little. As evidenced by the most recent Kelihos botnet takedown attempt, the criminals retained their primary distribution system and are already accumulating thousands of new victims per day with their latest Kelihos-variant campaign.

Signature-less Detection That Works

There was a period of time not long ago in which signature-based threat detection was cutting-edge. Antivirus, intrusion detection systems (IDS), data leakage prevention (DLP), content filtering and even anomaly detection systems (ADS) all continue to rely heavily upon static signatures. In recent years vendors have shied away from discussing their dependence on such signatures – instead extolling supplemental “non-signature-based” detection technologies.


In most cases these “non-signature-based” detection technologies appear to be largely a marketing ploy rather than some kind of new innovative solution; consider it a redefinition of what the dictionary would ordinarily define as a “signature” to overcome the inhibitions or bias of potential product purchasers. For example, if you define a signature as a single basic regular expression, then a threshold-based alerting policy could be considered a “non-signature” alternative.

I wanted to provide a walkthrough of the logic of a (real) non-signature-based detection technology. Perhaps “technology” isn’t quite the right word; rather let’s discuss it in terms of a confidence-based detection system. For example, for the purpose of edification, let’s consider a device reaching out to remote server – a destination that hasn’t been touched by the device ever before.


In the vast majority of cases a device will need to resolve a domain name in order to determine the remote IP address of the sever it is planning on connecting to. That DNS traffic contains a wealth of information if you can correlate with other external data sources and you know how to act upon it.


Signature-based detection systems could of course look up the IP address or domain name against their blacklist. If either of those data elements appear upon a list, then the traffic could be classified as malicious and blocked – if not, everything is probably “fine”. In the case of detection systems that utilize dynamic reputation, a scalar value representing the “suspiciousness” of the domain/IP and perhaps the class of threat would be returned (even if the domain and/or IP has never been seen before) and a threshold-based action would occur within the protection technology.


Supplemental to those signature-based detection approaches you could instead start dissecting the observation – compiling levels of suspiciousness and circumstantial evidence – and arriving at a conclusion of maliciousness.


Consider the following aspects of determining the nature of the interaction based purely from observing the DNS traffic related to the resolution of a single host lookup:

1. Armed with a database of passive DNS information observed over a period of years from around the world you can easily answer the questions:
   “Is this the first time anyone has ever looked up that domain name?”
   “Is this the first time that anyone has ever received that IP address for that domain name?”
   Knowing that a particular device (let’s assume that this entire example is being played out within a corporate network in the USA – say a Fortune 1000 company) is the first thing ever to resolve a domain name or be directed to a new IP address should raise a small degree of suspicion. This type of thing should be very infrequent.
   
2. Armed with geographic IP distribution information, you’ll know which country the returned IP address belongs to.
   “Is the destination IP address located somewhere unfriendly?”
   “Is this a country I want this kind of device connecting to?
   Knowing that the IP address belongs to a country with which the Fortune 1000 company doesn’t do any/much business with may be a little suspicious and worthy of more thorough study.
   
3. Armed with a list of ISP netblock associations, and knowledge of which of these netblocks are “residential subscribers”, can shed light on the nature of hosting environment of the destination server.
   “Is the destination address a residential IP address?”
   “Is the destination address a static or dynamic IP?”
   Knowing that a corporate device it trying to connect to a remote server located within a residential network should be suspicious to most network administrators – hinting at a number of threats or unwanted traffic.
   
4. Focusing a little on the domain name itself, characteristics of its registration may be used to determine levels of suspiciousness:
   “When was the domain name registered?”
   “Are there any features of the domain registrant details that cause concern?”
   Knowing that the domain name was registered 2 hours ago is significant. So too is knowing that details of the registrant match a large number of previously detected and categorized malicious domains.
   
5. Mining an exhaustive passive DNS database and correlating domain and IP information with historical threat information can yield even more information about the nature of the threat.
   “How many other domain names have pointed to the same IP address over the last 1/7/30 days?”
   “Are any of the domain names pointing at that IP address known to be related to a particular threat?”
   By associating an unknown or previously unclassified domain with domains with which historical information and threat attribution exists, enables the corporate entity to evaluate a “guilt by association” value.
   
6. Armed with information about the DNS servers that provide the authoritative answer for resolving the domain query provides further insight in to the nature of the destination.
   “Is the domain reliant upon free Dynamic DNS provisioning?”
   “What is the reputation of the authoritative DNS server?”
   “Which country is hosting the authoritative DNS server?”
   Dynamic DNS (DDNS) services are heavily abused by cybercriminals today – and are rarely used by large commercial entities. Understanding the location and past history of the authoritative DNS server (e.g. what proportion of domains hosted at the DNS server have previously been identified as malicious?) hints to the legitimacy of the destination IP address.
   
7. With visibility over live and historical domain name resolutions made to a particular authoritative DNS server, it becomes possible to glean information about the nature of the lookups and suspiciousness of the domain name:
   “How frequently is this domain name looked up?”
   “Which countries or organizations have also looked up this domain name?”
   “Who was the first to look up this domain name and get that response?”
   Knowing that a particular domain name has only been looked up by three US-based Fortune 500 companies in the last year is suspicious. Knowing that the same domain name points to an IP address in Iran and has never been looked up by anyone else in the world would be highly suspicious and indicate a level of targeted attack.
   
8. Then of course there’s the more obvious categorized information relating to the IP address:
   “Is the IP address a known sinkhole?”
   “Is the IP address associated with a commercial content delivery network?”
   “Is the IP address associated to a domain registrar’s holding page?”
   Knowing that the IP address is pointing to a sinkhole is a pretty obvious indicator that someone already thinks this particular domain name is malicious and associated with a threat. Meanwhile, knowing that the domain is pointing to a generic domain registration holding page could indicate that the domain has been taken down for being malicious, or is waiting to be staged by the criminals, etc.
   

There are more features that can be extracted from the successful resolution of a domain name than those listed above – but I’m sure you appreciate by now that a large amount of information can be obtained and associated with even a previously unknown domain name – and a certain degree of confidence can be obtained as to the suspiciousness (or maliciousness) of it.


For example, consider the following scenario:

1. A domain was looked up by a device, and that was the first time ever the domain has been looked up in the history of the Internet.
   
2. The authoritative DNS response came from a free DDNS provider in China.
   
3. The domain name points to a residential, DHCP assigned IP address, in Tehran.
   
4. There are 8 other domain names that have pointed to that particular IP address over the last 30 days.
   
5. Five of those domain names have been referenced within previously captured malware as C&C.
   
6. All 9 domain names (the original plus the uncovered 8 ) point to 3 IP addresses within the same subnet of the residential ISP in Tehran over the last 30 days – typical of a DHCP network assignment.
   
7. Only 6 of these domain names have ever been looked up, and all lookups of those domains over the last 90 days and have only been carried out by 11 major US-based pharmaceutical companies.
   
8. None of the domain names have ever been looked up from within Iran.
   
9. The first time each of the 6 domains were looked up, they were done from an IP address associated with a popular blackhat free-VPN service provider.
   

Obviously it shouldn’t take a genius to figure out that not much good is going to come from the device connecting to this particular remote host. All the evidence is circumstantial, but pulled together it becomes actionable intelligence. Most importantly though, all of this can be carried out using just a single DNS response (before any malware is downloaded, before any vulnerabilities are exploited, and before any user is socially engineered) – meaning that protection systems that can handle this level of non-signature-based threat determination engine can take preventative actions before the device has even begun to connect to the destination server.


When I think of non-signature-based detection systems, this is one approach that springs to mind. Such deterministic systems exist today – and they’re working very nicely thank you.

The (Cyber) War Scientists

A few weeks ago I finished reading the book "The War Scientists" by Thomas Craughwell. It's a book about the individuals - the Scientists if you will - that invented some of the most significant weapons of war. As you'd expect, the book examines their technologies in combat and impact at the time.

By itself, the book is pretty interesting but only in the capacity of bringing together an Encarta-level of history to the fore.

Despite that, it got me thinking about the "who and what" would be included in such a book if it was to be updated in 20 years or so. Would "Cyber War" be worthy of inclusion, what cyber weapon would make the grade and which individual(s) would be included?

Lets for the moment assume that real, full-on "cyber-war" will occur within the next decade or so, and that the weapons to be used in that war either exist today or are derived directly from technologies (or methodologies) present today. Big assumptions, but hey - in the realm of the possible, anything is probable.

Is there a particular technology - turned weapon of war - that leaps out as a game changer? Likely candidates for many I suppose could include Stuxnet or maybe Metasploit, or perhaps even the MBR rootkit. But to my mind these are all derivatives of older tools and tactics - engineering advancements of someone elses invention.

Would the owner of the mind behind the first vulnerability scanner constitute a "scientist"? (probably not).

The more I think about it, despite a lot of innovation in the cyber arms trade, I don't think that any "game changers" have been made to the public eye yet. Is there an equivalent to the torpedo or mustard gas? Sure, you can draw some comparisons between various malware formats or features if you were so inclined, but where are the one-sided game-changing weapon developments?

You know what, I suspect that some of those cyber-weapon game-changers have already been invented or are being invented right now. Unfortunately (or should that be fortunately?) those weapons haven't been used yet and their inventors aren't likely make headlines sometime after the dust settles.

Musings over Sizing of the Botnet Menace

Pinning down the number of infected computers is really, really hard. I’d go as far as saying it’s practically impossible to calculate, let alone observe. Still, that’s not going to stop people from attempting to guess or extrapolate from their own observations. Over the years I’ve heard “reliable” numbers ranging from 10% through to 60% – and I don’t trust any of them.


There’s a whole gaggle of reasons why the numbers being thrown out to the public are inaccurate and should ideally be interpreted with a lot of skepticism by any right-minded folks. If I had to boil it down to only a couple of categories of reasons they’d be semantics and observational bias. Semantic, because terms such as “infected computers” and “compromised devices” are different from “compromised users” and “victims”, and observational bias because no vendor is omnipresent and their perspective of the threat is represented only by their category of customer and the tools they employ.

These problems represent hurdles for a number of collaborative projects seeking to measure and track the botnet menace. There are several initiatives (e.g. the Online Trust Alliance) and working groups (e.g. the Messaging, Mobile, and Malware Anti-abuse Working Group) striving to collate disparate datasets and views of botnet infections with the hope that the industry can baseline the problem in order to track and measure the success of other initiatives designed to reduce the threat. The premise being if you can’t measure it, how do you know you’ve been successful in fixing the problem?

Given Damballa’s unique perspective of the botnet threat and participation in various working groups on the topic, I thought I’d share a little of what we’re observing – and the bounds of what that means.
First of all, it’s important to note that Damballa has two major product lines – one catering for large enterprise networks (Damballa Failsafe), and the other focused on ISP’s and Telco’s (Damballa CSP , for communications service providers). Given the nature of these products and the types of customers that purchase them, there are effectively two major “infection” statistics of note for this first part of 2012:

• When we deploy Damballa Failsafe we find that, on average, between 3-7% of assets within enterprise networks are identified as being infected and are actively searching for, or successfully connecting to, a cybercriminals C&C server.
• Within the ISP/Telco world that have chosen to deploy the Damballa CSP product, between 18-22% of unique subscriber IP addresses are actively seeking to connect to known C&C servers.

These infection statistics are not directly comparable. In the case of Damballa Failsafe deployments, we’re able to track and identify the unique device that has been infected by any number of crimeware instances, and separate out all of the C&C and data leakage communications, and differentiate between infections. In the case of the Damballa CSP product, because of ISP-level restrictions on deep packet inspection (DPI) and the fact that a subscriber IP address encompasses any and all devices within that subscribers personal network, we’re only able to deduce that at least one device within that subscriber’s network is part of a particular botnet (but we can enumerate each of the multiple botnets that may be operating within that subscribers network).

For the sake of this being a blog, let’s focus on the topic of “household botnet infections”. For all intents and purposes in the residential ISP world, a subscriber’s IP address is pretty close to being analogous to a “household”. Out of the aggregated 125 million subscriber IP addresses that Damballa CSP product monitors from within our ISP customer-base from around the world, the vast majority of those subscriber IP’s would be classed as “residential” – so it would be reasonable to say that roughly 1-in-5 households contain botnet infected devices.

From previous observations we also know that approximately 40% of infected devices have two or more botnet infections within them (see the H1 2011 Damballa Threat Report). Now if only we knew what the average number of devices within residential home networks is. Alas, I can’t find out that information (send me the info if you happen to know!). When I last looked at my poor wireless router’s admin panel at home, it would appear that I have something like 40 IP enabled devices chatting away and connecting to the Internet. Who knows, but I suspect that my household probably isn’t typical – and shouldn’t be used for any kind of extrapolation.

Anyhow, with all those numbers in mind, where in this “10% through to 60%” scale of global infected computers do I think the true numbers lie? Well there’s one more caveat to all this – it’s the semantic piece – infected computers is a superset of botnet infected devices. What Damballa product deployments are capable of enumerating (since they sit at the network level, and not at the host) are infected devices that are actively trying or successfully engaging with a criminals C&C infrastructure – and not all malware does this, and not all devices are “computers”. So malware that cannot be controlled or tasked remotely by a criminal, and malware that doesn’t upload stolen data somewhere over the network, aren’t going to appear in my observation statistics.

Given that the average number of devices within a residential subscriber network is going to be greater than one (let’s say “two” for now – until someone has a more accurate number), I believe that it’s reasonable to suggest that around 10% of home computers are infected with botnet crimeware.

With regards to “infected” computers (i.e. all types of malware – not just botnet malware), I don’t know what the ratio of botnet malware is to the overall malware installation problem. Of all the malware caught and shared globally amongst commercial antivirus vendors, the majority of malware samples would certainly seem to be “droppers” and “downloaders” (choose your terminology) – mostly because of serial variant production systems. Perhaps the desktop antivirus statistics are right with the 60%+ of computers being infected – but I doubt it (since the desktop antivirus products are only going to report the stuff they’re capable of detecting and stopping – not the slippery stuff).

Network Antivirus Evasion with XOR

Many organizations I speak with have instigated network filtering and security monitoring solutions targeted at identifying malicious binaries traversing their egress points. Something that they’ve been observing in recent months is the increase of suspicious binaries that are unsupported and non-executable. Ordinarily any intercepted binaries would be farmed off to static anti-virus scanners or tin-wrapped behavioral analysis engines for classification; however a growing volume of these binaries cannot be scanned or executed within virtual environments. What’s going on?

More often than not, these perimeter network defenses are encountering encoded and obfuscated malicious binaries – constructed purposefully by an attacker to bypass network threat detection products. These evasions aren’t anything new, it’s just that the tools and functionality to encode malicious binaries “on-the-fly” have become standard features in a growing number of automated attack delivery tools and DIY botnet construction kits.

The non-executable binaries are typically malicious binaries that have been encoded using simple, light weight, cryptographic techniques. They need to be decoded at the receivers end and decrypted back in to their “original” file format for proper malicious execution. In many cases the entire (original) malicious binary is encrypted using a simple XOR cipher. While there are no shortage of techniques that can be used (take a look at the default assortment of file encoders within the Metasploit MSFencode module for instance), XOR does seem popular and is more than “good enough” to bypass existing security technologies. Sometimes the simplest evasion techniques are the best.

Just to be clear though, we’re talking about malicious binaries that have been fully encrypted and require a separate application or script running on the victim’s computer to decrypt them (i.e. they cannot be executed by themselves) — not executable binaries that have been armored for local anti-virus evasion purposes and are shipped in an already executable format. There’s a very nice walk through of manually armoring files for the purpose of evading local host inspection of malicious binaries in “Bypassing Anti-Virus Scanners“, and there’s a good discussion of using Metasploit to construct evasive executable content over on offensive-security.com for comparisons sake.

The non-executable encrypted binaries are typically either droppers for newly compromised victims, or malicious updates for existing victim installations. In the case of newly compromised victims, the encrypted malicious binary will be requested by the attacker’s payload (e.g. shell script) and decrypted in memory before being installed on the victim. For existing victims, the encrypted binary will be requested by an already installed malware component, decrypted, and used to replace existing components of the malware installation.

There are other permutations to the theme as well. For example, the use of malicious Adobe Flash files that serve as the transport instigator for the malicious commands such as the imm32.dll dissected by Stop Malvertising, or the use of PNG files to propagate XOR’ed malware.

What does this mean for automated network-based anti-virus solutions (such as Norman, Trend Micro, etc.) and auto-signature-creator anti-virus products (such as FireEye, Palo Alto Networks’ Wildfire, etc.)? Since the intercepted binary is incapable of execution by itself and will never run under an emulated or virtualized behavioral system, it will not be automatically analyzed or classified as malicious. Only the targeted victim will uncover the unsavory payload.

Because the technique works so well and is so easy to do, we can expect the trend to continue. Organizations will capture more and more suspicious non-executable binaries going forward (often containing little or no hint as to what the file may potentially be).

What can be done? I believe a key perspective to dealing with this evasion technique is to better understand the suspicious nature of the file transfer. Consider it “contextual awareness”. Despite not being able to dissect and analyze the malicious binary directly, understanding the context of its transport will likely provide enough circumstantial evidence to arrive at a comfortable conclusion as to the maliciousness of the binary. For example:

• What is the reputation of the file download source? (E.g. a corporate entity, file sharing site, hacker forum)
• Is the source known to only host malware? (e.g. malicious files are encountered there frequently)
• Is the file requester already known to be infected? (e.g. the requester is a known botnet victim)
• Is there a cyclic pattern to the download file requests indicative of previous compromise? (e.g. an existing malware installation is requesting new updates)
• What is the nature of the source? (e.g. the hosting server is a known C&C server)
• What are the ports and protocols involved? (e.g. HTTP over a low (non-TCP80) port number)
• What are the parameters of the file? (e.g. the file is not readable by any commonly installed applications on the destination host, and is approximately the same size are regularly encountered malware samples)

At the end of the day, the attackers have the upper-hand when it comes to obfuscating malicious binaries. The increased application of XOR ciphers designed to evade network-based binary inspection tools and appliances makes sense, and is more than sufficient to thwart these protection technologies. In that case, it makes increasing sense to look for other network artifacts that can hint at the suspiciousness of the binaries makeup – and to use that contextual awareness to decide upon an automated defense reaction.

A Bug Hunter's Diary (book review)

A couple of months ago I got my hands on Tobias Klein's new book "A Bug Hunter's Diary" and have only recently managed to read through it and, I have to say, I liked it very much.

The book takes the reader through a guided tour of seven vulnerabilities uncovered by Tobias over the last few years. Unlike most books on the topic of bug hunting (which typically focus on walking through the tools of the trade and talking in generalities) this book takes on a pseudo-diary format - revealing the thoughts, assumptions and leaps-of-faith that go in to uncovering the kinds of bugs that make the headlines.

As someone who's worked extensively in the commercial bug hunting and vulnerability exploitation business, nothing beats the shoulder-surfing approach to knowledge transfer, and I think this book manages to achieve much of that experience.

Given the span of bugs, platforms and years between discoveries, it provides an interesting perspective on the responses of vendors (and product maintenance engineers) to bugs that come their way and their capability to respond/fix them. My, how times have changed (in a good way - generally).

As a technical book, I think it has legs and I don't think it'll date quickly. Tobias works through the bugs in a logical and well thought out way and, as long as the reader has some familiarity with debuggers and some coding prowess, it shouldn't be that technically taxing. The best bug hunters aren't elite coders and assembly guru's - they're folks that explore imaginative "what if?" scenarios within the software or devices they're looking at.

What bugs are covered? Well, there are several, but divided in to the following major categories:

• VideoLAN's VLC media player
• Sun Solaris kernel
• FFmpeg multimedia library
• WebEx ActiveX
• Avast! AV
• OSX TTY IOCTL
• iPhone

Who's going to benefit from this book? I think the book will be well suited to senior engineers charged with debugging glitches in their companies software and folks looking to make the leap from being tool-only penetration testers and security consultants. The kind of folks that have been to one or two Blackhat Las Vegas conferences in the past, listened to various bug hunters spout their latest findings from the podium, and figured that they'd like to give it a try for real.

Shoulder-surf in the comfort of your own home (or Kindle)!

Security: The Art of Compromise

“Security is all about protecting the user.” That’s the comment that came up the other week in the twittersphere that kicked off a not-unexpected trail of pro and con tweets.

Being limited to 140 characters makes it rather difficult to have a deep and meaningful discussion on the topic and the micro-blogging apparatus isn’t particularly conducive to the communicating the nuances of a more detailed thought. So I thought I’d address the topic here in blog format instead.

I suppose my first thought is that Internet security isn’t necessarily about protecting the user, in fact I’d go as far as saying that modern security approaches increasingly assume that the user themselves is the threat. In the most optimistic case security is about protecting assets (digital or physical) from harm or theft. Failing that, Internet security is about detecting change to assets that were deemed to merit protection.

As a community we tend to over use safety analogies when we’re trying to communicate the significance of a threat and the value of protection – which is why I believe there’s a prevailing assumption that Internet security is about protecting the user. For example, I’ve often heard (and abused it myself) the car analogy for putting defense in depth in to perspective – i.e. airbags, safety belts, crumple zones, etc. being metaphors for anti-virus, IDS and firewalls.

I think a more appropriate analogy for modern Internet security practices is that of protecting a bicycle. The cyclist is the user, and by protecting the bike itself we’re not actually doing much for the safety of the rider. In fact I’ll argue that over-protecting the bike may end up decreasing the safety of the cyclist – as we too often see in the cyber world (e.g. the lock’s so big & heavy that it affects the cyclists ability to actually ride the bike). By way of problem statement, we can consider the cyclist as a consumer of the technology (i.e. the bicycle) and, for him to be able to ride, he needs to ensure that his bike hasn’t been stolen or damaged.

When it comes to “protecting” the bike, there are a number of factors the would-be cyclist needs to take into account. Likely the most important concern is going to be how to lock-up the bike when not in use – especially when away from home. The most obvious solution is to purchase a dedicated bicycle lock.

Now this is where I think the analogy works better than most for the Internet security world… what are some of the deliberations the cyclist must make in selecting an appropriate protection solution?

• How big a lock do I need? A small lock can be trivially overcome, but is light and easy to carry on longer rides. A big heavy lock will likely be much harder to overcome, but is going to be troublesome to carry.
• How long a chain? A short chain is easier to carry. Meanwhile a longer chain offers me the flexibility to also lock up the wheels and wrap around bigger objects.
• How much do I want to spend? Some top-quality locks are almost as expensive as the bicycle they’re destined to protect. Meanwhile a more expensive lock may be lighter and more proficient at keeping thieves away.

Deciding upon a protection solution is a practice in compromise – comparing the risk likelihood with the inhibitors of the proposed solution. There’s also awareness that no matter how big and badass the lock may be, there’ll always be someone out there with more powerful bolt-cutters or a more imaginative way of subverting the lock. It may be a compromise, but hopefully it is an informed decision. The cyclist opts for a solution, forks out the money, and lives with the decision. If all goes to plan, their bicycle will be present the next time they go to use it.

The same applies to the Internet security world. You can’t protect against all the threats and, even if you could, you’d likely end up making the system you’re trying to protect unusable for the folks that need to use it.

But “protection” is only one side of the security coin. “Detection” is a critical element of modern security. Some may argue that detection is something you aim to do if you can’t protect. I’d have to politely disagree.

Locking up your bike is a realistic security solution if you’re looking to protect it – but ensuring that your bike is locked up somewhere highly visible (good lighting, etc.) and located where a potential thief is likely to be noticed by the cyclist and other passerby’s is a critical “detection” component. The threat of detection becomes part of the security strategy. Even if that deterrent fails, and the protection was also insufficient, the sooner the cyclist knows whether their bicycle has been stolen or tampered with, the quicker they can respond to the threat and take the corresponding actions.

Detection within the Internet security realm is as important as protection. For the last decade or so the emphasis upon security has been protection – despite acknowledging the limits of the compromise situations and product vulnerabilities. Knowing precisely when an asset has become the focus of a would-be thief or eventually succumbing to the threat is critical in how an organization must respond to the incident.
As anyone who has had a bike stolen will tell you, the quicker you notice it’s gone, the higher the probability you have of getting it back.

Virtual Execution and the Emperors New Clothes

Some ideas sound so attractive in principle that it’s hard to fathom why the Internet security industry hasn’t leapt up and down all over them already. Take for instance the idea of automatically processing malware within a virtualized host, capturing the network traffic that’s generated, automatically generating an IDS signature for the malicious traffic, and then deploying the signature to a network protection platform. It sounds so enticingly simple – almost elegant. So why isn’t everyone doing this? Why aren’t we protected from malware? Because it doesn’t really work, and never has – at least not against any contemporary portfolio of targeted threats and crimeware!

The idea of automatically generating signatures for network borne threats isn’t precisely new. In fact the concept dates back to the early 1980′s and predates network-propagated malware attacks by quite a margin.

Back in 1997, network-based intrusion detection systems (IDS) really started to take off as a viable commercial security technology – with ISS rolling out their RealSecure – and the first (internal vendor) prototype systems were developed for automatic signature development. Even in those early days where malware threats were exceedingly noisy and clumsy on the network, it was difficult to develop reliable signatures that wouldn’t let loose a tide of false positive results. Any automatically generated signature needed to be reviewed by skilled engineers and “soak tested” in pliable customer networks – and subsequently tweaked or dropped, depending upon false positives observations.

Fifteen years later new vendors continue to reinvent the wheel and rediscover the pain behind the promise. Take for example Palo Alto Networks’ Wildfire and FireEye’s Virtual Execution Engine approaches (what’s with the “fire” thing?). Both capture suspicious binaries observed traversing their customer’s networks and hand them over to a software virtualization system for controlled execution and (eventual) automatic signature generation. Those signatures (basically a Snort regex-based signature) and then pushed to IDS sensors to alert upon.

What’s the matter with that? Well, for anything beyond the most ubiquitous and dumb malware out there, the approach will either result in false positive alerts or fail to alert altogether. The latter is more common than the former because the systems are tuned that way. After all, no one likes to be woken up late at night due to a false positive – so if you miss the threat entirely, who’s ever going to know?

There are way too many reasons why automated signature systems like FireEye and Wildfire continue to fail in detecting and diagnosing todays targeted threats (note that this isn’t a problem unique to just them – it applies to the other dozen or so near-identical products from other vendors).

You may remember a paper I published mid-2011 on the topic “Automated In-Network Malware Analysis: Why Virtual Machines Can Sputter and Miss“. That paper alone provides numerous examples of what the badguys are doing to bypass automated analysis systems and the perils of not operating analysis platforms that replicate real victim systems (which is an Achilles Heel of these appliance-led “protection” systems).

Out of all the possible ways the badguys can usurp these auto-generated malware detection engines, let’s look at one of the more recent ones – malware that makes use of Domain Generation Algorithms (DGA).

DGA Evasion

Last week Damballa Labs released a report and case study on the current generation of DGA-based crimeware and you’ll likely have noted that of the 12 newly uncovered DGA’s, 6 of them were associated with advanced (but mainstream) crimeware families. Now here’s the rub, if you’re reliant upon an automated system for malware analysis to generate “protection” signatures for your IDS, I hope you’re enjoying the Emperor’s new clothes because you’re naked to the C&C communications of Shiz, Bimital, BankPatch, Expiro.Z, Bonnana, and recent generations of Zeus (to name but a few).

How is that possible? Simply put, several of the features of DGA’s are designed to intentionally bypass these kinds of dynamic protection technologies. By not using static domain names for their C&C lookups, and by ensuring that their network communications look like standard HTTP/HTTPS traffic, any automatically generated “call back” signature will be based upon redundant and outdated information before it’s even deployed – and subsequently never trigger against real crimeware traffic.

As the crimeware sample executes within the virtualized analysis environment (and assuming that it’s not bothering to conduct any anti-VM evasion in the first place (which is a fallacy of course)) it will initiate its DGA and attempt to look up a number of dynamically calculated domain names. Those domain names and the C&C servers they are hunting for reflect the current date/time of the malware’s execution. At a different date or time additional malware samples will generate their own sets. The net result is that, assuming the vendor’s product can create a signature in the first place, the signature will be for only a single malware sample at a fixed (past) point in time. Adding that signature to the IDS and alerting system can only result in false positive detections – and lost incident response cycles.

With an evasion technique as successful as the modern DGA, it is not a surprise to see it being adopted by additional crimeware families.

There is a technique for identifying DGA’s and the victims of DGA-based crimeware. The dynamism of DGA operational use means that the detection technologies need to more flexible and operate independently of static signatures. As such, the current generation of DGA-based threat detection focuses upon DNS-level observations and utilizes clustering and spectral analysis approaches.

Being Stealthy with DGA Technology

Do you remember all the fuss about Conficker many moons ago and its odd method of locating C&C servers? Instead of relying upon a static list of preconfigured domain names that corresponded to the location of the badguys C&C servers, it used an algorithm to calculate candidate domain names – and then tried reaching out to a handful of the candidates in a vein attempt to locate an active C&C server.

The authors behind the Conficker variants experimented with a number of algorithms but, at the end of the day, they failed to construct a cohesive botnet. Despite that “minor flaw”, Conficker infected devices still account for a sizable fraction of known malware infections around the work – years after the threat was studied to death and detection/protection/cleanup solutions are available everywhere.

There were a few other malware families that briefly rode the coattails of Conficker – further refining the Domain Generation Algorithm (DGA) technique as a form of evasion against protection systems that relied upon blacklists, signatures and pseudo-signatures. But, after a few months, the threat had been largely forgotten.

“Forgotten” probably isn’t the right word. The Conficker Working Group is still tracking victims and hoping against hope to identify the masterminds behind the botnet should they ever attempt to set up a real C&C server and provide new instructions to the derelict zombie hoard they created. By “forgotten” (complete with quotes), what I refer to is a general complacency for the threat and the inability of today’s network protection systems and monitoring solutions to identify malware that makes use of DGAs.

The DGA threat was a technique uncovered by malware reverse engineers and rose to public attention because of the novelty of the method – rather than any advancement’s in detecting and mitigating the threat. DGAs are a pain – they’re supposed to be. They exist to defeat network detection and blocking technologies, and they did it really well. Today they’re doing it even better!

While DGA’s disappeared from a media perspective, they also disappeared from a threat protection perspective too. They disappeared largely because the badguys got better – not because they stopped using them or consigned them to history.

Damballa Labs released a report recently covering new DGA discoveries and global trends, as well as a blow-by-blow case study of one DGA-based crimeware campaign. Whilst I recommend that you read the reports yourself, there are a number of really important findings that you should pay attention to:

1. DGAs aren’t dead. Instead, they’re being added to already-stealthy crimeware at an alarming rate.
2. DGAs are being adopted as backup strategies. Even if the crimeware family is well known and it’s traditional C&C infrastructure is blocked or disabled (e.g. web filtering, firewall blacklists, etc.), the DGA fallback plan is kicking in and allowing the crimeware to receive new instructions and upload stolen data.
3. C&C servers are becoming more agile. The criminal operators behind the DGA-based crimeware are exposing their C&C servers for the minimum amount of time. Domains are registered and DNS configurations are made “just in time” (i.e. a few minutes before the algorithm is supposed to call the domain), and the C&C servers are shut down and removed immediately afterwards – something that can be done within an hour.
4. Dynamic analysis and auto-gen signature systems are failing. Automated systems that perform dynamic malware analysis on the DGA-based crimeware are producing irrelevant detection signatures. By the time the crimeware passes through the analysis and a signature is deployed for alerting/blocking, the crimeware family is already on to the next C&C server possibility.

While the addition of DGAs to already stealthy and advanced crimeware is a significant threat to corporate defenses, it should be noted that the processes now employed by their criminal operators concerning the registration of domain names, configuration of DNS, and addition/removal of C&C servers is equally advanced. Like a delicate ballet dance, professional cybercriminals are optimizing their deployments for the maximum gain and the lowest exposure.

That said, there are still ways of detecting both the employment of DGAs and their crimeware victims – and this can be done on a very large scale. Employing a number of novel techniques applied to DNS traffic, it is possible to automatically enumerate the threat. It’s not easy and bound to result in a number of patents, but it can be done – as hinted at in the Damballa Labs research report.

The obvious question: “If DGAs can now be detected, won’t the bad guys stop using them or modify them so they’re less detectable?” Well, it’s OK with me if the bad guys stop using them – but I doubt that that’s likely. The technique is stealthier than any other they have in their arsenal and will continue to work against the traditional/legacy network protection platforms for a long time to come. As for modification, there’s very little they can do beyond reducing the number of domains the algorithm tries per day (which makes it more susceptible to traditional blacklist approaches if they do), or they employ less randomness in their algorithms – which makes the approach less flexible and more cumbersome (which would slow down the detection of small botnets, and have no effect on large infection outbreaks).

In the meantime it looks like a sizable number of new DGAs have been hiding out there for the last year – undetected by the multitude of security vendors and researchers. I’m sure there are many more awaiting illumination.