Signature-less Detection That Works

There was a period of time not long ago in which signature-based threat detection was cutting-edge. Antivirus, intrusion detection systems (IDS), data leakage prevention (DLP), content filtering and even anomaly detection systems (ADS) all continue to rely heavily upon static signatures. In recent years vendors have shied away from discussing their dependence on such signatures – instead extolling supplemental “non-signature-based” detection technologies.


In most cases these “non-signature-based” detection technologies appear to be largely a marketing ploy rather than some kind of new innovative solution; consider it a redefinition of what the dictionary would ordinarily define as a “signature” to overcome the inhibitions or bias of potential product purchasers. For example, if you define a signature as a single basic regular expression, then a threshold-based alerting policy could be considered a “non-signature” alternative.

I wanted to provide a walkthrough of the logic of a (real) non-signature-based detection technology. Perhaps “technology” isn’t quite the right word; rather let’s discuss it in terms of a confidence-based detection system. For example, for the purpose of edification, let’s consider a device reaching out to remote server – a destination that hasn’t been touched by the device ever before.


In the vast majority of cases a device will need to resolve a domain name in order to determine the remote IP address of the sever it is planning on connecting to. That DNS traffic contains a wealth of information if you can correlate with other external data sources and you know how to act upon it.


Signature-based detection systems could of course look up the IP address or domain name against their blacklist. If either of those data elements appear upon a list, then the traffic could be classified as malicious and blocked – if not, everything is probably “fine”. In the case of detection systems that utilize dynamic reputation, a scalar value representing the “suspiciousness” of the domain/IP and perhaps the class of threat would be returned (even if the domain and/or IP has never been seen before) and a threshold-based action would occur within the protection technology.


Supplemental to those signature-based detection approaches you could instead start dissecting the observation – compiling levels of suspiciousness and circumstantial evidence – and arriving at a conclusion of maliciousness.


Consider the following aspects of determining the nature of the interaction based purely from observing the DNS traffic related to the resolution of a single host lookup:

1. Armed with a database of passive DNS information observed over a period of years from around the world you can easily answer the questions:
   “Is this the first time anyone has ever looked up that domain name?”
   “Is this the first time that anyone has ever received that IP address for that domain name?”
   Knowing that a particular device (let’s assume that this entire example is being played out within a corporate network in the USA – say a Fortune 1000 company) is the first thing ever to resolve a domain name or be directed to a new IP address should raise a small degree of suspicion. This type of thing should be very infrequent.
   
2. Armed with geographic IP distribution information, you’ll know which country the returned IP address belongs to.
   “Is the destination IP address located somewhere unfriendly?”
   “Is this a country I want this kind of device connecting to?
   Knowing that the IP address belongs to a country with which the Fortune 1000 company doesn’t do any/much business with may be a little suspicious and worthy of more thorough study.
   
3. Armed with a list of ISP netblock associations, and knowledge of which of these netblocks are “residential subscribers”, can shed light on the nature of hosting environment of the destination server.
   “Is the destination address a residential IP address?”
   “Is the destination address a static or dynamic IP?”
   Knowing that a corporate device it trying to connect to a remote server located within a residential network should be suspicious to most network administrators – hinting at a number of threats or unwanted traffic.
   
4. Focusing a little on the domain name itself, characteristics of its registration may be used to determine levels of suspiciousness:
   “When was the domain name registered?”
   “Are there any features of the domain registrant details that cause concern?”
   Knowing that the domain name was registered 2 hours ago is significant. So too is knowing that details of the registrant match a large number of previously detected and categorized malicious domains.
   
5. Mining an exhaustive passive DNS database and correlating domain and IP information with historical threat information can yield even more information about the nature of the threat.
   “How many other domain names have pointed to the same IP address over the last 1/7/30 days?”
   “Are any of the domain names pointing at that IP address known to be related to a particular threat?”
   By associating an unknown or previously unclassified domain with domains with which historical information and threat attribution exists, enables the corporate entity to evaluate a “guilt by association” value.
   
6. Armed with information about the DNS servers that provide the authoritative answer for resolving the domain query provides further insight in to the nature of the destination.
   “Is the domain reliant upon free Dynamic DNS provisioning?”
   “What is the reputation of the authoritative DNS server?”
   “Which country is hosting the authoritative DNS server?”
   Dynamic DNS (DDNS) services are heavily abused by cybercriminals today – and are rarely used by large commercial entities. Understanding the location and past history of the authoritative DNS server (e.g. what proportion of domains hosted at the DNS server have previously been identified as malicious?) hints to the legitimacy of the destination IP address.
   
7. With visibility over live and historical domain name resolutions made to a particular authoritative DNS server, it becomes possible to glean information about the nature of the lookups and suspiciousness of the domain name:
   “How frequently is this domain name looked up?”
   “Which countries or organizations have also looked up this domain name?”
   “Who was the first to look up this domain name and get that response?”
   Knowing that a particular domain name has only been looked up by three US-based Fortune 500 companies in the last year is suspicious. Knowing that the same domain name points to an IP address in Iran and has never been looked up by anyone else in the world would be highly suspicious and indicate a level of targeted attack.
   
8. Then of course there’s the more obvious categorized information relating to the IP address:
   “Is the IP address a known sinkhole?”
   “Is the IP address associated with a commercial content delivery network?”
   “Is the IP address associated to a domain registrar’s holding page?”
   Knowing that the IP address is pointing to a sinkhole is a pretty obvious indicator that someone already thinks this particular domain name is malicious and associated with a threat. Meanwhile, knowing that the domain is pointing to a generic domain registration holding page could indicate that the domain has been taken down for being malicious, or is waiting to be staged by the criminals, etc.
   

There are more features that can be extracted from the successful resolution of a domain name than those listed above – but I’m sure you appreciate by now that a large amount of information can be obtained and associated with even a previously unknown domain name – and a certain degree of confidence can be obtained as to the suspiciousness (or maliciousness) of it.


For example, consider the following scenario:

1. A domain was looked up by a device, and that was the first time ever the domain has been looked up in the history of the Internet.
   
2. The authoritative DNS response came from a free DDNS provider in China.
   
3. The domain name points to a residential, DHCP assigned IP address, in Tehran.
   
4. There are 8 other domain names that have pointed to that particular IP address over the last 30 days.
   
5. Five of those domain names have been referenced within previously captured malware as C&C.
   
6. All 9 domain names (the original plus the uncovered 8 ) point to 3 IP addresses within the same subnet of the residential ISP in Tehran over the last 30 days – typical of a DHCP network assignment.
   
7. Only 6 of these domain names have ever been looked up, and all lookups of those domains over the last 90 days and have only been carried out by 11 major US-based pharmaceutical companies.
   
8. None of the domain names have ever been looked up from within Iran.
   
9. The first time each of the 6 domains were looked up, they were done from an IP address associated with a popular blackhat free-VPN service provider.
   

Obviously it shouldn’t take a genius to figure out that not much good is going to come from the device connecting to this particular remote host. All the evidence is circumstantial, but pulled together it becomes actionable intelligence. Most importantly though, all of this can be carried out using just a single DNS response (before any malware is downloaded, before any vulnerabilities are exploited, and before any user is socially engineered) – meaning that protection systems that can handle this level of non-signature-based threat determination engine can take preventative actions before the device has even begun to connect to the destination server.


When I think of non-signature-based detection systems, this is one approach that springs to mind. Such deterministic systems exist today – and they’re working very nicely thank you.

Network Antivirus Evasion with XOR

Many organizations I speak with have instigated network filtering and security monitoring solutions targeted at identifying malicious binaries traversing their egress points. Something that they’ve been observing in recent months is the increase of suspicious binaries that are unsupported and non-executable. Ordinarily any intercepted binaries would be farmed off to static anti-virus scanners or tin-wrapped behavioral analysis engines for classification; however a growing volume of these binaries cannot be scanned or executed within virtual environments. What’s going on?

More often than not, these perimeter network defenses are encountering encoded and obfuscated malicious binaries – constructed purposefully by an attacker to bypass network threat detection products. These evasions aren’t anything new, it’s just that the tools and functionality to encode malicious binaries “on-the-fly” have become standard features in a growing number of automated attack delivery tools and DIY botnet construction kits.

The non-executable binaries are typically malicious binaries that have been encoded using simple, light weight, cryptographic techniques. They need to be decoded at the receivers end and decrypted back in to their “original” file format for proper malicious execution. In many cases the entire (original) malicious binary is encrypted using a simple XOR cipher. While there are no shortage of techniques that can be used (take a look at the default assortment of file encoders within the Metasploit MSFencode module for instance), XOR does seem popular and is more than “good enough” to bypass existing security technologies. Sometimes the simplest evasion techniques are the best.

Just to be clear though, we’re talking about malicious binaries that have been fully encrypted and require a separate application or script running on the victim’s computer to decrypt them (i.e. they cannot be executed by themselves) — not executable binaries that have been armored for local anti-virus evasion purposes and are shipped in an already executable format. There’s a very nice walk through of manually armoring files for the purpose of evading local host inspection of malicious binaries in “Bypassing Anti-Virus Scanners“, and there’s a good discussion of using Metasploit to construct evasive executable content over on offensive-security.com for comparisons sake.

The non-executable encrypted binaries are typically either droppers for newly compromised victims, or malicious updates for existing victim installations. In the case of newly compromised victims, the encrypted malicious binary will be requested by the attacker’s payload (e.g. shell script) and decrypted in memory before being installed on the victim. For existing victims, the encrypted binary will be requested by an already installed malware component, decrypted, and used to replace existing components of the malware installation.

There are other permutations to the theme as well. For example, the use of malicious Adobe Flash files that serve as the transport instigator for the malicious commands such as the imm32.dll dissected by Stop Malvertising, or the use of PNG files to propagate XOR’ed malware.

What does this mean for automated network-based anti-virus solutions (such as Norman, Trend Micro, etc.) and auto-signature-creator anti-virus products (such as FireEye, Palo Alto Networks’ Wildfire, etc.)? Since the intercepted binary is incapable of execution by itself and will never run under an emulated or virtualized behavioral system, it will not be automatically analyzed or classified as malicious. Only the targeted victim will uncover the unsavory payload.

Because the technique works so well and is so easy to do, we can expect the trend to continue. Organizations will capture more and more suspicious non-executable binaries going forward (often containing little or no hint as to what the file may potentially be).

What can be done? I believe a key perspective to dealing with this evasion technique is to better understand the suspicious nature of the file transfer. Consider it “contextual awareness”. Despite not being able to dissect and analyze the malicious binary directly, understanding the context of its transport will likely provide enough circumstantial evidence to arrive at a comfortable conclusion as to the maliciousness of the binary. For example:

• What is the reputation of the file download source? (E.g. a corporate entity, file sharing site, hacker forum)
• Is the source known to only host malware? (e.g. malicious files are encountered there frequently)
• Is the file requester already known to be infected? (e.g. the requester is a known botnet victim)
• Is there a cyclic pattern to the download file requests indicative of previous compromise? (e.g. an existing malware installation is requesting new updates)
• What is the nature of the source? (e.g. the hosting server is a known C&C server)
• What are the ports and protocols involved? (e.g. HTTP over a low (non-TCP80) port number)
• What are the parameters of the file? (e.g. the file is not readable by any commonly installed applications on the destination host, and is approximately the same size are regularly encountered malware samples)

At the end of the day, the attackers have the upper-hand when it comes to obfuscating malicious binaries. The increased application of XOR ciphers designed to evade network-based binary inspection tools and appliances makes sense, and is more than sufficient to thwart these protection technologies. In that case, it makes increasing sense to look for other network artifacts that can hint at the suspiciousness of the binaries makeup – and to use that contextual awareness to decide upon an automated defense reaction.

Dialing in the Malware

Despite several decades of anti-malware defense development, the pro-malware industry is still going strong. As I listen to presentations here at VB2011 in Barcelona this week covering many aspects of malware-based cyber-crime and the advances in detection being made, I'm reminded of a recent posting I made on the Damballa site concerning the success of malware. At the end of the day it costs the attacker practically nothing to generate new malware instances and, with a little investment in a QA process, they can guarantee evasion...

There’s often a lot of discussion about whether a piece of malware is advanced or not. To a large extent these discussions can be categorized as academic nitpicking because, at the end of the day, the malware’s sophistication only needs to be at the level for which it is required to perform – no more, no less. Perhaps the “advanced” malware label should more precisely be reattributed as “feature rich” instead.

Regardless of whether a piece of malware is designated advanced or run-of-the-mill, and despite all those layers of defense that users have been instructed to employ and keep up to date, even that ever-so-boring piece of yesteryear malware still manages to steal its victims banking information.

How is that possible?

I could get all technical and discuss factors such as polymorphism and armoring techniques, but the real answer as to why the malware manages to slip by all those defenses is because the bad guys behind the attack tested it prior to release and verified that it was already “undetectable” before it was shipped down to the victim’s computer. Those host-based defenses had no chance.

It’s worthwhile noting that generating “unique” malware is trivial. Armed with a stock-standard off-the-shelf DIY construction kit, it is possible to manually generate several hundred unique variants per hour. If the cyber-crook is halfway proficient with scripting they can generate a few thousand variants per hour. Now, if they were serious and stripped back the DIY kit and used something more than a $200 notebook, they could generate millions of unique variants per day. It sort of makes all those threat reports by anti-virus vendors that count the number of new malware detected each month or year rather mute. Any cyber-criminal willing to do so could effectively choose what the global number of new malware will be and simply make enough variants to reach that target. I wonder if any online betting agencies will offer worthwhile odds on a particular number being achieved. It may be worth the effort.

Armed with a bag of freshly minted malware, the cybercriminal then proceeds to test each sample against the protection products they’re likely to encounter on potential victim’s computers – throwing out any samples that get flagged as malware by the anti-virus products.

Using a popular malware DIY construction kit like Zeus (retailing for $4,000, or free pirated version via Torrent download networks), the probability of any sample being detected even at this early testing stage tends to be less than 10 percent. If the cybercriminal chooses to also employ a malware armoring tool that average detection rate will likely drop to 2 percent or less.

Obviously this kind of testing or, more precisely, Quality Assurance (QA) is a potentially costly and time-consuming exercise. Never fear though, there are a lot of entrepreneurs only too happy to support the cybercriminal ecosystem and offer this kind of testing as a commercial service.

Today there are literally dozens of online portals designed to automatically test new malware samples against the 40+ different commercially-available desktop anti-virus and protection suites – providing detailed reports of their detection status. For as little as $20 per month cybercriminals can upload batches of up to 10,000 new malware samples for automated testing, with the expectation that they’ll receive a thoroughly vetted batch of malware in return. These “undetectable” malware samples are guaranteed to evade those commercial protection products. As a premium subscription service model, for $50 per month, many QA providers will automatically fix any of the malware samples that were (unfortunately) detected and similarly guarantee their undetectability.

Armed with a batch of a few thousand fully-guaranteed malware samples that are destined to be deployed against their victims in a one-of-a-kind personalized manner, it should be of little surprise to anyone precisely why run-of-the-mill or feature-rich malware manages to infect and defraud their victims so easily.

In situ Automated Malware Analysis

Over the past few years there's been a growing trend for enterprise security teams to develop their own internal center of excellence for malware investigations. To help these folks along, there's been a bundle of technologies deployed at the network perimeter to act as super-charged anti-virus detection and reporting tools.

There's a problem though. These technologies not only tend to be more smoke and mirrors than usual, but are increasingly being evaded by the malware authors and expose the corporate enterprise to a new range of threats.

Earlier this week I released a new whitepaper on the topic - exposing the techniques being used by malware authors and botnet operators to enumerate and subvert these technologies. The paper is titled "Automated In-Network Malware Analysis".

I also blogged on the topic yesterday over on the Damballa site - here.

Cross-posting below...

Automated In-Network Malware Analysis

Someone once told me that the secret to a good security posture lies in the art of managing compromise. Unfortunately, given the way in which the threat landscape is developing, that “compromise” is constantly shifting further to the attacker’s advantage.

By now most security professionals are aware that the automated analysis of malware using heavily instrumented investigation platforms, virtualized instances of operating systems or honeypot infrastructures, are of rapidly diminishing value. Access to the tools that add sophisticated evasion capabilities to an everyday piece of malware and turn it into a fine honed one-of-a-kind infiltration package are simply a few hyperlinks away.

Embedding anti-detection functionality can be achieved through a couple of check-boxes, no longer requiring the attacker to have any technical understanding of the underlying evasion techniques.

Figures 1 & 2: Anti-detection evasion check-boxes found in a common Crypter tool for crafting malware (circa late 2008).

Throughout 2010 these “hacker assist” tools have been getting more sophisticated and adding considerably more functionality. Many of the tools available today don’t even bother to list all of their anti-detection capabilities because they have so many – and simply present the user with a single “enable anti’s” checkbox. In addition, new versions of their subscriber-funded tools come out at regular intervals – constantly tuning, modifying and guaranteeing their evasion capabilities.

Figure 3: Blackout AIO auto-spreader for adding worm capabilities and evasion technologies to any malware payload. Recommended retail price of $59 (circa August 2010).

Pressure for AV++

In response to the explosive growth in malware volumes and the onslaught of unique one-of-a-kind target malware that’s been “QA Tested” by their criminal authors prior to use in order to guarantee that there’s no desktop anti-virus detection, many organizations have embarked upon a quest for what can best be described as “AV++”.

AV++ is the concept behind some almost magical array of technologies that will capture and identify all the malware that slips past all the other existing layers of defense. Surprisingly, many organizations are now investing in heavily instrumented investigation platforms, virtualized instances of operating systems or honeypot infrastructures – all the things that are already know to have evasions and bypassing tools in circulation – despite the evidence. Has fear overcome common sense?

An area of more recent concern lies within the newest malware creator tool kits and detection methodologies. While many of the anti-detection technologies found in circulation over the last 3-4 years have matured at a steady pace, the recent investments in deploying automated malware analysis technologies within a targeted enterprise’s network have resulted in new innovations and opportunities for detection and evasion.

Just as the tactic of adding account lockout functionality to email accounts in order to prevent password bruteforcing created an entirely new threat (the ability to DoS the mail system by locking out everyone’s email account) so we see the development of new classes of threats in response to organizations that attempt to execute and analyze malware within their own organizations.

In a “damned if you do, and damned if you don’t” context, the addition of magical AV++ technologies being deployed within the borders of an enterprise network has opened the doors to new and enhanced evasion tactics.

To best understand the implications and dynamics of the new detection and evasion techniques being used by the criminals targeting businesses I’ve created a detailed white paper on the topic.