Signature-less Detection That Works

There was a period of time not long ago in which signature-based threat detection was cutting-edge. Antivirus, intrusion detection systems (IDS), data leakage prevention (DLP), content filtering and even anomaly detection systems (ADS) all continue to rely heavily upon static signatures. In recent years vendors have shied away from discussing their dependence on such signatures – instead extolling supplemental “non-signature-based” detection technologies.


In most cases these “non-signature-based” detection technologies appear to be largely a marketing ploy rather than some kind of new innovative solution; consider it a redefinition of what the dictionary would ordinarily define as a “signature” to overcome the inhibitions or bias of potential product purchasers. For example, if you define a signature as a single basic regular expression, then a threshold-based alerting policy could be considered a “non-signature” alternative.

I wanted to provide a walkthrough of the logic of a (real) non-signature-based detection technology. Perhaps “technology” isn’t quite the right word; rather let’s discuss it in terms of a confidence-based detection system. For example, for the purpose of edification, let’s consider a device reaching out to remote server – a destination that hasn’t been touched by the device ever before.


In the vast majority of cases a device will need to resolve a domain name in order to determine the remote IP address of the sever it is planning on connecting to. That DNS traffic contains a wealth of information if you can correlate with other external data sources and you know how to act upon it.


Signature-based detection systems could of course look up the IP address or domain name against their blacklist. If either of those data elements appear upon a list, then the traffic could be classified as malicious and blocked – if not, everything is probably “fine”. In the case of detection systems that utilize dynamic reputation, a scalar value representing the “suspiciousness” of the domain/IP and perhaps the class of threat would be returned (even if the domain and/or IP has never been seen before) and a threshold-based action would occur within the protection technology.


Supplemental to those signature-based detection approaches you could instead start dissecting the observation – compiling levels of suspiciousness and circumstantial evidence – and arriving at a conclusion of maliciousness.


Consider the following aspects of determining the nature of the interaction based purely from observing the DNS traffic related to the resolution of a single host lookup:

1. Armed with a database of passive DNS information observed over a period of years from around the world you can easily answer the questions:
   “Is this the first time anyone has ever looked up that domain name?”
   “Is this the first time that anyone has ever received that IP address for that domain name?”
   Knowing that a particular device (let’s assume that this entire example is being played out within a corporate network in the USA – say a Fortune 1000 company) is the first thing ever to resolve a domain name or be directed to a new IP address should raise a small degree of suspicion. This type of thing should be very infrequent.
   
2. Armed with geographic IP distribution information, you’ll know which country the returned IP address belongs to.
   “Is the destination IP address located somewhere unfriendly?”
   “Is this a country I want this kind of device connecting to?
   Knowing that the IP address belongs to a country with which the Fortune 1000 company doesn’t do any/much business with may be a little suspicious and worthy of more thorough study.
   
3. Armed with a list of ISP netblock associations, and knowledge of which of these netblocks are “residential subscribers”, can shed light on the nature of hosting environment of the destination server.
   “Is the destination address a residential IP address?”
   “Is the destination address a static or dynamic IP?”
   Knowing that a corporate device it trying to connect to a remote server located within a residential network should be suspicious to most network administrators – hinting at a number of threats or unwanted traffic.
   
4. Focusing a little on the domain name itself, characteristics of its registration may be used to determine levels of suspiciousness:
   “When was the domain name registered?”
   “Are there any features of the domain registrant details that cause concern?”
   Knowing that the domain name was registered 2 hours ago is significant. So too is knowing that details of the registrant match a large number of previously detected and categorized malicious domains.
   
5. Mining an exhaustive passive DNS database and correlating domain and IP information with historical threat information can yield even more information about the nature of the threat.
   “How many other domain names have pointed to the same IP address over the last 1/7/30 days?”
   “Are any of the domain names pointing at that IP address known to be related to a particular threat?”
   By associating an unknown or previously unclassified domain with domains with which historical information and threat attribution exists, enables the corporate entity to evaluate a “guilt by association” value.
   
6. Armed with information about the DNS servers that provide the authoritative answer for resolving the domain query provides further insight in to the nature of the destination.
   “Is the domain reliant upon free Dynamic DNS provisioning?”
   “What is the reputation of the authoritative DNS server?”
   “Which country is hosting the authoritative DNS server?”
   Dynamic DNS (DDNS) services are heavily abused by cybercriminals today – and are rarely used by large commercial entities. Understanding the location and past history of the authoritative DNS server (e.g. what proportion of domains hosted at the DNS server have previously been identified as malicious?) hints to the legitimacy of the destination IP address.
   
7. With visibility over live and historical domain name resolutions made to a particular authoritative DNS server, it becomes possible to glean information about the nature of the lookups and suspiciousness of the domain name:
   “How frequently is this domain name looked up?”
   “Which countries or organizations have also looked up this domain name?”
   “Who was the first to look up this domain name and get that response?”
   Knowing that a particular domain name has only been looked up by three US-based Fortune 500 companies in the last year is suspicious. Knowing that the same domain name points to an IP address in Iran and has never been looked up by anyone else in the world would be highly suspicious and indicate a level of targeted attack.
   
8. Then of course there’s the more obvious categorized information relating to the IP address:
   “Is the IP address a known sinkhole?”
   “Is the IP address associated with a commercial content delivery network?”
   “Is the IP address associated to a domain registrar’s holding page?”
   Knowing that the IP address is pointing to a sinkhole is a pretty obvious indicator that someone already thinks this particular domain name is malicious and associated with a threat. Meanwhile, knowing that the domain is pointing to a generic domain registration holding page could indicate that the domain has been taken down for being malicious, or is waiting to be staged by the criminals, etc.
   

There are more features that can be extracted from the successful resolution of a domain name than those listed above – but I’m sure you appreciate by now that a large amount of information can be obtained and associated with even a previously unknown domain name – and a certain degree of confidence can be obtained as to the suspiciousness (or maliciousness) of it.


For example, consider the following scenario:

1. A domain was looked up by a device, and that was the first time ever the domain has been looked up in the history of the Internet.
   
2. The authoritative DNS response came from a free DDNS provider in China.
   
3. The domain name points to a residential, DHCP assigned IP address, in Tehran.
   
4. There are 8 other domain names that have pointed to that particular IP address over the last 30 days.
   
5. Five of those domain names have been referenced within previously captured malware as C&C.
   
6. All 9 domain names (the original plus the uncovered 8 ) point to 3 IP addresses within the same subnet of the residential ISP in Tehran over the last 30 days – typical of a DHCP network assignment.
   
7. Only 6 of these domain names have ever been looked up, and all lookups of those domains over the last 90 days and have only been carried out by 11 major US-based pharmaceutical companies.
   
8. None of the domain names have ever been looked up from within Iran.
   
9. The first time each of the 6 domains were looked up, they were done from an IP address associated with a popular blackhat free-VPN service provider.
   

Obviously it shouldn’t take a genius to figure out that not much good is going to come from the device connecting to this particular remote host. All the evidence is circumstantial, but pulled together it becomes actionable intelligence. Most importantly though, all of this can be carried out using just a single DNS response (before any malware is downloaded, before any vulnerabilities are exploited, and before any user is socially engineered) – meaning that protection systems that can handle this level of non-signature-based threat determination engine can take preventative actions before the device has even begun to connect to the destination server.


When I think of non-signature-based detection systems, this is one approach that springs to mind. Such deterministic systems exist today – and they’re working very nicely thank you.

Botnet Prevention with DLP Technologies

Last week I was asked a couple of times how good Data Leakage Prevention (DLP) products are at protecting against botnets. Before I get started describing the pro's and con's of DLP in combating professional botnet operations, there are a few things I probably need to make clear - as it'll help add some perspective to the angle I'm coming from.

As you probably already know, I spent a fair amount of time developing and improving Intrusion Prevention System (IPS) technologies in my tenure with ISS (and then later, under IBM). During that time there were a number of market dynamics that required me to spend quite a bit of time reviewing, analyzing and evaluating the various DLP technologies - both at the host and then network levels. In general though, I was not impressed with the technology - and still aren't. From my perspective, DLP is a bit of a white elephant and is probably going to go down in the annuls of Security History in the chapter next to NAC. Don't get me wrong, as a concept DLP has its place, but in practice it fails to provide any compelling features that aren't (or can't be) delivered using other more common (and existing) enterprise security technologies.

Now, being a networking kind of guy, the thing I find most interesting about network-based DLP is the show and dance the various DLP vendors make about Deep Packet Inspection (DPI) - you'd almost think that they invented the technology and that it only to DLP. Lets get this straight from the beginning - DPI existed within IPS (and IDS) for 5+ years before even the first DLP companies became incorporated and, whats more, products like ISS' Proventia fully parse many hundreds of networking and content-level protocols - many times more than even the most mature dedicated DLP product out there.

So, if you're thinking DLP is a new and vital technology to roll out in your enterprise (particularly at the network layer), my advice would be to look to a top-tier IPS appliance instead because you'll find better protocol and content inspection coverage, and higher capabilities in inspecting traffic for critical data leakage. One day I'd love to see a head-to-head appliance review of the various vendors products detecting and defending against all the most common data leakage techniques/tactics.

DLP and Botnets
So, how useful is DLP in combating botnets? First of all, we obviously need some degree of clarification about "combating botnets". Lets break this down in to three separate botnet attack phases:

1. Preventing hosts from becoming botnet victims,
2. Detecting and stopping the leakage of confidential corporate information,
3. Cleanup and remediation of bot infected victims.
   

Preventing hosts from becoming botnet victims
In order to understand DLP capabilities in preventing hosts from becoming botnet victims (from a network perspective), we need to bear in mind the limitations of DPI and the most common mechanisms hosts succumb to being compromised and joining a botnet.

• Criminals leverage a broad spectrum of attack vectors in order to compromise their target - with the most common being spam/phishing emails that convince the user to infecting themselves, malicious drive-by-download sites the exploit vulnerabilities in the Web browser and removable media worming (e.g. USB devices). Unless the DLP solution is configured to watch inbound network traffic and scrutinize URL's (perhaps using a URL blacklist for checking against), the probability of detecting the malicious payloads is remote - and anti-spam and perimeter Web gateway technologies would be a much more effective solution here. IPS technologies would also excel in dealing with the exploits being used to compromise the Web browser vulnerabilities.
• Inspection of the HTTP/FTP/etc. downloads or email attachments is of course possible - but it will be a struggle to to identify the malicious intent of the binary files, but should best be dealt with using anti-virus technologies - particularly products with good behavioral analysis engines and, in a pinch, virtual/sandbox dynamic-analysis of malware.
  

I think you would be hard pressed to use DLP technologies as an effective tool for preventing hosts from becoming botnet victims.

Detecting and stopping the leakage of confidential corporate information
Detecting the information leakage from bot infected hosts should be an easy task - after all, that's supposed to be DLP's bread and butter. Unfortunately it's not quite as easy as it sounds.

• The signatures (or "fingerprints") DLP devices use are generally tuned to specific forms of structured data. For example, SSN's, credit card details and address details have a specific structure. As such, DLP solutions are generally good at spotting this kind data being transmitted across a network and leaking from the enterprise (just as IPS's can too). As such, DLP appliances can easily detect the "clear text" transport of these kinds of data.
• Unfortunately, botnet operators tend not to transport/extract confidential data past perimeter inspection/detection technologies in "clear text". Obviously, if the bot agent chooses to transport the data to a remote server over HTTPS, then all the traffic will be encrypted. But botnet operators don't even need to do that...
  
• Purchasing, managing and configuring Web server certificates for HTTPS can be tedious and can often result in "invalid certificate" alerts - which would in turn alert the user of any folks inspecting the system logs. As such, many botnet operators have decided to not use HTTPS - instead they extract their stolen data over un-encrypted HTTP, but they compress and encrypt the data they're stealing from on the victims machine before sending. I.e. the transport is unencrypted, by the file being transferred is itself encrypted and cannot be inspected by DLP (or any other DPI technology).
• Armed with a blacklist of known botnet Command and Control (CnC) channels or file drop-boxes, the DLP solution could keep watch over who the victim system is communicating with and block those - but there are already plenty of IP/Domain/URL blocking technologies already out there that are more efficient.
• It's important to understand that many professional botnet operators have moved away from stealing classic datasets (e.g. credit card details, SSN's, etc.), and towards more valuable datasets (e.g. software source code, CFO banking credentials, prototype designs) - which happen to be considerably more difficult to detect with DLP technologies (especially if the data is encrypted of course).
• DLP is limited to specific protocols and specific file/attachment types for inspection. To evade detection, the criminal botnet operator just needs to use an "unsupported" protocol/format.
  

Cleanup and remediation of bot infected victims
Well, I can't think of anything that DLP offers in this realm.

Clobbering Botnets with DLP
In general, DLP makes for a very poor anti-botnet technology. DLP is adequate enough detecting the simple stuff - e.g. a user sending an email with 10,000 credit card details - but is ill positioned to detect an automated bot agent obfuscating or encrypting a compressed file of corporate secrets.

In fact, as far as I'm concerned, I can't really see a reason for it existing as a separate security technology anyway. Existing IPS technologies and signatures include just about all of the data leakage detection features already.

That all said, DLP is probably adequate enough for detecting stupid user mistakes, but useless for combating professional criminals - whether they're botnet operators or insider threats.

If you can't protect it, you'd better be able to detect it!

The security trend over the last half-decade has been towards "protection" and we've seen technologies such as IDS morph in to IPS and network sniffing evolve in to DLP.

What I find amusing/worrying is that this laser focus on protection means that organizations have increasingly dropped the ball where it comes to threats that currently have no protection solution on the market. Basically, an attitude of "if I can't protect against it, then I don't want to know about it" has become prevalent within the security industry.

So, on that note, I found it refreshing to read the brief story over at Dark Reading How To Protect Your Organization From Malicious Insiders by Michael Davis. It's been a long standing mantra of mine that "If you can't protect it, you'd better be able to detect it!"

The 'Insider Threat' is one of the more insidious threats facing corporates today (especially in economic turmoil) and there really are so many ways for a knowledgeable employee to screw things up if they wanted to. I've had to do a mix of forensics and internal pentests within these areas in the past and it's always a potential playground of carnage.

But it's a little distressing to me that with the global sales push on DLP solutions many organizations have essentially thrown away their common sense. What I've observed is that enterprises that were initially deeply concerned about the potential of insider threat jumped heavily on to the DLP bandwagon and see this class of security technology as a way of over coming the threat. Then once they've deployed the DLP solution it's as if a mental box is ticked - "insider threat = solved" - and they move on to their next priority.

The problem is that DLP sucks as a protection system against the real insider threat and its rollout within an enterprise can be a substantial distraction to security & audit teams responsible for tracking the threat. Add to that the fact the executive support for further insider threat protection strategies quickly wanes after DLP has been rolled out -- "DLP = job done".

DLP will help identify (and block) many clear-text data leakage routes from an enterprise, however it'll do nothing against an insider that backdoors a server or Easter-eggs a DB to self destruct in a couple of weeks time - yet the mindset is that an investment has been made in DLP, and that since these kinds of insider threats can't be "protected" against, it's a problem too tough to solve (even though it may have been "solved" previously to the DLP solution - but that budget has now been used up - and DLP is supposed to reduce costs).

What ever happened to "detection"? As far as the insider threat goes, if you can't protect against it, you'd damn-well better ensure you can detect it. Failing that, I hope you're budgeting enough for post-attack disaster recovery and forensics.

Think of it this way. Say you're running a public library. You can bag check everyone that leaves the library to make sure they aren't stealing your books - and that's a wise precaution. But that doesn't mean you should skimp on the smoke detectors. The threat is "book loss" but there are clear differences between protection and detection strategies.