Network ADS - Playing at Botnet Detection

With botnets and more recently APT's plastered across the news, you'll struggle to find a security vendor that hasn't spent a furious couple of weeks repositioning their preemptive detection technologies as "anti-botnet" or "anti-APT".

Needless to say, digging a little deeper in to these technologies - beyond their cursory marketing spin - is probably a good idea, especially if your company executives are thrashing about looking to take any steps that'll keep them from being the next Google-like victim and making headline news.

Yesterday I pulled together my thoughts on the use of network-based Anomaly Detection Systems (ADS) in their capacity as botnet detection tools. In a nutshell, NADS is fine for dealing with those big and noisy Internet botnets that everyone writes about in the news, but not much chop against the types of botnets normally found successfully operating within enterprise networks.

My thoughts and analysis can be found on the Damballa blog - Detecting Botnets with Network ADS - and is also cross-posted below...

-------

Many businesses have already deployed Anomaly Detection Systems (ADS) within their enterprises whether they know it or not. Most ADS technologies can be discovered operating at the host-level – typically integrated in to the popular desktop antivirus suites of the major security vendors – where they can be often be found functioning in a hybrid detection mode somewhere between a personal firewall and a behavioral analysis engine.

Network-based ADS (NADS) on the other hand serve a different purpose within large enterprises. Their deployments are far fewer than host-based ADS, and are often used by security teams to detect major changes in network activity – typically analyzing and regulating traffic flow. Optimized to view high volumes of network traffic across an enterprise as fast as possible (in real-time in many cases), they rely upon specially crafted abstract protocols of the traffic – such as NetFlow, JFLow, NetStream, etc. – where content visibility is sacrificed for analysis speed.

Over the last 2 years NADS technologies have increasingly been positioned as having an anti-botnet capability – which has caused much confusion amongst those responsible for managing ADS deployments and those responsible for enterprise-wide security. NADS do in fact have some value as an enterprise-level botnet detection tool, but their capabilities are all too often misrepresented.

How capable are NADS in detecting and mitigating botnets? What are their strengths and weaknesses? The following is a summary of my observations and experiences (garnered over the last 5+ years) in using NADS as an enterprise security technology – warts and all.

Botnet detection & mitigation strengths:

1. A correctly configured and baselined NADS deployment is capable of detecting the high-volume attack output from certain classes of botnets. By identifying voluminous email sending (e.g. spam agent or spam proxy operation) or crafted port-specific traffic (e.g. DDoS agent operation) and tracking that back to specific hosts, the infected systems operating in this manner can often be classed as being a member of a botnet.
2. Many general-purpose Internet botnet malware make use of worming capabilities to propagate around enterprise networks by exploiting unpatched software flaws in vulnerable hosts. If a NADS solution has been correctly baselined, it can be relatively easy to spot the anomalous traffic this propagating threat creates – thereby alerting security teams to a new malware outbreak.
3. By configuring the NADS system to account for “normal working hours”, different detection thresholds can be utilized to aid in the detection of hosts that are actively communicating with botnet Command and Control (CnC) nodes. For example, if a host suddenly commences HTTP or IRC communication with an IP address located in Tibet at 3:00am in the morning – this will likely be very suspicious.
4. If an organization has suffered a sudden and large botnet infection, the constant polling of some botnet malware variants will quickly become apparent and will aid in the identification of those bot infected hosts.
5. If the NADS system supports the use of blacklists, a list of known botnet CnC’s can be used as a means of tracking the volume of data that has been sent or received by enterprise hosts that are part of the botnet. Study of this logging can help reveal the scope of a breach and the types of information the criminal botnet operator is targeting.

Botnet detection & mitigation weaknesses:

1. Very few botnets encountered within enterprises nowadays are noisy and spew copious volumes of spam or participate in devastating DDoS attacks. Botnet masters have largely moved on from this activity – and will only order bot infected hosts to operate this way if they’ve already exhausted all other value from the compromised hosts, or if they never bothered to figure out the infected hosts were actually located within an enterprise (in which case, if they had, they would have probably sold them to someone else for a good price). Therefore, basing detection of a botnet infection on copious volumes of attack traffic is either too late in the botnet lifecycle or was just bothersome (and not a security risk) to begin with.
2. By basing botnet detection upon the identification of outbound malicious traffic, the enterprise security team have failed to preempt the malicious operation of the botnet and are forced to deal with the voluminous output of an ongoing attack. Detection and mitigation of the botnet control instructions that instructed the attack to begin would have been more efficient and less damaging.
3. Baselining an enterprise NADS deployment – and keeping that baseline current – is almost impossible in the vast majority of businesses. New application deployments and software updates, along with roaming users and peer-to-peer communications, mean that enterprise network traffic is not as consistent and predictable as it was even only half-a-decade ago. As such, it becomes increasingly difficult to spot the worming traffic generated by botnets attempting to propagate around the network. This has become further complicated by the fact that the malware authors themselves have learned much from past attacks and have intentionally become more stealthy and deliberately slowed their propagation pace to avoid anomaly detection systems.
4. Botnet malware is more often than not designed to function only when the infected host is actually in operational use by its authorized user. As such, it is increasingly difficult to identify anomalous traffic from real traffic as the user goes about their regular Internet surfing.
5. Most botnet malware found within enterprise networks are proxy aware. This means that they borrow the users credentials and funnel all their CnC traffic through the corporate proxy servers – i.e. they do not use non-standard ports or protocols to navigate out to the Internet or between internal systems. The vast majority of botnet malware rely upon HTTP or HTTPS for communication.
6. Timing is everything for botnet operators nowadays. No sooner has a host been compromised through a drive-by-download vector or Trojan file download, that it connects back to its CnC ready to receive both a updated malware package and a new set of instructions. As such, unless the NADS solution is configured to react in real-time to the identification of a new botnet infection, the threat will have either moved on or already become more severe.
7. Many of the more commercially-minded botnet operators invest in fast-fluxing and domain-fluxing CnC technologies. The unrelenting changes in CnC IP addresses and hosts names can quickly overwhelm NADS systems.

So, in summary, I’d say that NADS does have a role to play in botnet detection – but only a very minor one, and even that’s diminishing all the time. NADS deployments make for capable enterprise-wide network health monitoring systems, but have faltered against advanced threats like botnets and even more stealthy threats such as Advanced Persistent Threats (APT’s). I’d liken NADS to a school nurse – constantly overseeing the health of the entire student population and dealing with the odd knee scrape and cut lip – but not trained or equipped to deal with major head trauma or the results of a shooting spree.