Threatology

Just a recap on some thinking covering threats and the folks who study them...

One of the key principles to understanding the threat is having the ability to monitor it. Within an enterprise environment security teams instrument the network in the form of protection technologies and stream alerts back to management consoles or aggregate multiple alert streams into centralized SIEM’s (or equivalent). Without sounding too depreciating, as difficult as it is to monitor threats within an enterprise, it’s nothing like monitoring Internet bound threats.

I know that plenty of organizations profess to monitoring threats as they propagate the Internet – often providing threat feeds to caring organizations (typically for a fee), or incorporating the processed threat data into tools and technologies behind the scene. The problem is that much of this monitoring is based upon point sampling and is heavily biased to the organizations geographic presence – and that’s before we get into the technical aspects of the monitoring systems in play.

In very basic terms you could think of it a bit like radio. Geographical distance and topology affect our ability to listen to a particular radio channel. The type of radio set and the frequency range it is capable of intercepting (e.g. AM, FM and shortwave) dictate the overall “richness” and quality of what we’re listening too. The mix of just these few simple variables greatly affects our globe-spanning listening pleasure. Even then, given a top-of-the-range radio placed on the highest mountain with the clearest “line of sight” in the world, reception capability is still limited and it probably isn’t going to interpret digital terrestrial TV signals.

Understanding the threats that plague the Internet and infiltrate the enterprise network is more than just instrumentation and regular mechanical sampling. To grasp the threat you need to understand the limitations of your threat visibility, constantly upgrade and extend the monitoring systems, and finally augment that visibility with data analysis systems capable of managing, correlating and analyzing huge volumes of streaming data. Even then there’s still a high degree of “art” to interpreting the nature of an Internet-spanning threat.

To my mind the methods, skills and acumen to understanding and tracking Internet threats are eerily similar to meteorology. Perhaps I’m biased – I specialized in Atmospheric Physics at University after all – but those skills and experiences I gained in meteorology can increasingly be applied to studying Internet threats. In particular, those of forecasting and dealing with abrupt changes of chaotic systems.

Let me propose the concept of Threatology – the study and analysis of Internet threats – and the Threatologists who study and understand it. Much of threatology is still an art – but that’s OK. Sure, there are millions of sensors scattered around the Internet (in the form of IDS sensors, AV sensors, web crawlers, spam traps, etc.) feeding data back to the threatologists for analysis – just as there are rain gauges, barometers, thermometers, anemometers and Doppler radar, etc. feeding data to meteorologists – but the real work goes into feeding the big modeling systems designed to digest the streaming data and forecasting what’ll happen next.

Today’s threatologists are still battling the intricacies and limitations of the sensors they’ve deployed (or have access to) around the Internet. Take for example the data feeds gained from the tens-of-millions of deployed desktop anti-virus products out there that phone-home with the latest things their subscribers have been infected with. An analogy would be the millions of amateur meteorologists submitting their latest rain gauge data back to the national meteorology department. Intricacies such as make and manufacturer of the gauge (affecting what’s actually being measured), physical location (e.g. under a tree or patio, or in the middle of a one-acre yard), geographical location (95% located in suburbia, 3% in farms, etc.), cleaning regime (the sensor’s full of autumn leaves or mud) and technical skill of the amateur operator – greatly limit the usefulness of this “invaluable” data source.

Over the last five decades meteorologists have employed ever-more advanced weather modeling systems that take in all this sensor data, apply historical trends and prediction routines, and manage to provide fairly accurate forecasts a few days out into the future. Threatologists meanwhile only have a couple of years playing with their own threatology modeling systems – and there’s a long way to go. There’s a lot to be learned from meteorology and the tools that have been developed thus far. Sure, there are many differences in the specifics of the data and nature of the threat – but the dynamic and chaotic characteristics (using the mathematical definition) of the threat are things that have already been “solved” in meteorology.

Welcome to the era of threatology and the professional threatologists.

Reinventing the Sandpit

Sometimes it feels that the IT security world loves innovation as much as it loves to reinvent the wheel – particularly when it comes to wrapping sheets of tin around a previously established security technology and labeling it as advancement. The last few weeks have been no exception in the run up to the annual RSA conference in San Francisco and the recent “innovation” going on in dealing with next generation malware (or AV+ as some folks refer to it) as multiple vendors launch new appliances to augment their product portfolio.

The latest security technology to undergo the transformation to tin is of course the automated analysis of suspicious binaries using various sandboxing techniques. For those of you not completely familiar with sandboxing, a sandbox is effectively a small self-contained version of an computer environment offering a minimal suite of services and capabilities. As the name applies, a sandbox serves as a safe environment for running various applications that may be destructive in other circumstances – yet can be rapidly built up and torn down as necessary.

In an enterprise security context, sandboxes are regularly encountered in two operational security implementations – safe browser sandboxes (designed to wrap around the web browser and protect the operating system from any maliciousness that may occur while the user is browsing the web and prevent attacks from contaminating the base operating system) and gateway binary introspection (i.e. the automatic duplication or interception of suspicious binary files which are then executed within a sandbox that mimics a common operating system configuration for the purpose of identifying and classifying any malicious binaries they come across).

The sandbox approach to malware identification is often referred to as signature-less and offers many advantages over classic anti-virus technologies, but they also suffer from their own unique set of limitations and inconveniences – most have to do with the way in which malware can discover that it is being executed within a sandboxed environment and thus act benignly, and limitations to the faithfulness with which the sandbox imitates a genuine targeted system (e.g. installed applications, application version, Internet connectivity, etc.). In general though, sandbox approaches to automated malware inspection and classification are more sophisticated and accurate than signature-based anti-virus approaches.

Despite what you may have heard in the flurry of newly released AV+ solutions, automated malware sandbox approaches aren’t precisely new – in fact they’ve had over a decade of operational and, dare I say it, “hostile” use. For example, Damballa has been operating sandboxing technology in the cloud pretty much since the inception of the company. We’ve chosen to use multiple sandbox technologies (along with bare-metal systems, manual analysis, etc.) to automatically process the mountains of new malware captured every day to mechanically extract their network characteristics, automatically cluster new malware families, and provide attribution to multiple criminal organizations.

Note that, from a product perspective, Damballa doesn’t run malware sandboxing technology from within a customer’s environment – there’s little to be gained from doing so, and the risks greatly outweigh the possible gain. Instead, the automated analysis of suspicious and vetted binaries using cloud-based malware enumeration technologies (which includes very sophisticated sandbox approaches amongst other specialized malware dissection engines) has proven to be more accurate, efficient and secure.

Over the years, many different malware analysis sandbox technologies have been developed. For example (not a complete list):

Norman Sandbox (2001) – In 2001 Norman presents its sandbox technology for the first time at the Virus Bulletin conference in Prague and offers a commercial sandbox version in 2003.

CWSandbox (2007) – Originally created by researchers from University of Mannheim. Available commercially by GFI Software (formerly Sunbelt Software) and free/academic use via http://mwanalysis.org

Sandboxie (2006)

Anubis (2006)

Joebox (2007)

Azure (2008)

BitBlaze (2008)

ThreatExpert (2008)

Ether (2009)

Each sandbox technology tends to be implemented in a different way – usually optimized and tuned for specific classes of malware (or aspects of malware) – and typically utilize either an emulator or virtual-machine approach. Emulators tend to be much smaller and faster in analyzing specific classes of malware, but suffer from their greatly limited range of supported (i.e. emulated) operating system API’s. Virtual machine approaches tend to be much more flexible, but are larger and slower.
Over the last decade, virtual machine (VM) based approaches have risen to the fore for automated sandbox approaches to malware investigation. The VM approach allows multiple guest OS images to be loaded simultaneously in order to run the malware within a self-contained and disposable environment. Interestingly enough, as a side note, did you know that the concept of running multiple, different operating systems on a single computer system harkens back to the 1970’s following research by IBM and the availability of the IBM VM/370 system? Talk about coming a full circle with “what’s old is new” again in security.

For sandboxing technologies, a combination of API hooking and/or API virtualization is often used to analyze and classify the malware. A term you will often see is “instruction tracing” – which refers to the observations recorded by the sandbox technology which are eventually used to derive the nature of the binary sample under investigation. This instruction tracing lies at the heart of sandbox-based approaches to automated malware analysis – and is the Achilles heel exploited by evasive malware.

Instruction tracing is typically implemented in one or more of the following ways:

User-mode agent – a software component is installed within the guest operating system and reports all user-based activity to the trace handler (think of this kind of like a keylogger).

Kernel-mode Patching – The kernel of the guest operating system is modified to accommodate tracing requirements (think of this kind of like a rootkit).

Virtual machine monitoring – The virtual machine is modified and instrumented itself to observe the activities of the guest operating system

System emulation – A hardware emulator is modified to hook appropriate memory, disk IO functions and peripherals (etc.) and report activities (think of this as a hall of mirrors approach). Emulation approaches are great for more difficult operating systems (e.g. Android, SCADA systems, etc.)

Unfortunately each of these sandboxing techniques exhibit system characteristics that can be detected by the malware being analyzed and, depending upon the nature of the malware, can be used programmatically to avoid detection.

Despite all these limitations, the sandbox approach to malware analysis has historically proven to be useful in analyzing the bulk of everyday malware.
In more recent years the techniques have become less reliable as malware developers have refined their sandbox detection methods and evolved more subtle evasion techniques. Many of these detection techniques are actually independent of the sandboxing technique being used – for example, the multitude of network-based discovery and evasion techniques discussed in my previous whitepaper “Automated In-Network Malware Analysis”.

The sandbox approach to automated malware identification and classification needs to be backed up with more advanced and complementary malware detection technologies. Organizations facing the brunt of targeted attacks and advanced persistent threats should make sure that they have access to sandbox analysis engines within their back office for the bulk processing of malware samples (running multiple configurations of the standard desktop OS builds (or gold images) deployed within the organization), and include a mix of bare-metal and honey-pot systems to handle the more insidious binary files. Even then, executing malware within your own organizations network or physical location is risky business for the reasons I covered in an earlier blog on the topic – you’re “damned if you do, and damned if you don’t”.
If you’re going to go to all the effort of installing and maintaining malware analysis sandboxes within your own organization, my advice is to look beyond the latest installment of tin-wrapped hype and take a closer look at the more established sandbox technologies out there. There’s plenty of choice – and many are free.

Post-emptive Detection

In the week before RSA I managed to pull together a blog on the Damballa site covering several of the problems with approaches that focus upon storing "all" the data and (eventually) data mining it in the quest for security alerts - aka Store it all in my barn. Here's what I had to say...

The other week I spoke at the DoD Cyber Crime Conference here in Atlanta and had a number of questions asked of me relating to the growing number of vendors offering “store it all” network monitoring appliances. That whole approach to network monitoring isn’t an area of security I’ve traditionally given much credence to – not because of the practical limitations of implementing it, nor the inefficiencies and latency of the techniques – but because it’s an inelegant approach to what I think amounts to an incorrectly asked question.

Obviously, given the high concentration of defense and law enforcement attendees that such a conference attracts, there’s an increased emphasis on products that aid evidence gathering and data forensics. The “store it all” angle effectively encompasses devices that passively monitor an organizations network traffic and store it all (every bit and PCAP) on a bunch of disks, tapes or network appliances so that, at sometime in the near future, should someone ever feel the need to or were compelled to, it would be conceptually possible to mine all the stored traffic and forensically unravel a particularly compelling event.

Sounds fantastic! The prospect of having this level of detailed forensic information handy – ready to be tapped at a moment’s notice – is likely verging on orgasmic for many of the “lean forward” incident response folks I’ve encountered over the years.

The “store it all” network monitoring approach is a pretty exhaustive answer to the question “How can I see what happened within my network if I missed it the first time?” But shouldn’t the question be more along the lines of “How can I detect the threat and stop it before the damage is done?”

A “store it all” approach to security is like the ultimate safeguard – no matter what happens, even if my 20 levels of defense-in-depth fail, or someone incorrectly configures system and network logging features (causing events to not be recorded), or if multiple layers of internal threat detection and response systems misbehave, I’d still have a colossal data dump that can eventually be mined. Believe me when I say that I can see some level of comfort in adopting that approach. But the inefficiencies of such a strategy make my eye twitch.

Let’s look at some scoping numbers for consideration. Imagine a medium-sized business with a couple-hundred of employees. Assume for the moment that all those folks, along with several dozen servers, are located at the same building. A typical desktop system has a 1Gbps network interface nowadays, and the networking “backbone” for a network of 250 devices is likely to have a low-end operating capacity of 10Gbps – but let’s assume that the network is only 50% utilized throughout the day. After a little number crunching, if you were to be capturing all that network activity and seeking to store it, you’d be amassing 54TB of data every day – so, perhaps you don’t want to capture everything after all?

How about reducing the scale of the problem and focusing upon just the data going to and from the Internet via a single egress point? Let’s assume that the organization only has a 10Mbps link to their ISP that’s averaging 75% utilization throughout the day. After a little number crunching, you’ll arrive at a wholesome 81GB of data per day. That’s much more manageable and, since a $50k “store it all” appliance will typically hold a couple of Terabytes of data without too many problems, you’d be able to retain a little over three weeks of network visibility.

How does this help your security though? Storing the data isn’t helping on a protection front (neither preemptive nor reactive), and it’s not going to help identify any additional threats you may have missed unless you’re also investing in the tools and human resources to sift through all the data.

To use an analogy, you’re a farmer and you’ve just invested in a colossal hay barn, you’ve acquired the equipment to harvest and bundle the hay, and you’re mowing fields that are capable of growing more hay than you could ever seek to perpetually store. Then someone informs you that one of their cows died because it swallowed a nail that probably came from your hay – so you’d better run through all those hay bales stored in your barn and search for any other nails that could kill someone else’s cow. The fact that the cow that died ate from a hay bale that’s no longer stored in your (full) barn is unfortunate I guess. But anyway, you’re in a reactive situation and you’ll remain in a reactive phase no matter how big your barn eventually becomes.

If you’ve got a suspicion that metal objects (nails, needles, coins, etc.) are likely to be bad juju, shouldn’t you be seeking them out before you’ve gone to all the work of filling your barn with hay bales? Wouldn’t it make more sense to perhaps use a magnet and detect those metal objects at the time you’re cutting the hay – before you’re putting it in a bale, and before you put those bales in your barn? Even if you had no forethought that metal objects in your hay could cause eventually a problem, do you persist with a strategy of periodically hunting for the classic “needle in a haystack” in your barn despite now knowing of the threat?

Getting back to the world of IT security and threat detection (and mitigation)… I’ve found that there are greater efficiencies in identifying threats as the network data is streaming by – rather than reactive post-event data-mining approaches.

I guess I’ll hear some folks ask “what about the stuff they might miss?” There are very few organizations that I can think of able to employ the skills and resources needed to analyze the “store it all” network traffic at a level even remotely comparable to what a security product vendor already includes in their commercial detection offerings – and those vendors are typically doing their analysis in a streaming fashion (and usually with something more sophisticated than magnets).

My advice to organizations looking at adopting “store it all” network monitoring appliances is the following:

1. If you already have all of your protection and detection bases completely covered, maybe deploying these appliances makes sense – provided you employ the dedicated security analysts and incident response folks to make use of the data.
2. Do you know what you’re trying to protect? “Store it all” approaches are designed to fill in the gaps of your other threat monitoring and detection systems. Is the threat going to be present at the network egress point, or will you need to store traffic from other (higher-volume) network segments? If so, be cognizant of how far back you can roll your eventual analysis.
3. If you’re in to hording data for the purpose of forensics and incident response, a more efficient and cost effective approach may be to turn on (and optimize) your logging capabilities. Host logging combined with network logging will yield a very rich data set (and will often be richer than simply storing all network traffic) which can be mined much more efficiently.
4. If host-based logging isn’t possible or is proving to be too unwieldy, and you find yourself having to maintain a high paranoia state throughout the organization, you may want to consider implementing a flow-based security approach and invest in a network anomaly detection system. That way you’ll get near real-time alerting for bespoke threat categories – rather than labor-intensive reactive data-mining.
5. If you have money to burn, buy the technology and begin storing all the PCAP data you can. Although I’d probably opt for a Ferrari purchase myself…