Predicting Crime Hotspots

There’s a new sheriff in town and he’s riding the horse of “predictive policing”. Back in July the Santa Cruz Police Department began deploying police officers to places where crime is likely to occur in the future – making use of new predictive modeling programs that are designed to provide daily forecasts of crime hotspots – thereby allowing the Department to preempt more serious crimes before they occurred. You can find a story describing how Santa Cruz is sending in the police before there’s a crime in The New York Times.

In essence, this is another physical-world application of machine learning and clustering technologies – applied to preempting a criminal problem. In the cyber-world we’ve been applying these techniques for a number of years with great success. In fact many of the most important advances in dealing with cybercrime revolve around the replacement of legacy IP reputation systems and domain filtering technologies with dynamic reputation systems – systems easily capable of scaling with both the threat and an ever-expanding Internet (e.g. IPv6).

Just last week Manos Antonakakis (a principal scientist at Damballa Labs) presented at the USENIX Security 2011 conference in San Francisco about a new generation of technology capable of identifying domain names being used for malicious purposes weeks, if not months, in advance of malware samples being intercepted, analyzed and “protected” against by legacy anti-virus approaches.

The patent-pending technology utilizes passive DNS observations within the upper DNS hierarchy, and the paper describing the first generation of research (and cybercrime proof-points) can be found in the paper “Detecting Malware Domains at the Upper DNS Hierarchy“. The system running here within Damballa Labs is affectionately known as “Kopis” and has proved its worth time and again preemptively identifying new botnets and cybercrime campaigns – keeping our Threat Analyst team busy with enumerating the real-world criminals behind the domain abuse.

The Kopis system extends many of the principles and research we learnt and formulated when developing the Notos technology – a next generation dynamic reputation system for DNS.

In several ways the Santa Cruz Police Department’s modeling systems approximates an early generation of such a dynamic reputation system – utilizing a mix of long term observations and historical information, combined with real-time crime updates, the output of which is a forecast capable of predicting hotspots for daily crime.

Damballa Labs utilizes Notos and its derivative output evolutions in a number of ways. For example, we’re able to take any observed DNS record (e.g. domain name and resolved IP address) and provide a real-time score of its reputation – even if this is the first time anyone on the Internet has ever tried to resolve that particular domain name. In practice this means that we can predict (with a scale of confidence) that connecting to a device utilizing that particular domain name (or IP) is malicious (or good) and the nature of the threat it represents – all done through passive means, and without having to have observed the maliciousness directly associated with the device anytime in the past.

Systems like Notos make use of big data (i.e. colossal volumes of historical and streaming data) gathered from a global array of sensors. The mix of historical observations and real-time data feeds means that prediction models can be dynamic enough to keep pace with truly agile threats (and threat operators) – and can yield new approaches in unveiling advanced and sophisticated threats. For example, a possible query could be “provide me a list of domain names that are pointing to residential DSL IP addresses within Villianstan, that have never been looked up by any hosts within the country of Villanstan, that have only been looked up by hosts located within Fortune-100 companies in the USA, and that the number of Fortune-100 companies doing so is less than 5 over the last 12 months.” The result of the query would be a (long) list of domain names that are very high contenders for APT victims, which then drives specialist counter-intelligence analysts and law enforcement to uncover the nature of the threat.

In the meantime I’ll be watching with keen interest the successes of the Santa Cruz Police Department and their new modeling programs. Here at Damballa we’ve had phenomenal success in using machine learning and advanced clustering techniques in unveiling and forecasting new threats.

Practical Packet Analysis Book Review

This week I had the opportunity to read Chris Sanders’ newly released book “Practical Packet Analysis” (second edition) – published by No Starch Press. While I’m not a frequent reader of technical computing books (they’re always a little too bulky for flights and carryon), I was looking for a book I could recommend and pass on to junior security consultants and threat analysts (as well as a few engineers).

Practical Packet Analysis proved to be a good read and I even managed to pick up a few tips on recent features within Wireshark that I’d not previously had a chance to experiment with; but am now looking forward to applying to real-world traffic.

While the book isn’t deeply technical (it’s not meant to be), it performs a very nice walk through of the practical aspects of performing network analysis and investigating packet captures. All too often in the past I’ve encountered network analysis books that either skim through the real-world problems an analyst or engineer will encounter, and rapidly descends in to the weeds of some obscure and contrived examples. Chris manages to navigate these waters is a clear and informative way. The practical analysis examples provide a breadth of understanding of not only the nuances and features of Wireshark, but also the common problems encountered by analysts tasked with troubleshooting their own networks. The sort of things they need to know asap if they’re going to be productive in a minimal amount of time

A chapter I particularly appreciated for its inclusion centered on how and where you should tap a network in order to perform analysis. You wouldn’t believe how many times that chapter alone could have prevented much wasted effort – if only folks had had access to it (and read it).

On the whole, I’d recommend this book to junior network analysts, software developers and newly minted MCSE/CISSP/etc. – folks that just need to roll up their sleeves and get started troubleshooting network (and security) problems. My copy of the book has already been passed on to a third pair of hands for reading and brushing up on the practical application of Wireshark. Great work Chris!

Not Endgames Again

With the Blackhat and Defcon conferences back to back, the melting pot that is Vegas has served its purpose in bringing together so many of the worlds leading security researchers, consultants, and opinions together. It’s been a tough slog through long days and longer nights, but it’s been so worth it.

While many of the presentations this time round may not have been worthy of previous years conferences, the true value of the event really lies in the hallway discussions and logistical movements between the vendor parties – trading invites for favors, and negotiations over beers pre- and post- party. I know that many folks would agree with me when I say that more business deals are secured and contacts negotiated at the Galleria bar of Caesars Palace than all the other event locations combined.

This year there was a lot of discussion in the Galleria Bar relating to exploit development (a big change from the past decades worth of vulnerability disclosure debate) – mostly due to the media attention garnered by the HB Gary Federal and Endgame Systems (Endgames) disclosures/revelations over recent months.

Each evening I’d inevitably get pulled into (new) discussions as folks I hardly know (or had only just been introduced to) tried to pump me for insider information about Endgames – somehow assuming I’m involved with that company. Let’s be clear – I have nothing to do with the Endgames business! It’s important that people understand that. The fact that both Endgames and Damballa (where I work) are in the same building in Atlanta is a reflection of shared Georgia Tech heritage and talent recruitment - not to mention $$$ per-square-foot office space rental costs – and is not a conspiracy seeking new enlightenment. And No, I don’t (and have never) worked for Endgames.

By way of preempting the next recycled batch of grilling from security nuts, weirdo’s and conspiracy theorists, here are some facts…

1. Back in 2005 I was enticed to leave NGS Software and London, and assume the role of Director of X-Force in Atlanta after Chris Rouland (the former Director of X-Force – and current CEO of Endgames) took on the role of CTO at Internet Security Systems, after Christopher Klaus (an ISS founder) vacated that particular position. As it happened, I took over responsibility for X-Force just after the Blackhat/Defcon events of 2005 – immediately after the Mike Lynn and Ciscogate (so that wasn’t anything to do with me). So, yes, Chris and I have both held the same titles at ISS and No, Ciscogate was not my fault.
2. While I was the Director of X-Force, the X-Force group (which consisted of R&D, threat research, detection/protection engineering teams and signature development teams, etc.) reported up through the VP of Engineering. The professional services teams (some of which were/are commonly tagged as “X-Force”) were regionally focused and organized, and so tended to report up through the regional sales organizations (i.e. not my responsibility). This is an important distinction, because ISS wasn’t unfamiliar with some of the professional services that would eventually transfer with the people that kicked off Endgames. So, No, I was not responsible for things labeled as “X-Force” within the professional services division in the US, and Yes, the professional services group(s) did have access at the time to all the latest vulnerabilities and 0-days uncovered by the X-Force R&D teams.
3. When IBM acquired ISS in October 2006, there were a lot of changes. ISS became IBM ISS and an “Office of the CTO” was established. Given integration challenges and the hope that a center of excellence could be created within IBM to bring together all the great security research done throughout IBM globally – and the hope that the derivative technologies would make it in to products within IBM ISS – the responsibilities for X-force were to be divided and I took on the role of Chief Security Strategist – reporting in to the new “Office of the CTO” – working with Chris Rouland and another founder of Endgames. So, Yes, Chris and I (and several of the eventual founders of EndGames) worked together for a couple of years in the same “office” for IBM ISS.
4. Some of the (PSS) services ISS had previously provided were not well suited to a company such as IBM and needed to be shutdown or were left to passively wilt while contract renewals wouldn’t be pursued. Several of these services (derivatives and extensions) are directly related to how Endgames came to exist – after the ISS professionals familiar with their delivery and a belief their commercial viability struck out from IBM ISS to create Endgames and satisfy those customer needs. I was never part of that side of the IBM ISS business. For one thing, I’m a foreigner and didn’t have the appropriate security clearances to get involved. For another, I find some aspects of that particular business model unsavory. So, No, I never had a hand in that side of ISS/IBM ISS’ business.
5. You can’t swing a stick in Atlanta without hitting an ex-ISSer. The number of security professionals that have passed through ISS over the last decade-and-a-half and gone on to establish and populate new security startups in Atlanta is amazing. This is why you’ll find so many ex-ISSer’s working at both Endgames and Damballa – and dozens of other security companies in the area! So, Yes, we all know and respect each other and tend to get on well. Endgames is on the same building one floor below Damballa, and there are several bars within spitting distance of our respective offices.
6. In the early days of Damballa (which is a startup that sprung out of Georgia Tech), Chris Rouland was on the companies Technical Advisory board. Damballa for it’s first few years of existence was focused on tracking botnets, enumerating the bot infected victims, and providing that insight as commercial intelligence feeds. Shortly after my joining Damballa in 2009, Damballa stopped providing commercial threat intelligence feeds and focused on appliance-based threat detection solutions. Chris Rouland elected to leave the Damballa Technical Advisory Board shortly before Endgames launched their IPTrust brand/service. So, Yes, in the past there was a relationship between Damballa and Chris Rouland (after all, he created the original X-Force and has been a thought leader in the security community for quite some time) – just not what some people have assumed.

There’s probably a whole bunch of additional questions that folks were battering me with this week in Vegas related to Endgames (and HB Gary Federal by proxy) that I couldn’t be bothered answering then, and I’m not going to bother answering now.

There is no commercial relationship between Endgames and Damballa. Damballa and Endgames are separate commercial entities – doing completely different things in totally different ways, with different objectives, customers and employees. The histories of several folks working at both companies are entwined with the history of ISS and IBM ISS – but that’s it.

And so on to the last conspiracy theory questions; No, I know of no cases of ISS selling vulnerabilities to any foreign entities. And, Yes, I’m still an opponent to middle-men financial models relating to the buying and selling 0-day vulnerabilities.