Corelight Blog

We make the world's networks safer.

Databricks + Corelight – A powerful combination for cybersecurity, incident response and threat hunting — July 17, 2018

Databricks + Corelight – A powerful combination for cybersecurity, incident response and threat hunting

Screen Shot 2018-07-16 at 2.23.15 PM

By Alan Saldich, CMO, Corelight and Brian Dirking, Sr. Director Partner Marketing, Databricks

Incident response, threat hunting and cybersecurity in general relies on great data. Just like the rest of the world where virtually everything these days is data-driven, from self-driving cars to personalized medicine, effective security strategies also need to be data-driven.

Whatever security solution, service or business process you implement, it probably relies on data from just four sources: logs, networks, hosts and third party intelligence feeds. Some or all of that data is fed into an analytics stack like Apache Spark™ where advanced analytics, artificial intelligence, machine learning and other techniques can be applied.

When it comes to the data about network traffic though, most organizations are stuck between “not enough data” and “way too much.” The former is normally NetFlow data that provides basic information about network traffic, source / destination IPs, time and date, bytes sent / received and few other important (but sparse) pieces of information. The latter is typically PCAP (packet capture) where every bit on the wire is stored. Since big networks carry a tremendous amount of data, storing it all is non-trivial and can become expensive and cumbersome.

The Alternative

Screen Shot 2018-07-16 at 2.07.56 PMSo what’s the alternative? Well for over 20 years, an open source project called Bro has been used in production at some of the world’s largest organizations and biggest networks, namely government agencies, research universities and very large / web-scale companies.

Bro is a network monitoring framework that ingests a copy of all traffic on a network, parses and analyzes it in real time using its event processing engines and scripts, and then outputs data (“Bro logs”) to some external analytics system like Spark. There are dozens of Bro logs for most common protocols like SMTP, SSL, SMB, DHCP, DNS and many others. Each Bro log contains 10 to 40 fields describing that part of the network traffic.

img bro logs id tracking.png

Furthermore, Bro logs include key pivot points in the data that allow incident responders to follow their instincts or observations by taking advantage of unique identifiers for critical aspects of network traffic: all connections (Connection UID) and files (FUID) are uniquely identified. Along with consistent and precise timestamps across all logs, Bro gives incident responders access to everything they’ll need to resolve most security incidents quickly without having to resort to PCAP except in rare instances.

Corelight was founded by the creators of Bro to deliver network security solutions built on this powerful and widely used framework. Since Corelight does not provide an analytics application, we are excited to work with leading companies like Databricks to combine the power of Bro data with the intelligence of Apache Spark and all of its analytic capability.

The challenge of managing threats in a big data world

Staying abreast of the latest threat isn’t the only challenge. The increasing volume and complexity of threats require security teams to capture and mine mountains of data in order to avoid a breach. Yet, the Security Information and Event Management (SIEM) and threat detection tools they’ve come to rely on were not built with big data in mind resulting in a number of challenges:

  • Inability to scale cost efficiently
    Companies deploy logging and monitoring devices across their networks, end-user devices and production machines to help detect suspicious behavior. These tools produce petabytes of log data that need to be contextualized and analyzed in real-time. Processing petabytes of data takes significant computing power. Unfortunately, most SIEM tools were built for on-premises environments requiring significant build-outs to meet processing demands. Additionally, most SIEM tools charge customers per GB of data ingested. This makes scaling threat detection tools for large volumes of data incredibly cost-prohibitive.
  • Inability to conduct historic reviews in real-time
    Identifying a cybersecurity breach as soon as it happens is critical to minimizing data theft, damages, and creation of backlogs. As soon as an event occurs, security analysts need to conduct deep historic analyses to fully investigate the validity and breadth of an attack. Without a means to efficiently scale existing tools most security teams only have access to a few weeks of historical data. This limits the ability of security teams to identify attacks over long time horizons or conduct forensic reviews in real-time.
  • Abundance of false positives
    Another common challenge is the high volume of false positives produced by SIEM tools. The massive amounts of data captured in OS logs, cloud infrastructure logs, intrusion detection systems and other monitoring devices produce events that in isolation or in connection with other events may signify a compromised network. Most events need further investigation to determine if the threat is legitimate. Relying on individuals to review hundreds of alerts including a large number of false positives results in alert fatigue. Eventually, overwhelmed security teams disregard or overlook events that are in actuality legitimate threats.

In order to effectively detect and remediate threats in today’s environment, security teams need to find a better way to process and correlate massive amounts of real-time and historical data, detect patterns that exist outside pre-defined rules and reduce the number of false positives.

Enhancing threat detection with scalable analytics and AI

Screen Shot 2018-07-16 at 2.25.24 PMDatabricks offers security teams a new set of tools to combat the growing challenges of big data and sophisticated threats. Where existing tools fall short, the Databricks Unified Analytics Platform fills the void with a platform for data scientists and cybersecurity analysts to easily build, scale, and deploy real-time analytics and machine learning models in minutes, leading to better detection and remediation.

Databricks complements existing threat detection efforts with the following capabilities:

  • Full enterprise visibility
    Native to the cloud and built on Apache Spark by the original creators of Apache Spark, Databricks is optimized to process large volumes of streaming and historical data for real-time threat analysis and review. Security teams can query petabytes of historical data stretching months or years into the past, making it possible to profile long-term threats and conduct deep forensic reviews to uncover backdoors left behind by hackers. Security teams can also integrate all types of enterprise data – SIEM logs, cloud logs, system security logs, threat feeds, etc – for a more complete view of the threat environment.
  • Proactive threat analytics
    Databricks enables security teams to build predictive threat intelligence with a powerful, easy-to-use platform for developing AI and machine learning models. Data scientists can build machine learning models that better score alerts from SIEM tools reducing reviewer fatigue caused by too many false positives. Data scientists can also use Databricks to build machine learning models that detect anomalous behaviors that exist outside pre-defined rules and known threat patterns.
  • Collaborative investigations
    Interactive notebooks and dashboards enable data scientists, analysts and security teams to collaborate in real-time. Multiple users can run queries, share visualizations and make comments within the same workspace to keep investigations moving forward without interruption.
  • Cost efficient scale
    The Databricks platform is fully managed in the cloud with cost-efficient pricing designed for big data processing. Security teams don’t need to absorb the costly burden of building and maintaining a homegrown cybersecurity analytics platform or paying per GB of data ingested and retained.

How a Fortune 100 company uses Databricks and advanced cybersecurity analytics to combat threats

A leading technology company employs a large cybersecurity operations center to monitor, analyze and investigate trillions of threat signals each day. Data flows in from a diverse set of sources including intrusion detection systems, network infrastructure and server logs, application logs and more, totaling petabytes in size.

When a suspicious event is identified, threat response teams need to run queries in real-time against large historical datasets to verify the extent and validity of a potential breach. To keep pace with the threat environment the team needed a solution capable of:

  • Large data volumes at low latency: Analyze billions of records within seconds
  • Correct and consistent data: Partial and failed writes cannot show up in user queries
  • Fast, flexible queries on current and historical data: Security analysts need to explore petabytes of data with multiple languages (e.g. Python, SQL)

As an example of how customers are using advanced cybersecurity analytics, check out this recent video from Apple.

For another more general explanation of how Corelight and Databricks can be deployed together, Databricks produced this video that includes an explanation of how they ingest Bro logs as part of their security solution.

Screen Shot 2018-07-16 at 2.17.44 PM

The Challenge

It took a team of twenty engineers over six months to build their legacy architecture that consisted of various data lakes, data warehouses, and ETL tools to try to meet these requirements. Even then, the team was only able to store two weeks of data in its data warehouses due to cost, limiting its ability to look backward in time. Furthermore, the data warehouses chosen were not able to run machine learning.

Screen Shot 2018-07-16 at 10.20.15 AM.png

The Solution

Using the Databricks Unified Analytics platform the company was able to put their new architecture into production in just two weeks with a team of five engineers.

Their new architecture is simple and performant. End-to-end latency is low (seconds to minutes) and the threat response team saw up to 100x query speed improvements over open source Apache Spark on Parquet. Moreover, using Databricks, the team is now able to run interactive queries on all its historical data — not just two weeks worth — making it possible to better detect threats over longer time horizons and conduct deep forensic reviews. They also gain the ability to leverage Apache Spark for machine learning and advanced analytics.

Screen Shot 2018-07-16 at 10.20.21 AM

Final Thoughts

As cybercriminals continue to evolve their techniques, so do cybersecurity teams need to evolve how they detect and prevent threats. Comprehensive network traffic extracted by Bro is the highest quality data available to incident responders and threat hunters, and an essential ingredient to the Databricks analytics platform for cybersecurity. Big data analytics and AI offer a new hope for organizations looking to improve their security posture, but choosing the right platform is critical to success.

Download our Cybersecurity Analytics Solution Brief or watch the replay of our recent webinar “Enhancing Threat Detection with Big Data and AI” to learn how Databricks can enhance your security posture, including ingestion of Bro logs for network traffic monitoring.

Screen Shot 2018-07-16 at 5.13.05 PMScreen Shot 2018-07-16 at 2.17.44 PM

If you’re not familiar at all with Bro, watch Corelight’s two-minute video


How Bro logs gave one company better DNS traffic visibility than their DNS servers — June 11, 2018

How Bro logs gave one company better DNS traffic visibility than their DNS servers

By Howard Samuels, Director of Sales Engineering at Corelight

Bro provides enriched network visibility for top organizations around the world, and there are many use cases for Bro logs.   The security field uses Bro data for incident response and cyber threat hunting. But Bro log use cases don’t always have to involve finding bad actors, identifying breaches or attack blueprints.  The clean, structured, and enriched data from Bro can also be used to simply provide necessary protocol information not otherwise easily obtained from servers.

In this example, we show how a company can obtain DNS visibility and extend the lifespan of a large production deployment of DNS servers via a few Corelight sensors that generate Bro DNS logs. This use case features a utility company with over 50 DNS servers worldwide.  A federal governing body told them that there were DNS lookups going to known cyber threat sites originating from their network and therefore they weren’t sufficiently self-governing their DNS activity. Given the importance and focus on safeguarding utility companies from cyber attacks, getting in front of this DNS issue was of paramount importance.  

The company had limited DNS network visibility because their DNS servers could not effectively log activity.   The DNS logging service on their servers didn’t give enough functional information – and therefore visibility – to identify these malicious communications.  Moreover, the data their DNS servers did provide was a flat file with too much noise and the logging mechanism also had a negative impact on the DNS servers’ overall performance.    

They considered upgrading all their DNS servers, but given the cost they discarded this option.  They determined that a more comprehensive, faster, and cost-effective solution was to deploy Corelight sensors in their main data centers to obtain enriched Bro DNS logs.  With Corelight and Bro they could easily capture both DNS requests and answers to queries and quickly stream them to a SIEM. The benefit was immediately clear when an analyst who had previously tried to identify non-authoritative DNS lookup records was able to easily achieve this with the logs provided by Corelight sensors.

The utility company used packet broker to distribute raw traffic to the Corelight sensors which then do the heavy lifting of extracting the DNS information into a log or data stream for export to their SIEM.  The solution architecture is simply network taps sending traffic into a packet broker. Behind the broker is a Corelight sensor. The packet broker filters are configured to send only DNS traffic to the Corelight Sensor.  The enriched logs are spooled from the Corelight sensor to a datastore and ultimately consumed in a SIEM. Problem solved, customer happy, money saved and the technology team are now heroes. Their security team is now looking at network logs and will look at expanding their use to the SOC.  

Thinking more broadly beyond DNS, consider other crucial network services and the struggle to keep accurate time stamped logs and your ability to get enriched data from these services?  What if network service logging for DHCP, Kerberos, or Radius is set to OFF, FATAL, WARN or ERROR – i.e. not nearly enough data? Or, ALL, DEBUG or TRACE – too much noise? What if INFO misses the data you need?  What if there is only one logging service level and therefore not configurable? Bro provides better network data that can spare operational cycles and extend the life of network services, thereby saving money and just as importantly reducing operational headaches.

Another cool thing about Bro: SMB analysis! — May 29, 2018

Another cool thing about Bro: SMB analysis!

By James Schweitzer, Federal Solution Engineer at Corelight

If you’re reading this blog, you probably know that Bro can uncover indicators of compromise and discover adversary lateral movement by monitoring east-west traffic within the enterprise. But you may not know about one of the best sources of data for this purpose, the Bro server message block (SMB) logs.  Bro’s SMB protocol analyzer has undergone several iterations, and it is now a built-in feature that many Bro users might have overlooked. If you are running Bro 2.5, all that is needed is to manually load the SMB policy.

corelight-youtube-smb-schweitzer (1)

SMB is used for many purposes. Most users of Windows networks rely on SMB every day when accessing files on network drives, and network administrators use the same protocol when they perform remote administration. Unfortunately the adversary, whether script kiddies or nation-state actors, also uses SMB! By the way, do you know whether SMBv1 is running on your network… and how can you be sure?

The video that accompanies this blog provides an introduction to the power of Corelight’s advanced filtering and the content contained in Bro’s SMB logs to monitor SMB usage for remote scheduled tasks and file access. If you use Bro to monitor SMB, please share tips here so others can benefit – if you don’t use Bro, would you like to learn how it transforms raw network traffic into comprehensive, organized logs? If you are interested in learning more detail about Bro’s ability to detect malicious activity hidden in SMB, this SANS paper is a great place to start.

I hope you enjoy this short introductory video. Good luck and good hunting!

How we decide what Bro capabilities to include in our Sensor — April 5, 2018

How we decide what Bro capabilities to include in our Sensor

By Seth Hall, Co-Founder & Chief Evangelist at Corelight

We started Corelight to bring the power of Bro network monitoring to an audience that is interested in security, stability, and long-term sustainability. Even though we created and built Bro over the last 20 years, when we developed our commercial product we made some design decisions that make running the Corelight Sensor slightly different from running open-source Bro… changes to improve performance, security, and deployability.  Here we’d like to explain the rationale behind a few of these decisions.

We take a diligent approach to new features on our platform because Bro’s biggest benefit of programmability can also be a liability.  From the beginning, we wanted to expose the full richness of Bro analysis to our customers, but we also chose to limit some functionality at first, because they did not meet the high quality and security standards required by an enterprise-class commercial product.  As we continue along our development roadmap, we’re revisiting these items to find new and better ways to implement these features in our sensors and potentially contribute back to open-source Bro.

Let’s go through a few of these differences, along with some of the extra features that you get with the Corelight sensor that distinguish it from open source Bro.


Users of open source Bro know how to run Bro with broctl (BroControl) because it helps with managing everything from a single process to large multi-system or even multi-location clusters. This works very well in the open-source community where users are familiar with running software at the command line, but we felt that we shouldn’t offer this same interface for our commercial customers, because with it comes a lot of complexity. Our appliance is fully API-driven, and we currently expose that API through two mechanisms: directly from our command-line client (, or over SSH through our terminal-based GUI. Broctl gives our open-source users a great deal of capability, but we took the approach of wrapping most of that functionality in other mechanisms to simplify management for our customers. We are currently exploring how to take what we’ve learned from offering this experience to our customers back into the open-source world because a core mission of the company is to continue improving Bro.

Intelligence (Intel) framework

The Intel framework is used by Bro to read in indicators of compromise (IOCs) from external sources at runtime and do matching deep into network traffic. For example, if you load in an email address, Bro will watch for that email address in, for example, the “emailAddress” object identifier of X.509 certificates used in SSL/TLS session establishment (as well as a number of other places).

The problem is that the way people typically load intelligence data into Bro is with a format that I specified in 2013 when I created the Intel framework which has literal tabs in the file separating fields. It’s like CSV, but using an invisible character! This causes trouble for people who are hand-creating these files, as you can imagine. We left the intelligence integration out of our Bro appliance from the beginning because of the number of problems we’ve seen people encounter with formatting the files correctly. We also noticed that many of our enterprise customers were instead applying their IOCs to the logs in their downstream data analysis system.

Eventually, we will want to cleanly integrate with threat intelligence management platforms where data can be pulled in more quickly and there are no concerns with file format accuracy. In the interim, we are now working on integrating the Intel framework into our appliance with some guardrails. Our platform API will enable you to load in the “normal” Bro intelligence files, but our system will extensively sanity-check them so you get a response immediately from the API if you made a mistake. This should provide a nice mix of people that want to sit as close to the metal of Bro as possible, while still getting the enterprise approach to stability that we press so hard for at Corelight.

Bro scripts and Sandboxing

One of the absolute best features of Bro is the scripting language. It’s the true differentiator between it and many other network monitoring systems. Users can write their own scripts which add new chunks of functionality to Bro, even as far as creating logs that reflect their own environment and unique challenges. For an example of this, take a look at the script Salesforce published for fingerprinting SSL/TLS clients ( The team at Salesforce was driven by their internal needs to understand SSL/TLS usage on their network, and that need resulted in a script that any Bro users can now load.

The problem at Corelight was that the intense customizability provided by Bro could be a liability for our customers because scripts can have adverse effects on the stability and correct behavior of the software. Due to that, our stance on custom scripts was to initially not make them available, but while keeping an eye toward eventually doing so, in a safe and robust manner. Ultimately, after a lot of discussion, we designed a sandbox that the scripts run inside so that you can feel confident that the scripts you are loading on our appliance are safe. The sandbox restricts a number of functions from being used, along with some behaviors that we know to be non-performant. For example, we bar the new_packet event from use due to its potential for causing major performance issues. This trades a small reduction in capability for removing a feature fraught with side effects.

The sandbox took quite a bit longer to create than we initially expected but we felt that rushing it out would fail to live up to the level of quality we strive to always provide. One thing I’ve always found interesting is that sometimes through restrictions, the most creative results arrive. By creating an environment that doesn’t give you every possibility in the world we’ve created a bounding box where creativity and problem solving can thrive.

Specialized Hardware

Bro doesn’t natively support any specialized hardware. Typically, when Bro users have something like a specialized NIC (network card) they take advantage of it through libpcap wrappers that hide the NIC complexity behind an API that Bro does natively support. This tends to work fine, but it doesn’t provide the ability to take advantage of the full specialized capabilities that the NIC provides. At Corelight, we viewed this as our chance to work with a vendor to really push the state of the art in terms of integration. We formed a great partnership with Accolade Technology where we’ve pushed each other to develop ideas for offloading processing and creatively using their NICs in previously unexpected ways.

Our customers benefit from the tight integration with the specialized NIC hardware in two ways.  First, there are performance benefits due to our use of specialized capabilities on the card such as high performance injection of packets into system memory.  Secondly, our customers get to experience the benefit of this non-commodity NIC without having to spend the time understanding it and integrating it into their own deployment.


Bro’s flexibility has given us the ability to create a network monitoring appliance that is truly ready to use “out of the box” but continues to have a number of doors that can be opened for further exploration and modification.  As we move into the future and continue developing Bro and the Corelight Sensor, we will continue our deliberative approach to providing production-ready features with the full programmability of Bro.

Announcing The New Corelight for Splunk App — March 29, 2018

Announcing The New Corelight for Splunk App

We’re proud to announce the Corelight for Splunk app is available!  Using the new app (and its associated Technology Add-on (TA)), you can now monitor the health and performance of Corelight Sensors in Splunk and explore the rich data Bro provides through a series of dashboards.

pasted image 0


The Corelight for Splunk App, associated TA, and Q&A page are all on Splunkbase now.

If you’re using open-source Bro and you want to use Corelight’s app, you need to send your Bro logs to Splunk in a streaming format using JSON. To do so, install the json-streaming-logs Bro package using the Bro Package Manager, also directly available via GitHub.

In the next few months, we’ll be publishing more information about the app, including an FAQ and a longer blog post dedicated to highlighting its functionality and benefits.  

In the meantime, let us know if you have any questions or concerns installing or using the new app:

The Corelight Team

Joining a New Company Selling 20 year-old Software — March 26, 2018

Joining a New Company Selling 20 year-old Software

By Brian Dye, Chief Product Officer at Corelight

I’ve enjoyed meeting many companies and leaders in the Bay Area over the past few months. The best surprise I had in doing so was with Corelight (where I recently joined as their chief product officer). Despite many years in security, when they proudly proclaimed “we’re bringing an easier, faster, commercially supported version of Bro to the market” I had to respond with a less than glorious “OK … but what is Bro?”

To find out, the first people I talked to were top incident responders … the ones with battle scars, the SANS trainers, the folks you call when “it” hits the fan. This was my first surprise:  The immediate answer was “of course I know Bro. I use it all the time, even to teach SANS security incident investigations classes.” It turns out, Bro creates a uniquely useful set of insights out of network data; insights that are far richer than NetFlow but far more concise and searchable than a full PCAP. Bro is the “Goldilocks” insight level for security investigations.

Next, I talked to CISOs I especially respected. They knew about Bro too, for how valuable the data was and for what it helped their teams do. These CISOs knew that Bro was taking off as part of the industry focus on improving SOC effectiveness and better arming their investigators. What they didn’t like was that open source Bro was a complex “roll your own” solution and deploying it required expert-level UNIX people, so getting access to the valuable data Bro provides meant taking their (scarce!) talent and putting them on infrastructure management. Those same people were often the very incident responders and threat hunters who should be focusing on defending networks, not installing technology. That is where Corelight comes in: the Corelight Sensor radically simplifies the deployment and operation experience, resulting in a lower hardware and operational cost. The number and caliber of customers signing on with Corelight as we speak is proof positive of that value.

After all that, there was a lingering question in my mind… if Bro is so awesome, why isn’t everyone using it? (While Bro is well known by some, its adoption by enterprises has lagged behind government agencies, universities and web-scale companies). The answer is actually pretty simple: Bro’s capabilities, while critical for 20 years at national labs, intelligence agencies and other organizations with existential threats from determined adversaries, were not needed by typical enterprises in the 1990s and even 2000s. Bro was created before the cloud, mobile, SAAS and high bandwidth links were in common use by “normal” companies. And most companies didn’t have SOCs or the level of security + technical expertise to get Bro working. Obviously, all that has changed — the problems faced by enterprises have grown into the long-extant capabilities of Bro.

My last question was “where can we go from here?” One of the clearest long-term trends in cyber is that better data enables better security. As a result, at Corelight there is a wide range of opportunities to both give organizations new insights and solve existing problems in far better ways. One example, driven by a mindset shift: many organizations wrestle to make their data analysis and investigation as seamless and effective as possible … but they treat the incoming data as immutable.  Corelight proves that it isn’t, and by improving both the quality and structure of that data the entire investigation stack gets better – and we can continue enriching the quality of that data. It reminds me of the old BASF ads … “we don’t make the things you buy, we make the things you buy better.”

This, of course, is just the beginning. I’m excited to join the Corelight team, and can’t wait to show you what we can do for you. Better security starts with better data.

Runtime Options: the Bro Configuration Framework — February 13, 2018

Runtime Options: the Bro Configuration Framework

By Johanna Amann, Senior Engineer at Corelight

If you are familiar with Bro scripts you have probably encountered redefs, which allow you to change a number of Bro settings. One commonly used redef is Site::local_nets, which lists the networks that Bro considers local.

As the name redef implies, redefs allow the re-definition of already defined constants in Bro. This is often done in local.bro (but can be done in any loaded script-file). To modify Site::local_net, you can use code similar to this:

redef Site::local_nets = +={};

A disadvantage of redefs is that these redefinitions can only be performed when Bro first starts. Afterwards, Site::local_nets is just a normal constant and can no longer be modified.

However, it is clearly desirable to be able to change many of the configuration options that Bro offers at runtime. Having to restart Bro causes Bro to lose all connection state and knowledge that it accumulated. To solve this problem, Corelight has created the Bro configuration framework, which allows changing configuration options at runtime. We designed the configuration framework in a way that is easy to use and unobtrusive, while also giving great power and flexibility when needed. Using the configuration framework in your script only requires minimal changes. To declare a configuration option, you just prefix it with the newly introduced option keyword:

module OurModule;

export {
    option known_networks: set[subnet] = {};
    option enable_feature: bool = F;
    option system_name: string = "testsystem";

Options lie in between variables and constants. Like constants, options cannot be assigned to at runtime; trying to manipulate an option will result in an error. However, there are special calls that can be used to modify options at runtime; these are also used internally by the scripts that power the configuration framework; we discuss this further below.

Given those three options defined above, we just need to tell Bro where to find the configuration file. Simply add something akin to this to local.bro:

redef Config::config_files += { "/path/to/config.dat" };

config.dat contains a mapping between the option names and their values:

OurModule::enable_feature  T
OurModule::system_name  prod-1

Now the options are updated automatically each time that config.dat is changed. Additionally, a new log file, config.log contains information about the configuration changes that occurred during runtime.

Behind the scenes, the config framework uses the Bro input framework with a new custom reader. Users familiar with the Bro input framework might be aware that the input framework is usually very strict about the syntax that it requires. This is not true for configuration files: the files need no header lines and either tabs or spaces are accepted as separators.

For more advanced use-cases, it is possible to be notified each time an option changes:

function system_change_handler(ID: string, new_value: string): string
    print fmt("Value changed from system_name to %s", new_value);
    return new_value;
event bro_init()
    Option::set_change_handler("OurModule::system_name", system_change_handler);

This code registers a change handler for the OurModule::system_name option. Each time that the option value is changed, the system_change_handler function will be called before the change is performed. As you might already have deduced from the function signature, the change handler also can change the value before it is finally assigned to the option. This allows, for example, checking of parameters values to reject invalid input. It is also possible to chain together multiple change handlers: Option::set_change_handler takes an optional third argument that can specify a priority for the handlers.

Note that change handlers are also extensively used internally by the configuration framework. If you look at the script level source code of the config framework, you can see that change handlers are used for logging the option changes to config.log.

If you inspect the scripts further, you will also notice that the script-level config framework simply catches events from the Input framework and then calls Option::set to set an option to the new value. If you want to change an option yourself during runtime, you can call Option::set directly from a script.

The following figure shows the data flow and the different components that make up the entirety of the config framework:

unnamed (4).png

You can try the configuration framework today! It has been merged into Bro and will be part of Bro 2.6. To try it, either install Bro from source or install one of the nightly builds.

That’s a Wrap! The Bay Area’s First Open-Source Bro Meetup — January 18, 2018

That’s a Wrap! The Bay Area’s First Open-Source Bro Meetup

By John Gamble, Director of Marketing at Corelight

Last Tuesday Corelight hosted the Bay Area’s first meetup for the open-source Bro network security monitor and we saw a great turnout of Bro fanatics and first-timers alike at our San Francisco headquarters.

Meetup attendees mingled over pizza, salad and drinks before Vern Paxson, the creator of Bro, kicked off the discussion, followed by engaging Bro lightning talks by Aashish Sharma of Lawrence Berkeley National Laboratory (Berkeley Lab) and Seth Hall, a core contributor to the open-source project.

Notably, all three individuals are members of the Bro Leadership team.  

Aashish walked the audience through Berkeley Lab’s network architecture and showed how Bro plays a critical role, providing them with network insights for cybersecurity. They have had Bro running in their environment since 1996!

bromeetup 1Aashish explaining Berkeley Lab’s network architecture

Aashish observed that security vendors and incident responders tend to focus on specific threat indicators, but vendor alerts don’t usually explain WHY they fired, leaving the analyst to fill in the gap as part of a lengthy investigation.  He urged attendees to evolve from an indicator-centric detection approach to a more attack-centric approach that attempts to identify malicious behaviors at every step of the attack, from scanning to data exfiltration and misuse. 

“Bro allows us to design attack-centric detections,” Aashish said. He used the example of a phishing attack to show how Bro can see every step of the attack as it passes through the network, from the URL click to the phishing form, to the victim’s entry of stolen credentials. With corresponding Bro detection scripts, you could alert at every stage of the attack and light it up “like a Christmas tree”. Aashish closed his talk by calling on the Bro community to share more intel and best practices, including attacker M.O.s and methods so that we can collaboratively develop more effective Bro detection scripts.

Seth Hall’s lightning talk covered the use of flame graphs to analyze Bro performance and resource utilization trends and anomalies that are not readily apparent from looking at the logs alone. Flame graphs are an open-source visualization tool developed by Brendan Gregg (currently at Netflix) and they can help identify the most frequent code-paths of an analyzed piece of software.  

bromeetup 2Seth walking through a flame graph!

Seth remarked that “real traffic is never like sampled PCAP…there is always something to surprise you” and showed attendees a number of flame graphs he produced of Bro processes running on real network traffic in production.

Seth showed an eye-catching plateau in one flame graph that revealed a particular Bro process behaving abnormally by spending 80% of its execution time in a single function. When he dug into the issue he said he realized a set of tables were filling up, causing this issue, and was able to successfully troubleshoot it.

Corelight has made strong commitments to supporting and promoting the open-source Bro project and community: we’re a sponsor of the project and recently hired our first employee whose sole responsibility is open-source project development. You can learn more about Bro at and sign-up for the mailing lists there to get in touch with other Bro enthusiasts and experts.

If you’re in the Bay Area, I’d encourage you to join our open source Bro group: and attend the next meetup event!

Extensibility as a Guiding Principle — December 6, 2017

Extensibility as a Guiding Principle

By Christian Kreibich, Senior Engineer at Corelight

If you’ve ever used Bro, you’ve likely noticed that it’s rather more flexible than other network monitoring solutions. This is not coincidence — it reflects a core principle that has underpinned the evolution of the Bro platform since its beginnings two decades ago. This principle has afforded users a wealth of benefits that continue to shape today’s product vision here at Corelight.

Let’s start with Bro’s basic design. It uses a traffic-parsing core to feed protocol events into its built-in script interpreter, which separates mechanism (the nitty-gritty parsing of traffic) from policy (what to do in response to observed activity). These events (450+ different types) cover a vast semantic range and span the entire protocol stack. Combined with the Bro scripting language — designed specifically to simplify network-typical compute and state-keeping tasks (a key difference to other popular languages such as Lua) — extensibility is not just a feature: it defines the system. Let that sink in for a moment: while Bro is a fantastic network flight recorder (consider the breadth and depth of the traffic logs the Corelight Sensor produces), this log production is merely one configuration of the system. Behavioral profiling of your end-hosts? Check. Arbitrary cross-flow/protocol state-keeping? Check. Lateral movement? Check. The system provides the building blocks, you provide the analysis — either in real-time on the appliance or in the form of actionable data in your broader analytics pipeline.

Extensibility doesn’t stop at Bro’s core design. It’s designed from the ground up to support clustering at the process and machine levels and features a powerful communication and data-persistence infrastructure to scale your deployment as your network grows.

Your needs go beyond those 450 event types? Bro has you covered. Its plugin architecture supports adding compiled code that allows you to add new functionality — for example a new protocol analyzer or packet source — on your own and without ever needing to patch the Bro source tree. No more worrying about release cycles, licensing, or development workflow. The core is designed to support extensibility. At the script level, you can derive new event types as needed. Concerned about managing those extensions? The Bro package manager makes it just as easy to maintain your in-house Bro feature set as it is to manage other distributions in your infrastructure.

What about integrating Bro with the rest of your infrastructure? BroControl provides a handy remote control for your Bro installation and supports plugins. On the network side, the NetControl framework provides a wide range of connectors to mesh Bro into your enforcement infrastructure. Naturally, the framework is designed for extensibility so you can add additional connectors with ease. Thinking of leveraging your existing threat intel feeds in Bro? The input framework makes it easy. Your file processing pipeline to check whether those PDFs are malicious? Bro’s file analysis framework feeds right into it. Finally, Bro’s forthcoming osquery integration allows it to include host-based events.

As a Corelight customer, you benefit from all of these features: we’re committed to running open-source Bro on our Sensors. We’ve streamlined custom script installation via our APIs, taken care of data export to Splunk, Kafka, or your favorite SIEM, and taken the hassle out of managing and tuning fast packet analysis pipelines for you. To ensure stability, the Sensor doesn’t yet expose all of Bro’s functionality — for example, customers cannot currently deploy their own plugins — but we’re aiming for feature convergence over time.

To us, extensibility is not an afterthought that we try to tuck on in a few release cycles. It permeates the way we think about network monitoring and has enabled scalability, visibility, profiling, learning, and detections battle-tested over two decades of real-world use in some of the world’s most demanding network environments.

Finding Very Damaging Needles in Very Large Haystacks — September 26, 2017

Finding Very Damaging Needles in Very Large Haystacks

By Vern Paxson, Chief Scientist at Corelight

Some of the most costly security compromises that enterprises suffer manifest as tiny trickles of behavior hidden within an ocean of other site activity.  Finding such incidents, and unraveling their full scope once detected, requires far-ranging network visibility, such as provided by Corelight Sensors, or, more broadly, the open-source Bro system at the heart of our appliances.

Wading through vast amounts of data to find genuine threats can prove intractable without aids for accelerating the process. An important technique for enabling detection is the development of highly accurate algorithms that can automate much of the task.  In this post, I sketch one such algorithm that I recently worked on at UC Berkeley along with two Ph.D. students, a member of the security team at the Lawrence Berkeley National Laboratory (LBL), and a fellow UCB computer science professor.  The research concerns detection of spearphishing and we published it last month at the USENIX Security Symposium, where it won both a Distinguished Paper award and this year’s Internet Defense Prize, a $100,000 award sponsored by Facebook.

As you probably know, spearphishing attacks are a form of social engineering where an attacker manually crafts a fake message (sent via email or social networking) targeting a specific victim.  The message often includes carefully-researched details that improve the likelihood that the victim will believe the message is legitimate, when in fact it isn’t.  We call this facet of spearphishing the lure.  The message entices the victim to take some unsafe action (the exploit), such as providing login credentials to a fake web page posing as an important site (e.g., corporate GMail), opening a malicious attachment, or wiring money to a third party.

In our work, we collaborated closely with the cybersecurity team at LBL to tackle the problem of detecting spearphishing attacks that result in victims entering their credentials into fake web pages.  The Lab – an enterprise with thousands of users – maintains rich and extensive logs of past network activity generated by Bro, and for real-time detection operates numerous Bro instances. (I worked at LBL when I first developed Bro in the mid-1990s, and the Lab has been a key user of it ever since.)

For our recent work, we drew upon LBL’s Bro logs of 370 million emails, along with all HTTP traffic transiting their border, and LDAP logs recording the authentications of Lab users to the corporate Gmail service.  (To get a sense of the richness of Bro data, check out the information it provides for SMTP and HTTP.)  We were also able to cross-reference with their security incident database to assess the accuracy of the different approaches we explored and developed.  All of this data spanned 4 years of activity.

The key idea we leveraged was to extract all of the URLs seen in incoming emails and then look for later fetches of those URLs by Lab users.  Such a pattern of activity matches that of a common type of credential spearphishing, where the lure is a forged email seemingly from a trusted party.  The lure exhorts the recipient to follow a link to take some sort of (urgent) action.  However, this activity pattern also matches an enormous volume of benign activity too.  Thus, the art for successfully performing such detection is to find ways to greatly winnow down the raw set of activity matching the behavioral pattern to a much smaller set that an analyst can feasibly assess – without discarding any actual attacks during the winnowing.

In previous projects, I’ve likewise tackled some needle-in-haystack problems.  I worked with students and colleagues on developing detectors for surreptitious communication over DNS queries and for attacks broadly and stealthily distributed across many source machines.  From these efforts, as well as this new effort on detecting spearphishing, several high-level themes have emerged.

First, it is very difficult to apply machine learning to these problems.  The most natural machine learning techniques to use “supervised” ML, require labeled data to work from.  For spearphishing, this would be examples of both benign email+click instances and malicious ones, which comprise two separate classes. The ML would then analyze a number of features associated with email+click instances to find an effective classifier that, when given a new instance, uses that instance’s features to accurately associate it with either the benign or the malicious class.

While supervised ML can prove very effective for some problem domains, such as detecting spam emails based on their contents, for needle-in-haystack problems it runs into major difficulties due to the enormous “class imbalance”.  For the spearphishing problem, for example, we can provide an ML algorithm with 370 million examples of benign email+click instances, but fewer than 20 malicious instances, since such attacks only very rarely succeed at LBL.  In these situations, the ML will very often overfit to the class with very few members, emphasizing features among its instances (such as the specific names used in emails) that have no general power.

Detectors for highly rare attacks, whether or not based on ML, face another problem concerning the base rate of the attacks.  A simple way to illustrate this is to consider a detector for email-based spearphishing that has a false positive rate of only one-in-1,000 (0.1%).  When processing 370 million emails, this seemingly highly accurate detector will generate 370,000 false positives, completely overwhelming the security analysts with bogus alerts.

In working on past needle-in-haystack problems, I’ve come to appreciate the power of (1) not trying to solve the whole problem, but rather finding an apt subset to focus on, (2) devising extensive filtering stages to reduce the enormous volume of raw data to a much smaller collection that will still contain pretty much all of the instances of the (apt subset of the) activity we’re trying to detect, and (3) aiming not for 100% detection, but instead to present the analyst with high-quality “leads” to further investigate.

For our spearphishing work, the subset of the problem we went after was email-based attacks that involve duping the target into clicking on a URL, and for which the target indeed did wind up clicking.  We framed our approach around an analyst “budget” of 10 alerts per day.  That is, on average, on any given day our detector will not produce more than that many alerts.  We set the number of alerts to 10 because the LBL security staff deals with a couple hundred monitoring alerts per day, so getting 10 more does not add an appreciable burden – assuming that alerts that are false positives are cheap enough to deal with, a point I return to below.

We then identified a set of features to associate with emails containing URLs and any subsequent clicks on those URLs.  This part took a great deal of exploration – indeed, the entire project spanned two years of effort.

For the most part, the features draw upon the site’s history of activity.  For example, for emails one feature is for how many days in the past a given email From name (e.g., “Vern Paxson”) was seen together with a given From address (e.g., “<>”).  For clicked URLs, one example of a feature is how many clicks the domain that hosts the URL received prior to the arrival of the associated email.  To detect spearphishing sent from already-compromised site accounts, we also analyze LDAP logs to correlate emails with the preceding corporate Gmail authentication used by the account that sent the email, drawing upon features such as how many of the site’s employees have previously authenticated from an IP address located in the same city as was used this time.

We wound up identifying eight such features (though it turns out we don’t use all of them together).  For each email+click instance, we compute the values of the relevant features and score the combination in terms of how many previous instances were strictly less anomalous than it (i.e., had more benign values for every one of the features).  Our detector then flags the instances with the highest such scores for the analyst to investigate further, staying within the budget of an average of no more than 10 alerts per day.

The detector has proven to be extremely accurate.  In our evaluation of 4 years of activity, it found 15 of the 17 known attacks in LBL’s incident database.  In addition, it found 2 attacks previously unknown to the site.  It achieved this with a false positive rate less than 0.005%.

Finally – and this is critical – we measured how long it takes an analyst to deal with a false positive, and it turns out that the vast majority can be discarded in just a few seconds.  This is because, for most of the false positives, it’s immediately clear just from their Subject line or their sender that surely they do not represent a carefully crafted spearphish.  An attacker will not dupe a user into clicking on a link and typing in their credentials using a Subject line such as “DesignSpark – Boot Linux in a second” (an actual example from our study).  As soon as an analyst scans that Subject line, they can discard the alert from further consideration.  As a result, it typically takes an analyst only a minute or two per day to deal with the alerts from our detector.

In summary, our work showed that we can make significant strides towards combating spearphishing attacks by (1) cross-correlating different forms of network activity (URLs seen in emails, subsequent clicks on those seen in HTTP traffic, LDAP authentication) in order to (2) find activity that we deem suspicious because, for a carefully engineered set of features, the activity manifests a constellation of values rarely seen in historical data.  We also, crucially, (3) aim not for 100% perfect detection, but to present a site’s analyst with a manageable volume of high-quality alerts to then further investigate, and (4) find that these investigations take very little time if the alert is a false positive.

This sort of detection underscores some of the major benefits Bro can provide: illuminating disparate forms of network activity, producing rich data streams that sites can archive for later examination and enabling analysts to zero-in on problematic behavior.