Corelight Blog

ILLUMINATE YOUR NETWORK

Another cool thing about Bro: tracking files! — September 15, 2017

Another cool thing about Bro: tracking files!

By Vincent Stoffer, Director of  Customer Solutions at Corelight

You probably know that Bro generates real-time data about network flows, highly valued by threat hunters & incident responders around the world.  But Bro can do a lot more, and in this blog series, we’ll highlight lesser-known features from time to time.  

Today: tracking files!  

First the problem statement: how do you monitor the files that go back and forth across your network? Of course, there are logs for some of your enterprise services, and maybe you’re getting info in the form of URLs or hashes from your proxies or other security tools…but what about everything else?  If you were given the hash of a file that you knew was malicious, how would you figure out if it had ever been on your network? What if that file never triggered an alert or system log?

Visibility into all files – not just network flows – is a powerful, under-appreciated feature of open-source Bro. Bro’s file analysis capabilities are pretty amazing, and the data it captures is a great resource for detection, response, and prevention.

Here’s how the feature works: whenever a file is transferred over the network using a protocol that Bro knows about, the file is tracked, hashes are created, and detailed data is logged to the file and its associated connections.  

As an example, here’s a visit to the Slashdot web page by a browser, including an AJAX post as recorded by Bro’s files.log:

Screen_Shot_-_Greg_s_version_of_files-log_blog_post_-_Stoffer_September_2017_-_Google_Docs

This is all part of a single HTTP connection and includes the HTML, favicon, some plain text, plus the JSON.  All these components have been recorded in the Bro logs with a number of important details:

  • The first field is the UNIX timestamp.  The Corelight Sensor outputs this in a standard ISO 8601 date/time format and it’s super precise, helping to pinpoint exactly when a specific event happened in regards to a file.
  • The second field is the file UID.  This is a unique ID/string generated per file seen.  You can reference this to look up other connections which transferred the exact same file.
  • Third and fourth fields are the transmit and receive hosts for the file.
  • The fifth field is a list of all the connections UIDs which this file was transferred over, often it’s just one but it could be part of a series of connections.  This same UID can be used to track an individual connection across any of Bro’s logs.
  • The sixth field is the protocol source that Bro’s analyzers saw the file and extracted it from.

A few other interesting fields include file type, file name (if available), byte counts of various types, calculation of the entropy of the file, and hashes – MD5, SHA1, and optionally on the Corelight Sensor, SHA256.  

Without delving into all the details of what’s available from Bro’s file analyzer, we see that a whole lot of actionable info is created for each of these files.  Remember that this same detail is recorded for EVERY FILE on your network.  And even better, it doesn’t matter what protocol the file was transferred over… as long as Bro can decode it, the file can be extracted – that includes HTTP, SMTP, FTP, IRC, SMB, etc.  In fact, Bro has 50 protocol analyzers. You can perform indicator matching and hunting across everything from web traffic to email attachments.

That’s an amazing amount of data, and as an incident responder, I relied heavily on the files log to help paint a picture of what might have happened for a particular event or series of file transfers.  

But what if you need more than just the derived data about the transfer?  The Corelight Sensor can also extract all of the associated files and export them to a file server.  You can leave them for future investigations, and plumb them into a static or dynamic analysis pipeline – providing not just data about the connection and transfer but indicators and data extracted from the file itself.

Bro doesn’t stop there. The same level of forensic detail is available for individual protocols as well…we’ll get into some of the other logs in a future blog post.  

Do you have some unique ways you use the files.log or questions about how it could help your security team?  Drop us a line – info@corelight.com.

Securing the Corelight Sensor — September 6, 2017

Securing the Corelight Sensor

By Steve Smoot, VP Customer Success @ Corelight

Have you ever considered how security tools can be a source of risk? They process untrusted data 24/7, have access to sensitive flows, and (like everything on the Internet) can be exploited if not patched regularly.  

At Corelight, we want our products to be a source of visibility and insight, not risk, and we’ve done a lot of thinking about how to secure them. In my first post, I’d like to take the opportunity to explain some of the techniques we use.

We work hard to limit the attack surface for each sensor.  Except for initial login over a physical connection, all access is disabled by default.  Even after initial configuration, access is limited to SSH, HTTPS or the Corelight API (which uses TLS), according to the options you choose.  Each of these modes is password/key protected, and all disk volumes are encrypted.

Corelight’s user environment is sandboxed from the execution environment, and we use features in the Linux kernel to provide security isolation between major functional components. We don’t run Bro as root, a habit we’ve seen in many open-source deployments. Scripts run in a separate, security-isolated environment from Bro itself.  And we’ve implemented dozens of additional features for your protection, features we prefer to keep under the hood.

Third-party software requires special care, and we monitor and address vulnerabilities actively.  Fortunately, most CVEs don’t even apply to our products, because we strive to minimize the number of packages used.  In the spirit of transparency, we list resolved CVEs on the Corelight support site and announce high-profile CVEs on the public web site.  In addition, we call out high-profile but inapplicable CVEs on the customer support for extra assurance (i.e., the ones that hit the news, but we’re not vulnerable to).

Everyone knows that unpatched systems give rise to many breaches, and automatic updates are a real boon to operational security. The Corelight Sensor can of course help you find those unpatched systems on your network, but we’ve also made automatic updates simple and painless.  In fact, we default to automatically updating our software when new releases are available. Although we give customers configuration knobs to control when to apply updates, most of them choose to update immediately after each new release.  That’s good!

We also strive to be transparent.  It’s really disappointing when a partner you trust hides a potential problem from you. Responsible disclosure requires a careful balancing act, but it’s important for us to be as transparent as we can about risk – hence we will be posting to this blog and other customer-alerting mechanisms we employ including regular email updates / security bulletins.  

Finally, we set high standards internally to protect our development environments and to shield you from third-party attacks or undesired leakage of information through your relationship with Corelight.  We’ll itemize those internal standards like code signing, peer review, security best practices for internal infrastructure, etc in an upcoming blog post. Stay tuned.

Please rest assured, we’re thinking about how to improve the security of our products constantly.

What’s the riskiest part of your Bro deployment? It may be you. — August 16, 2017

What’s the riskiest part of your Bro deployment? It may be you.

By Seth Hall, Co-founder & Chief Evangelist at Corelight

Don’t overlook the obvious: the answer may be you 😉

Let me explain, because I’ve watched the following story unfold many times.  A curious person gets super excited about Bro, deploys it widely in their organization, and makes a big impact on the local SOC.  Everyone on that team becomes more effective, because Bro data helps them understand and respond to security incidents so much faster. Over time, this Bro advocate becomes the local Bro expert – responsible for configuration, tuning, documentation, patching, integration, etc.  It’s a full-time job.  And that’s OK, until the local Bro expert is hired away for the experience he or she just acquired!  

It happens. Just think of all the skills that person gained along the way: about Bro itself, specialized network cards, BIOS/UEFI firmware options, network stack tuning, file systems, memory allocators, etc.  This means your ‘local Bro expert’ is an asset but also a risk.  Because so many companies are looking for Bro experts now, it cuts both ways.  I’ve seen wonderful Bro deployments fall into disrepair when a key person leaves.

In fact, it was watching that pattern unfold several times that led us to develop the enterprise ready, turn-key Corelight Sensor about 18 months ago because we had identified that just creating Bro wasn’t quite enough.  We sweated many, many details so that customers could confidently deploy Bro in less than 30 minutes, focusing effort on incident response, forensics, and threat hunting.

As a very small example of the tiny details we take care of for you on the Corelight Sensor and because I’d like to provide some useful tidbit for people running Bro on their own, I’d like to finish the post with a note about using tcmalloc.  Tcmalloc is an alternative memory allocator that was originally created by Google as part of their Google Perftools package for memory debugging.  The package has since been renamed to gperftools (found here: https://github.com/gperftools/gperftools) and is no longer officially maintained by Google. It’s intended to perform especially well in multithreaded applications and it has a number of other tweaks that make it an appealing choice as a memory allocator.   A number of years ago we discovered that Bro performs noticeably better when tcmalloc is the memory allocator.  This led to a change in the build system to use tcmalloc by default on Linux if it is discovered.  Bro has been doing this for a long time but we’ve never publicly told everyone that they should be using it.

You should use whatever package system your OS uses to install gperftools and tcmalloc.  On CentOS, it’s named “gperftools” and on Ubuntu it’s named “google-perftools”.  After you install the package, you will want to reconfigure Bro with whatever configure arguments you used previously.  If tcmalloc was found, you will see the following toward the end of the configure output:

gperftools

If it show that gperftools is found and tcmalloc is found then you’re all set to build and reinstall.  If you’ve had trouble getting rid of the last few percentage points of packet loss in your own Bro deployment, this easy change could possibly get rid of it right away!  As you remove more and more of these small problems and Bro’s output becomes better, all of your downstream analysis is improved.  Better data in equals better data out.

On the Corelight Sensor we are already using tcmalloc along with many other specialized configurations and an accelerated FPGA network card.  This is all maintained and updated with zero effort from you so that you can focus on data and discovering intrusions.

And that’s just one example of how you’re covered if your Bro expert disappears one day.

Corelight Accelerated by Venture Funding — July 18, 2017

Corelight Accelerated by Venture Funding

By Greg Bell, Corelight CEO

Welcome to the Corelight blog!

I’m kicking off this series with an update about the company, but future posts will be a lot more technical.  You can expect information and musings from Vern Paxson, Robin Sommer, Seth Hall, Johanna Amann, Christian Kreibich, Vince Stoffer, and others on our team.

Many of you know this – but for those who don’t, Corelight was founded by the creators of open-source Bro and leaders in the open-source community.  We have two goals: 1) to build incredibly effective security solutions on a foundation of Bro, and 2) to channel money and human cycles into the open-source project, helping it grow and thrive.  

To date we’ve been bootstrapping the company.  Even though bootstrapping taught us a lot about focus, it hasn’t helped support the Bro community as much as we’d hoped. Yes, Corelight made a significant donation to the Bro Foundation last year – and we were very happy to do it. But Robin and Seth’s cycles (and Johanna’s too) have been redirected more than we’d like by the process of getting this company off the ground.  

Late last year we decided to look for venture funding, and we have just announced a $9.2M round of investment. Our primary investor is Accel Partners, which has expertise in the art of nurturing and sustaining ‘open’ business models. Accel has invested in Facebook, Docker, Slack, Cloudera, and many other great companies – both open-source and proprietary. Our new board member Eric Wolford brings operational experience, deep roots in product management, and wisdom.  

I’m delighted to announce that Steve McCanne has become an investor in Corelight as well, and an independent board member.  Steve was founding CTO of Riverbed Technology and CTO of Inktomi.  And here’s a coincidence: Steve was creating ‘libpcap’ in the mid-1990s at Lawrence Berkeley National Laboratory while sharing an office with fellow grad student Vern Paxson, who was creating Bro at the same time. So our Series A funding reunites two networking legends, who are working on the same team again.

Since closing the funding round, we’ve relocated to San Francisco and gotten busy hiring – attracting senior leaders who will help us serve our customers better. A good example is our advisor and acting CMO Alan Saldich, previously VP Marketing for Cloudera, and a person who truly understands the dynamics of open-source companies. We’re also busy adding new capabilities to our flagship Corelight Sensor, a turn-key appliance with features and integrations large enterprises need.  We’ll have more to say about our product vision in future posts.    

OK, that’s it for the company update!

In the future, this blog will shine a spotlight on Bro. Whether it’s used to validate or disprove alerts from other tools, piece together complex security incidents, or support threat hunting teams, Bro’s powerful and actionable data is at the center of the world’s most capable security operations. 

The Corelight blog — June 6, 2017

The Corelight blog

In the near future we’ll be starting a regular blog for Corelight. Our company, formerly known as Broala until the beginning of 2017, includes the creators and maintainers of the Bro open source network visibility framework.

Our product, the Corelight Sensor, is an enterprise-class appliance that makes deploying, integrating and operating Bro simple for organizations who need world class visibility into their network traffic to help detect and prevent attacks.

Check back here regularly for posts by our founders, engineers and others at Corelight.