loader

Network Security Monitoring (NSM) Using Elastic

Companies nowadays are investing more in network security monitoring tools to detect and respond to intrusions on their networks. This makes adopting the right way to search and analyse data generated from the various toolsets of vital importance to reduce blind spots. Challenges are acute when the Security Operation Centre (SOC) analysts want to gain actionable insights from events generated from diverse sources. This blog post explores how to overcome these challenges and boost the effectiveness of your network security monitoring (NSM) using the powerful Elastic Stack. 

Network Security Monitoring Tools landscape and the challenges they create

Realizing that the landscape of cyberattack is increasing, companies are implementing a broad set of network security monitoring tools to detect and prevent threats from both outside and within their networks. While the ever-growing number of safeguards provides more complete security coverage, the SOC team also faces challenges when they try to make the best use of those network security monitoring tools. The top two challenges are:

  • Complexity in creating monitoring points

One undeniable truth is that the sophistication of nowadays network architecture increases the complexity of network monitoring. Companies may use a variety of tools to detect malicious activities in early stages such as Arkime (formerly known as Moloch) for packet capturing, and Zeek (formerly known as Bro) for network analysis, intrusion detection and prevention. The security team requires full visibility of indicators generated from these various tools, which in turn, requires a scalable solution to collect and visualise relevant monitoring data from various tools.

  • Difficulty in generating insightful information

Network Security Monitoring data can be generated from diverse sources. Even if all the data is gathered and stored in a central repository, security analysts need to find correlations between events to search, analyse and investigate network traffic for potential attacks. Additionally, with the high volume of data collected, SOC teams can be overwhelmed by too many alerts generated from different security tools.

A centralised SIEM platform that is customisable, scalable and can work with a variety of tools such as the Elastic SIEM, is the solution that can provide the capability to search, analyse and manage network security data and triggered alerts.

You need centralised visibility to scrutinise your network security

Elastic SIEM, with its powerful log management capability, out-of-box detections and scalable Machine Learning is one of the best options to fulfil the requirements to address the above issues. The Elastic SIEM can process data from various sources and support in-depth analysis of the processed data. Some of the key components required for your network security monitoring capability are:

  • Beats – Lightweight data shippers that collect data and send it to Elasticsearch. Packetbeat is Elastic’s real-time network packet analyzer. While Filebeat is another member of the Beat family, which is used to forward log data from other network security monitoring tools. The Filebeat has a variety of modules used to process logs. 
  • Logstash or ingestion pipelines – Used to parse and enrich the log data.
  • Elasticsearch – Data is centrally stored here and becomes ready for search and analysis. 
  • Kibana –  Frontend application for users to query and visualize data indexed in Elasticsearch. In particular, it provides the UI for SIEM features and enables the SOC team to view alerts generated from network security data.

With the integrated solution provided by Elastic, you can streamline the process to monitor network traffic and have a complete picture to detect anomalous behaviours. You can respond quickly when malicious parties try to gain unauthorised access to the network, or when suspicious activity is detected by your IDS/IPS. Using Elastic you can collect relevant information to investigate security incidents and reduce the impact of cybersecurity attacks.

Building the centralized monitoring environment 

In a real-world environment, the network traffic will be forwarded to an endpoint via a Switch port Analyzer (SPAN), mirror port or TAP where a network security monitoring tool is installed. A beat agent will be installed on the same endpoint to ship the events generated from the network security monitoring tool to Elastic.

For simplification, we will demonstrate below how to configure and use the centralized environment in a test environment where Zeek, Suricata, Snort and Arkime are installed on one host and shipping the logs to an Elastic Cloud instance using Filebeat. We will also show how to enable the community ID that is used to correlate events between different tools.

Configuring network security and monitoring tools in the test environment – 

  • Suricata

After you install Suricata on a host, you need to edit the configuration file /etc/suricata/suricata.yml, to particularly select which interface and IP address you want Suricata to listen on and which IP/network is not local. Also enable community flow hash id by setting community-id as true.

You can run Suricata with the help of following command:

sudo /usr/bin/suricata -c /etc/suricata/suricata.yaml -i eth0

Suricata logs can be viewed in: /var/log/suricata/eve.json

  • Zeek

After you install Zeek on a host, you need to edit the file /usr/local/zeek/etc/node.cfg to select which interface to monitor and /usr/local/zeek/etc/network.cfg to identify networks which are local to the environment being monitored. 

You can run Zeek with the help of following command:

./zeekctl

after navigating to /usr/local/zeek/bin/

Zeek logs can be viewed in: /usr/local/zeek/logs/current/

Logs for different events like capture_loss, dns, http etc. can be found there. 

To enable community id flow hashing, bro-community-id plugin needs to be downloaded and installed. Once done, Zeek needs to be configured to load this plugin. This can be done by adding the following line in /usr/local/zeek/share/zeek/site/local.zeek:

@load Corelight/CommunityID

Restart Zeek with the new configuration to load the plugin.

  • Snort

After you install Snort on a host, you need to edit the configuration file snort.conf, to identify which network  address you want Snort to listen on and which IP/network is not local. Also uncomment the syslog output in step 6. 

output alert_syslog: host=127.0.0.1:514 LOG_AUTH LOG_ALERT

Configure rsyslog to listen on 514(tcp) and forward the logs to Filebeat on port 10514(tcp)

You can run Snort with the help of following command:

sudo snort -c /etc/snort/snort.conf -i eth0 -s -A console

Snort logs can be viewed in: /var/log/snort/snort.log

  • Arkime (formally known as Moloch)

After you install and configure arkime on a host, you can start Arkime service with the help of following command:

systemctl start molochviewer.service

Packet capture can be started using the following command:

./moloch-capture -c ../etc/config.ini

after navigating to /data/moloch/bin/

  • Enable Filebeat modules

After you install and configure the Filebeat on a host, you can enable the built-in modules to simplify the collection, parsing and visualisation of common log formats. The Zeek, Snort, Suricata module can be enabled with the running the command:

sudo filebeat modules enable suricata zeek snort

Then you need to edit the configuration file of each module, zeek.yml snort.yml suricata.yml , particularly when you want to edit the entries for getting in logs. 

  • suricata.yml

var.paths: [“/var/log/suricata/eve.json”]

  • zeek.yml

Specify var.paths for each event type. For example:

dns:

var.paths: [“/usr/local/zeek/logs/current/dns.log”]

  • snort.yml

var.input: tcp

var.syslog_host: localhost

var.syslog_port: 10514

Now you can start Filebeat using:

sudo systemctl start filebeat

 

  • View logs in the Discover page

Once you have the Elasticsearch cluster in place, you can use the Filebeat to transform the network logs and store them in Elasticsearch as Filebeat indices. On the Discover page, you can find how to make the network data visible with simple queries. 

 

  • View built-in dashboards for Filebeat

Filebeat Zeek overview:

Filebeat Suricata Events Overview:

Filebeat Suricata Alert Overview:

 

  • View Elastic Security and create detection rules

Now you can view your network security data in the Elastic Security UI and have a look at how many events are collected by Filebeat. 

Moreover, you can enable some detection rules or build your own to generate alerts for malicious activities. You can also use machine learning to identify anomalous network behaviours.

 

  • Correlating Suricata, Zeek and Arkime events

Suricata and Zeek events can be correlated with each other and pcap in Arkime with the help of ‘Community Id’ field present in Arkime and ‘network.community_id’ field present in elastic for Zeek and Suricata. As you can see in the below screenshot, the pcap in Arkime and Zeek and Suricata events in elastic have the same community id.

What’s next

Network Security Monitoring is crucial for your security detection capability. You must build an effective solution to gain centralised visibility into your network. If you have any questions regarding how to do that, please Contact Us. Next, we are encouraging you to think about how to improve your efficiency by leveraging SOAR solutions and automate the process of detecting threats and responding to incidents. Read more about the topic in our Using Elasticsearch to Trigger Alerts in TheHive blog post.

Reference:

Elastic, Collecting and analyzing Zeek data with Elastic Security, available at: https://www.elastic.co/blog/collecting-and-analyzing-zeek-data-with-elastic-security

Written by: Abhishek Jayachandran, Astrid Liu