Cyber Attacks Global Incident Report Statitstics

We are now generating weekly Global Incident Reports that provide statistics of the invariants present in our global detection infrastructure. The detection infrastructure receives approximately 8 million events per day from a variety of Institutions ranging from small commercial enterprises to very large multinational corporations.

The statistics below are from three main detection components.

The invariants from the events reported by these detection components are extracted and their relative contribution is compared. The contribution of the invariants is measured in three different dimensions:

  • The true positive rate  (tpr) of an invariant is measured by dividing the number of confirmed true positive hits by the number of occurrences of the same invariant (whether they are a true positives or not). The true positive rate implicitly also measures false positive rate (1-tpr). For clarity the tpr is called <strong>detection rate</strong> in the Network Anti-virus tables.
  • Severity ranges from 0 to 100 and measures the likelihood that an invariant in a cyber attack compromises the integrity or confidentiality of a system. The severity is scaled down by the tpr and is calculated by multiplying the average priority (0-100) of the invariant times its tpr (which is always less than 1). A low severity score (0-10) typically implies that the cyber attack may reduce security but the loss of security is minimal (for example detecting an ADWare plug-in in your browser). Higher severity scores imply that the cyber threat becomes increasingly important.
  • Prevalence measures how widespread a given cyber attack is across multiple networks. Prevalence is also weighted against the tpr of a given invariant. Prevalence does not have an upper limit because it depends on how many cyber attacks we find in a given time period.

Selection_020Here is an example bubble graph which visually represents the statistics of the top IDS rules which triggered a true positive.

Mousing over the bubbles reveal the actual invariant and its associated statistics.


How to access the statistics

  • The anonymized global report across all of our networks is at https://www.metaflows.com/stats/. From this report there are some hyper-links that query you own database (if you are MetaFlows customer) to see if any of the invariants a re present in you event data.
  • If you are a MetaFlows customer, you can also access a specific report for your own domain which has both (1) links to the invariants found on your own domain and (2) links to the incident reports used to derive the invariants.

Note that both types of reports compare the invariants to the global counts; so, they both should help you understand how widespread and how serious the associated cyber- threats are.

MetaFlows New Packet Data Viewer

Since our first public release, MetaFlows users have been able to use the MetaFlows Real Time and Historical interfaces to download and view raw packet data from their MetaFlows sensors. Today, we are excited to release a major upgrade to the Packet Data page.

New Features:

  1. Packets are now parsed, separated, and colored red or blue depending on which host sent the data. No more digging through raw tcpdumpoutput to find where one packet ends and another begins! Packets are now visibly separated. Server packets use red text. Client packets have blue text.
  2. Packet Data now includes the content of any IDS rules triggered by the flow. Ever opened the Packet Data for a flow and forgotten which alert you were investigating? We now include the full content of all IDS rules that triggered, along with the packet data for the flow.
  3. Packet Data Matcher! This was the most exciting feature to add! The new Packet Data page can highlight the specific content in a packet that caused an IDS alert to trigger! Matching packet data is highlighted in red or blue (depending on whether the packet was sent by the server or client) and includes numbered markers for each condition in the triggered rule. You can even hover your mouse over these markers to see the specific content or pcre condition that matched, along with the condition’s modifiers!  For example, the screenshot below shows the packet data for an event matching 1.2003492: ET MALWARE Suspicious Mozilla User-Agent - Likely Fake (Mozilla/4.0):Example of Packet Data Viewer with Packet Data Matcher results.The Packet Data Matcher works best for text flows, such as HTTP requests and responses. For encrypted or binary flows, we will still attempt to match against non-printable characters by using other conditions and context, but some matches might be inaccurate.

This feature is available to all MetaFlows customers starting today. We look forward to your feedback. Happy hunting!

Automatic DNS Resolve

Thank you for your feedback!

We added a new preference setting in Account->Preferences to automatically resolve internal and/or external IP addresses to DNS names. As soon as the browser receives an IPv4 address from the sensor or a historical query, it will try to resolve it to a DNS name using the DNS server(s) used by the sensor (specified in /etc/resolv.conf). Obviously, not all IP addresses can be resolved to a DNS name and, therefore, some will stay in dotted-decimal format.

if you have some IPv4 addresses that are not in the DNS system and you want to resolve them, you can add them to the file /nsm/etc/hosts with the format. If the file does not exist, you can just create it:

#this is a comment 1.2.3.4 <dns_name> 5.6.7.8 <other_name>

Whenever 1.2.3.4 or 5.6.7.8 are seen by the browser (if you have auto resolve turned on), they will be translated to their corresponding names.

As always, let us know if you have any questions at support@metaflows.com.

Improved Packet Logging

We have added support for logging the first 512,1024,2048,4096 bytes of each session rather than the full session. You can select how much of a session length to log in the sensor configuration page under the Store packets on Sensor option.

Unless you have a requirement to log all packets, we recommend changing the default all  to 1024 or 2048. This does the following:

  • Makes much more efficient use of the disk space (you can record important payloads for much larger time windows).
  • When querying for packet data, you will see both sides of the session (client requests and server replies). This helps in getting better context when doing forensic analysis.

Also, we have added some color coding to better discern the two different sides of the conversation.

The only drawback of selecting a session size other than all is that the file carving function accessible through Look for Files in Flow(s) may not be able to reconstruct files unless one of the two IP addresses was tracked due to an incident report.

If you want to carve files for arbitrary flows (regardless of incident reports), you need to keep the session length set to all.

As always, let us know if you have any questions! Do not hesitate to call us or send us email.

New Packet Data Viewer

Since our first public release, MetaFlows users have been able to use the MetaFlows Real Time and Historical interfaces to download and view raw packet data from their MetaFlows sensors. Today, we are excited to release a major upgrade to the Packet Data page.

New Features:

  1. Packets are now parsed, separated, and colored red or blue depending on which host sent the data. No more digging through raw tcpdump output to find where one packet ends and another begins! Packets are now visibly separated. Server packets use red text. Client packets have blue text.
  2. Packet Data now includes the content of any IDS rules triggered by the flow. Ever opened the Packet Data for a flow and forgotten which alert you were investigating? We now include the full content of all IDS rules that triggered, along with the packet data for the flow.
  3. Packet Data Matcher! This was the most exciting feature to add! The new Packet Data page can highlight the specific content in a packet that caused an IDS alert to trigger! Matching packet data is highlighted in red or blue (depending on whether the packet was sent by the server or client) and includes numbered markers for each condition in the triggered rule. You can even hover your mouse over these markers to see the specific content or pcre condition that matched, along with the condition’s modifiers! For example, the screenshot below shows the packet data for an event matching 1.2003492: ET MALWARE Suspicious Mozilla User-Agent - Likely Fake (Mozilla/4.0):
    Example of Packet Data Viewer with Packet Data Matcher results.
    An example of packet data in the MetaFlows Packet Data Viewer, with matching content highlighted by the Packet Data Matcher.

    The Packet Data Matcher works best for text flows, such as HTTP requests and responses. For encrypted or binary flows, we still attempt to match against non-printable characters by using other conditions and context, but some matches might be inaccurate.

This feature is available to all MetaFlows customers starting today. We look forward to your feedback. Happy hunting!

Network Antivirus White List and Minimum VT Score

vt2We have added support for further customizing the behavior of our network antivirus system.  Not all application providers adhere to signing their executables and/or use sound software engineering principles. This causes some Virus Total Antivirus solutions to exhibit some false positives.

White List

To remedy this situation, you can now set up a white list using regular expressions to exclude the virus scanning from certain sources. The user-definable white list is in /nsm/etc/carverwhitelist on your sensor.

Some example white list entries are:

washingtonpost\.com.*\.zip
lavasoft\.com.*\.zip

Notice that you need to escape special characters like ‘.’

The two expressions above would skip content analysis from the washingtonpost.com and lavasoft.com domain. If you see some repeated Virus Total false positives from specific URLs, please add them to this list so that the false positives can stop. You can have comment lines beginning with the character #.

Minimum Score

We also added support for raising the minimum threshold to declare a sample to be malicious. Our default value is 4, meaning that at least 4 out of 55+ antivirus solutions need to report a hit in order for us to generate an incident report. It was noted that this limit might be too low in certain diverse environments. So, we added the ability for customers to change this value by setting the environment variable VTMINSCORE.

To change this on your sensor edit the file /nsm/etc/mss.sh and add the statement anywhere after the first line of the script:

export VTMINSCORE=<minimum vale>

Setting <minimum_value> to anything other than 4, will change the threshold for us to generate incident reports and email alerts. If you set it to less than 4, you will get more reports. If you set it to more than 4, you will get less reports.

As always, do not hesitate to call us at 1-877-664-7774 or send us an email at support @ metaflows.com for any questions.

OpenAppID Support

 

2000px-Cisco_logo.svg

Cisco released OpenAppID, their answer to Palo Alto Networks’ AppID feature, which allows administrators to know exactly what applications are running in the network.
It has been released as a plugin of the Snort distribution. We have recently upgraded our sensor software to support this feature. OpenAppID results appear as an additional field in the IDS alerts to give better context for the alerts. We also gather this information to associate it with the internal host IP addresses, whether or not they generate an IDS event.

For example, when a user uses Facebook, it will trigger one or more of these:

Facebook Apps
Facebook
Facebook Chat
Facebook Comment
Facebook Read Email
Facebook Send Email
Facebook Status Update
Facebook search
Facebook event
Facebook post
Facebook video chat
Facebook message
Facebook video

If your software has been upgraded, the file /nsm/bin/snort/src/.version should contain 2.7.9.0. If it does not, you can upgrade by executing this command: /nsm/etc/mss.sh restart (Note: MetaFlows UTM appliances do not support OpenAppID yet).

To turn on this feature, check the OpenAppID checkbox in your sensor configuration page and reload or restart the sensor.

Once this feature is turned on, you can look at the daily reports and see the top AppID summary or look at the AppIDs in your IDS events. You can now create user-defined policies that match specific AppIDs!

This new feature requires 40% more memory and in some cases, even though we install it, the system automatically turns it off if you do not have enough memory. You need at least 2 GB RAM per core. For example, if your subscription is for 16 cores and your sensor has 24 GB RAM, the system would disable OpenAppID automatically.

If you do not process a lot of data and have a low memory system, you can force the loading of OpenAppID by adding the line export forceappid=1 at the top of the /nsm/etc/mss.sh script. Note that because it uses about 40% more memory, your sensor might slow down if you do not have enough RAM. Please monitor your drop rate closely if you force the OpenAppID functionality.

We highly recommend using this feature. If you have any questions, please do not hesitate to contact our engineers at support@metaflows.com for more information.

Throttle in Passive Mode

throttling in passive mode!?!?Sometimes users can knowingly, or unknowingly, abuse a network by using a lot of bandwidth. With the proliferation of video on demand services such as Netflix, Hulu, and Amazon Prime, some institutions are once again finding themselves battling bandwidth issues.

Until now, one needed an in-line device, such as a firewall, to throttle traffic by allocating certain bandwidth to certain flows or applications. Any in-line device also adds latency and reduces reliability, especially on high speed links. Wouldn’t it be nice to throttle specific traffic in a way that does not impact performance of the traffic you do care about? This has been an age-old conundrum for network engineers.

Well, continuing on our hot streak of innovations, MetaFlows recently developed (in collaboration with one of our university customers) an unprecedented technique to throttle traffic in passive mode! It works a bit like active response, where spoofed packets are injected into the traffic stream to shut down flows. In this case, we are not shutting down flows, we are forcing them to slow down.

The result is that you can identify any TCP flow using one or more of our 20,000 signatures (appID is coming very soon), and limit its bandwidth. This means you can have zero impact on performance and reliability of your production traffic while you can achieve very fine grain control of the traffic you do not care about!

Get Packet Payloads with Splunk

It is fairly easy to create a workflow action to access the MetaFlows File-Carving and PCAP extraction interface.

Step 1: Extract the flow information from the MetaFlows event feed.

If you already use CEF log output from MetaFlows, or if you want to change to it, then the required fields should already be extracted:
src, dst, spt, dpt, start

Or, if you are using the standard syslog output then you will need something similar to the following extraction regex to make sure each record has those fields:
\{1,(?<rank>\d+)\}\s+\[\d+:(?<sid>\d+):\d+\]\s+(?<msg>[^\{]*)\{IP\}\s+(?<src>[^:]*):(?<spt>\d+)\s-\>\s(?<dst>[^:]*):(?<dpt>\d+)

Additionally, you will need to append “|eval start=_time” to your queries in order to get the start field unless you already have a derived field which gives you a unix timestamp to use in the query.

Or, if you have your own parsing in place that uses different field names which correspond to ‘Source IP, Destination IP, Source Port, Destination Port, Timestamp‘ then you may need to adjust the field names in the URI under step 2 to match. You will still need to make sure that you can provide a unix timestamp field.

Step 2: Create a workflow action.

Go to settings->FIelds->Workflow actions->Newand set the following fields:
Label: Extract PCAP / Carve Files $src$ $dst$ $start$
Show action in: Both
Action Type: Link
URI: https://nsm.metaflows.com/sockets/historical.php?w=carver&srca=$src$&dsta=$dst$&srcp=$spt$&dstp=$dpt$&st=$start$&sensor_sid=<your sensor’s SID>
Open link in: New Window
Link method: Get

Your sensor’s SID should be a hash listed on the view sensors page, or in the file /nsm/etc/UUID on the sensor itself.

Step 3: Test the Setup.

Test by selecting Extract PCAP / Carve Files from the Event Actions menu for any event in a search. You need to be logged into MetaFlows for this to work. It should take you straight to the File Carving interface which will provide a link to the PCAP data as well.

Note that if you have ‘Log All Packets‘ enabled, you will most likely see the PCAP slice as well as all the files that where downloaded/uploaded in that flow or set of flows. If you do not have ‘Log All Packets‘ enabled, you will only see the PCAP slice corresponding to the packets logged by the IDS system.

Real Time Host Discovery

As you may have noticed we gather quite a bit of information about the hosts running on your network such as OS type, DNS, HTTP agents, DNS, etc. This information is available on the assets report or as a mouse-over when you hover any of the hosts in the HOME_NET. Unfortunately, some of this host information changes very rapidly and it is hard  to correlate with specific events. For example a single host may be using many agents or a proxy may show many OS types. Also DHCP information will show that the same IP may have multiple MAC addresses.

For this reason we added the latest host information to the BotHunter, Tracker and Network Antivirus reports to tell you what specific host information was available precisely at the time of the incident. We are still looking for other improvements, if you have any suggestions, please do not hesitate to send us email at support@metaflows.com.