Websockets Are Here

Adobe Flash is one of the original sins. It is everywhere and yet it is a huge security risk. Websockets is an HTML5 standard that, for us, provided an alternative to Adobe Flash.

For now we support both. The browser will try to use Adobe Flash first, and if it is not present or it is disabled, it will try using Websockets (which are hard-coded in your Browser). I you want to keep using Adobe Flash, you do not have to do anything; things should keep working as before.

If your sensors are configured as clients, and you do not want to use Flash anymore, just disable it and the Browser will do the rest. You will be using MetaFlows SSL certificate.

If your sensors are configured as servers, and you do not want to use Flash anymore, well, it’s a bit of work to use Websockets because current Browser implementations do not allow self-signed SSL certificates (this is probably a good thing). To use Websockets on sensors configured as servers:

  1. Add your sensors’ static IPs to the DNS (like: <sensorname1>@mydomain.com)
  2. Generate a valid SSL certificates  that matches the DNS name in step 1 (cannot be self-signed). If you do not want to generate a separate certificate for each sensor, you can also buy a *.mysensordomain.com certificate to share by all your sensors.
  3. Bundle the certificates with the command:

# cat my_certificate.crt my_certificate.key bundle_certificate.crt > sensorcert

  1. Copy sensorcert to /usr/local/etc/ntop/sensor-server.pem on your sensors’ hard disk.
  2. Go to nsm.metaflows.com and replace the static IP address of your sensors as a server with the names you setup in step 1
  3. Make sure your browser can reach the sensors on ports 3009 and 3010
  4. Restart your sensors as a server with the command:

#/nsm/etc/mss.sh restart

Adding Websockets support was a fairly extensive change in our system; so there could still be some issues. As always feel free to contact us if you have any questions or you see any problems.

Thank you for exploring the unknown with us!

The MetaFlows Team.

Product Update: Reconsider Event Classifications

Recently, the engineers at MetaFlows have improved the Event Classification Menu within the MetaFlows software, allowing each user to further customize events through actions and event views.  This introduces four key features to the Event Classification Menu that users will find helpful in employing the MetaFlows IDS.

Classifications Window.png

The first improvement allows users to see a comprehensive list of their classifications.  Now, users can access a new classification interface that breaks the classifications down by action.  There are seven action types:  Highlight, Block, E-mail, Ignore, Delete, Rank, and Disabled.  The Highlight function matches the records in the Real-Time, Historical, and Reports with the selected color.  The Block action triggers the Soft IPS for matching records, causing connections matching the classification to be blocked.  The E-mail function produces a PDF report of matching records that will be sent every ten minutes, or as frequently as possible.  The Ignore action ignores events that match the classification.  The Delete function removes matching records from the browser in order to free up memory.  The Rank action increases the priority/rank of the records that match the classification.  The Disable function allows a user to disable a classification without deleting it.

Classifications_list.png

The Search functionality of the classification interface now allows users to search against a classifications’ name, category, IP address, IDS alerts, service alerts, and log message values.  All a user has to do is type a value into the Search field to find classifications to match that query.  The search will match against values in the classification name, category, addresses, and events field.

Classification_options_panel.png

Once upon a time, deleting a classification was an irreversible action.  Now, that can be undone.  If the user deletes a classification only to realize later that they need it, they can restore the classification from the Trashed Classifications list.

Transferring classifications is now much easier.  By employing the Upload Classifications feature, a user can transfer classifications in bulk between two different domains.  The option is listed as the Upload Classifications button and selecting this opens the uploader.  Classifications must be in JSON format and contain all the required information for the classification.

More information regarding the recent improvements in the Event Classification menu can be viewed on the MetaFlows User Manual.  If using any of the four new features causes any confusion, or if there are any questions, do not hesitate to contact the MetaFlows team for assistance.

Feature Highlight: Snort Rule Editor

Recently, the Snort Rule Editor as a part of the Rules Management Interface has been updated.  This redesign allows for increased flexibility and provides the user with more of a handle on the IPS rules settings.

How It Works

Entering the Rules Management Interface is easy and can be accessed from two possible locations.  From the View Sensors page, the user can select the Edit Rules link to enter the Rules Management Interface. The user can also navigate to the Rules Management Interface from the Main Menu link.

Upon selecting the Rules Management Interface, the user will be prompted to select a sensor from the dropdown and then choosing the Load Rules button.  From there, the user will be able to manage the rule sets on a sensor-by-sensor basis.

Selection_003

After selecting a sensor, the user will have the option to modify properties of the rule sets, and issue commands to the sensors with the tools on the Menu Bar. In order for the changes made in the rules interface to take effect, commit the changes using the Save button.  (Please note that the Save/Cancel options only appear if the rules have been modified.) After selecting the Save option, a window will appear indicating that the changes are being verified. This process ensures that there are no issues with any changes that were made to the rule sets and that the sensor will correctly load the rules. Once the Save process is finished, the user will be prompted to reload the sensor.

To update the rule files, select the Get Updates button on the Snort Rules Controls. Most of the sensor controls buttons will be disabled/greyed out, and the Get Updates button will have a spinner icon until the update process finishes.  Next, the page will refresh to indicate if the updates were applied successfully. After the Get Updates process completes, select to Save the changes and Reload the sensor configuration for the changes to take effect.

Get_updates_1Get_updates_2

The Rule File List portion of the interface displays a complete listing of all the rule sets in the sensor configuration.  The local.rules file contains the pass rules (if any) that have been generated using the Tune IDS feature and any rules that were uploaded by the user.

Rules_listings_per_file

In the Rule List, each rule can be enabled or disabled by selecting the checkbox under the Active column next to each rule. If the checkbox under the Drop column next to a rule is checked, the sensor will drop flows that trigger the rule. Although we only recommend it for advanced users, each specific rule can also be edited by clicking on the rule itself. This will open the Rule Editor window, displaying the original rule, the editor, and some statistics that have been collected for that rule. Advanced Snort users can use the Rule Editor to make changes to the content of individual rules. When modifications are made, the Diff section will show changes to a rule since the most recent Save of the rule sets.

Manual_rule_editor

Pass Rules inform the IDS system that packet matching these rules should not generate alerts. These are helpful for eliminating false positives without having to disable the offending rule altogether. The Pass Rules have system-generated SIDs so that they do not conflict with the original rules that to which they refer. The user can utilize the tuning interface to add pass rules to the local.rules file. Please remember to Save any changes to the rule sets through the Rules Management Interface.

Selection_004

For users that want to run a reduced rule set for performance reasons, there is an Automatic Tuning option under the Bulk Edit menu. This option will disable rules that are unlikely to trigger based on our observations across all customer networks.  After clicking on the Automatic Tuning option, the changes will be merged with any prior changes that have made, meaning that rules that have specifically enabled will stay enabled. If the user wants to revert to a default minimal set, they should consider first using the Rules Defaults option.  Once the Automatic Tuning has processed the rules, it will then automatically fetch the latest updates and disable all of the appropriate rules.  To complete this process, the user will need to select Save so that the changes are committed.

Full color images and even more detailed instructions regarding The Rules Management Interface can found in the MetaFlows Wiki.

Uncovering True Positives

MetaFlows is now using our sandbox results as an intelligence feed for ranking events.  This method of using the sandbox as an intelligence source for ranking signatures allows us to catch infections or high-risk behavior, even if we only see one piece of the traditional malware life cycle.  The picture below illustrates a sandbox report that shows where the signature was first observed in association with malware.

Selection_002

How It Works

Individual IDS signatures can now be ranked as a priority threat if they have been triggering inside the MetaFlows sandbox in association with malware. These signatures are only considered for special ranking if they are statistically rare among events across all MetaFlows monitored networks.  Given their nature, these events are likely to missed by an analyst among the many other events that may be normally low ranked. The image below displays a ranked event on the user’s dashboard showing an alert identified with the new threat category.

Selection_001

You can see what kinds of events are triggering in the MetaFlows sandbox by visiting our statistics page.

Predictive Global Correlation Feed

After months of data gathering, we turned on a new global correlation feature that complements the existing local multi-session correlation. The aim is to further tighten the net and catch more bad stuff while also decreasing false positives.

We now show the  ranking as total/global when we display an alert. When the global ranking is missing, it is because that event is only ranked locally and the global portion is unknown. When the total and global rank are the same (like 187/187 in the example below), it means that an event was ranked exclusively using global relevance and it would have been missed by the local analysis.

You can see the global ranks by going to the IDS rule management interface. IDS rules listed there will have the current global rank assigned to them for that day (if any).

blog

This additional information complements the local multi-session correlation analysis by trying to look at things from a global intra-domain prospective:

If a domain similar to yours has experienced a significant amounts of high-priority network security incidents involving a particular IDS signature, that signature will receive a positive global rank in your domain.

The key here is the word “similar”.  The events each customer generates are used to compute a similarity matrix that tells us how similar each network is to the others. Using this information, rather than recommending all high-priority signatures to all domains (we call this simple prediction), we only recommend what is most likely relevant to your domain (we call it predictive correlation).

 

Let us know how this works for you and if you have any questions.

Thanks!

User to IP Address Mapping Through Active Directory

We have added support for extracting successful user logins through MS Active Directory for Silver and Gold subscriptions. You can now install a MetaFlows agent (nsm_logc) on your Active Directory servers to export Windows logs to your sensor(s). The agent will also export other critical Windows events to the sensor so that you can record that information and perhaps correlate it with other security events. Although we recommend installing the MetaFlows agent nsm_logc, this mechanism will also work if you install Snare (commercial) or eventlog-to-syslog (open source) instead of nsm_logc.  One advantage of nsm_logc is that the logs are exported through an encrypted channel rather than being sent in clear text.

In any case, the end result will be that any time a user logs in from a specific IP Address, (1) a real-time service discovery alert is generated and logged, and (2) his/her identity is associated with any alert which involves that IP Address. The user identity is appended to the alert messages and is therefore searchable as any other string. Information on how to install the AD agents and some screen shots are here.

As always, do not hesitate to contact us if you have any questions or if you encounter any problems implementing this exciting new feature.

Happy Hunting!

The MetaFlows Team

Command Line Interface Is Here!

As a new feature to the MetaFlows MSS, we have added the ability to query the MSS for both historical flow data (with payload coming from the sensor) and historical event data (coming from our data base). The flow data text output can be formatted to look like what Argus provides (-f -X) or what Bro provides (-c), depending on the options that are selected. We have also added JSON output (-J) for those of you that use Splunk, and want to do some integration work. The MetaFlows Wiki has been deeply revamped and the CLI documentation is at:

https://docs.metaflows.com/Command_Line_InterfaceScreen Shot 2015-09-14 at 4.33.05 PM

The CLI client is written in Perl and can be copied to any system after initialization – so you may execute the CLI queries from any host. Also, if you dare, you can modify the Perl code to change the output formats or to add your own command line switches.

To wet your appetite, the following query will return all the latest malicious content from your sensors:

getflows.pl -u api1:xxxx -B

This query, for example, shows all of the clients on your network attacking Chinese Web Servers:

getflows.pl -E -u api1:xxxx -w 360000 -Q modsec_out | grep CRITICAL | grep :CN

We encourage you to invest some time in looking at this CLI interface. As always, do not hesitate to drop us a line at support@metaflows.com.

Thanks!

The MetaFlows Team

Real Time Email Alerts

We have developed a feature that minimizes delay in generating email notifications. The emails are generated within seconds of the event occurrence to catch extremely time sensitive incidents such as crypto-locker infections. To enable this new feature simply define a shell variable near the top of the start up script mss.sh such as:

export emailaddress=”user@mydomain.com”

Also make sure the sensor can send emails by executing the command:

netstat -tapn | grep 127.0.0.1:25

A line similar to the one below should appear:

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

If you do not see it, please enable your email daemon on the sensor by executing:

/etc/init.d/postfix start
chkconfig postfix on

Then restart the sensor with the command:

/nsm/etc/mss.sh restart

The sensor will now send real time email alerts matching any of the email notification policies you have defined. Note that:

  • This will not replace the email reports you are receiving already but it will provide advance notifications of the alerts contained in those same reports.
  • The email alerts will originate from the sensor which does not have an MX record; therefore your SPAM filters will most likely block them. Please white list the sensor IP address to bypass the SPAM filter.

As always, thank you for your feedback,

The MetaFlows Team.

Cyber Attacks Global Incident Report Statitstics

We are now generating weekly Global Incident Reports that provide statistics of the invariants present in our global detection infrastructure. The detection infrastructure receives approximately 8 million events per day from a variety of Institutions ranging from small commercial enterprises to very large multinational corporations.

The statistics below are from three main detection components.

The invariants from the events reported by these detection components are extracted and their relative contribution is compared. The contribution of the invariants is measured in three different dimensions:

  • The true positive rate  (tpr) of an invariant is measured by dividing the number of confirmed true positive hits by the number of occurrences of the same invariant (whether they are a true positives or not). The true positive rate implicitly also measures false positive rate (1-tpr). For clarity the tpr is called <strong>detection rate</strong> in the Network Anti-virus tables.
  • Severity ranges from 0 to 100 and measures the likelihood that an invariant in a cyber attack compromises the integrity or confidentiality of a system. The severity is scaled down by the tpr and is calculated by multiplying the average priority (0-100) of the invariant times its tpr (which is always less than 1). A low severity score (0-10) typically implies that the cyber attack may reduce security but the loss of security is minimal (for example detecting an ADWare plug-in in your browser). Higher severity scores imply that the cyber threat becomes increasingly important.
  • Prevalence measures how widespread a given cyber attack is across multiple networks. Prevalence is also weighted against the tpr of a given invariant. Prevalence does not have an upper limit because it depends on how many cyber attacks we find in a given time period.

Selection_020Here is an example bubble graph which visually represents the statistics of the top IDS rules which triggered a true positive.

Mousing over the bubbles reveal the actual invariant and its associated statistics.


How to access the statistics

  • The anonymized global report across all of our networks is at https://www.metaflows.com/stats/. From this report there are some hyper-links that query you own database (if you are MetaFlows customer) to see if any of the invariants a re present in you event data.
  • If you are a MetaFlows customer, you can also access a specific report for your own domain which has both (1) links to the invariants found on your own domain and (2) links to the incident reports used to derive the invariants.

Note that both types of reports compare the invariants to the global counts; so, they both should help you understand how widespread and how serious the associated cyber- threats are.

MetaFlows New Packet Data Viewer

Since our first public release, MetaFlows users have been able to use the MetaFlows Real Time and Historical interfaces to download and view raw packet data from their MetaFlows sensors. Today, we are excited to release a major upgrade to the Packet Data page.

New Features:

  1. Packets are now parsed, separated, and colored red or blue depending on which host sent the data. No more digging through raw tcpdumpoutput to find where one packet ends and another begins! Packets are now visibly separated. Server packets use red text. Client packets have blue text.
  2. Packet Data now includes the content of any IDS rules triggered by the flow. Ever opened the Packet Data for a flow and forgotten which alert you were investigating? We now include the full content of all IDS rules that triggered, along with the packet data for the flow.
  3. Packet Data Matcher! This was the most exciting feature to add! The new Packet Data page can highlight the specific content in a packet that caused an IDS alert to trigger! Matching packet data is highlighted in red or blue (depending on whether the packet was sent by the server or client) and includes numbered markers for each condition in the triggered rule. You can even hover your mouse over these markers to see the specific content or pcre condition that matched, along with the condition’s modifiers!  For example, the screenshot below shows the packet data for an event matching 1.2003492: ET MALWARE Suspicious Mozilla User-Agent - Likely Fake (Mozilla/4.0):Example of Packet Data Viewer with Packet Data Matcher results.The Packet Data Matcher works best for text flows, such as HTTP requests and responses. For encrypted or binary flows, we will still attempt to match against non-printable characters by using other conditions and context, but some matches might be inaccurate.

This feature is available to all MetaFlows customers starting today. We look forward to your feedback. Happy hunting!