How to Deploy Open Source Network IDS/IPS in AWS

Open Source Network IDS/IPS in Amazon AWS

The easiest way to deploy a Network IDS to monitor your AWS instances is to setup a Linux security gateway. It does require some amount of IP networking knowledge but it is a very flexible way to manage your cloud assets as if they where in your LAN.

The EC2 security gateway routes IP traffic between the VPC and the Internet and therefore has complete visibility of the full-duplex traffic to and from your protected instances. The Network IDS running on the EC2 gateway instance will then allow you to identify and shut down threats as if it was deployed in a physical network.

Setting up a Linux Security Gateway in AWS

Create a VPC

Launch a VPC (Amazon’s virtual private cloud network) and give it a non-routable network range (ex. 10.0.0.0/8). Your VPC will need a private subnet (ex. 10.1.1.0/24) and a public subnet (ex. 10.1.100.0/24), if you do not already have two subnets then go ahead and create them.

Set up the gateway in AWS:

Launch a Linux EC2 instance on the public subnet of your VPC to be your network gateway, this will probably be the only instance on the public subnet for most deployments. Any Linux OS should be fine, but we prefer and use examples from CentOS.

Your gateway instance will need to be assigned at least one Elastic IP Address (EIP), this will be the public address that people will use to reach your network and the gateway will map that address to the correct instance on the private subnet.

You will need to modify the network adapter for your gateway instance to DISABLE src/dst Checking, this is required for it to properly function as a router.

Configure the gateway as a Router

After it starts, configure the gateway as a router for your private subnet. Execute the following commands assuming your private network subnet is “10.1.1.0/24”:

sudo -s
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -s 10.1.1.0/24 -j MASQUERADE

The above commands first give you a root shell (required to make networking changes), second tell the system to forward network packets that are destined for other networks, and third act as the source for all network traffic originating from your private subnet.

Add additional IP addresses on the public subnet (if needed):

EC2 will automatically assign an address to your instance, that is part of the public subnet, once it is launched. Each instance can have additional IP addresses on the public subnet.

For each of these IP addresses you can assign an Elastic IP Address to correspond to it, thus allowing your router to receive traffic for multiple public IP addresses and route it to multiple internal private hosts. Limits may apply depending on the type of instance you choose.

Set up the routing tables:

The public subnet should have a default route (0.0.0.0/0) to an amazon Internet Gateway device. If your VPC doesn’t yet have an internet gateway, you will need to add one for the public subnet.

The private subnet should have a default route (0.0.0.0/0) to the public facing interface id of the gateway instance. Do not add a route for your private subnet to an amazon Internet Gateway Device, otherwise they will route through it instead of your Linux gateway.

Launch the instances to be monitored

If you haven’t already, launch the EC2 instances that you wish to be monitored in the private subnet.

Add port forwarding

For each of the private subnet instances, add port forwarding rules to the iptables on your linux gateway for their publicly accessible services. You can follow these instructions to do that https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/4/html/Security_Guide/s1-firewall-ipt-fwd.html

Add Network IDS software

Once you have the traffic for your Amazon EC2 assets going through your own Linux gateway, you can deploy any traditional IDS systems in order to monitor your traffic. For example, in our example scenario, the gateway interface for the private subnet is “eth1”, and so we can invoke Snort to monitor all of our amazon traffic by pointing it to that interface:

# snort -f -c /nsm/etc/snort.serv.conf -A console -y -i eth0 --daq-dir /usr/local/lib/daq --daq pfring --daq-var clusterid=88
Running in IDS mode

--== Initializing Snort ==--
...
Commencing packet processing (pid=22129)
Decoding Ethernet
12/03/18-14:51:21.844473 [**] [1:2022775:1] ET USER_AGENTS BLEXBot User-Agent [**] [Classification: Misc activity] [Priority: 3] {TCP} 148.251.139.168:52916 -> 10.10.1.253:80
12/03/18-14:52:40.396441 [**] [1:2025534:10000] ET WEB_SPECIFIC_APPS Drupalgeddon2 8.3.9 8.4.6 8.5.1 RCE Through Registration Form (CVE-2018-7600) [**] [Priority: 0] {TCP} 45.37.49.53:35458 -> 10.10.1.253:443
12/03/18-14:52:40.483478 [**] [1:2025534:10000] ET WEB_SPECIFIC_APPS Drupalgeddon2 8.3.9 8.4.6 8.5.1 RCE Through Registration Form (CVE-2018-7600) [**] [Priority: 0] {TCP} 45.37.49.53:35458 -> 10.10.1.253:443
...

Qradar Support

The MSS now fully supports the Qradar SIEM from IBM in CEF log format. Qradar is an excellent SIEM but requires classifying and mapping every event type it sees to an internal category. Qradar comes with a large number of common IDS rules (~50,000) already classified but not mapped.  Besides having to manually map all these rules  Qradar-MSS users  would also need to continuously create additional Qradar IDs (Qids) to map to the much larger rule set used by the MSS (which  changes daily). All this required a mechanism to update Qradar dynamically as new rules are published. With this update released today, no manual classification or mapping operations are necessary.

The MetaFlowsCEF log source automatically parses the 13 event types generated by the MSS and presents them in the Qradar default view. All MSS events are automatically mapped to new or existing Qids without any user manual operations. This makes the Qradar SIEM much easier to use.

To setup Qradar for the MSS perform the following steps:

    1. Download the MetaFlowsCEF log source https://nsm.metaflows.com/sensordevicetype-search-ContentExport-20180809173340.zip to the Qradar box
    2. Import it with the command /opt/qradar/bin/contentManagement.pl -action import -f sensordevicetype-search-ContentExport-20180809173340.zip
    3. Verify the import was successful and assign the MetaFlows sensors to this log source. Also make a note of the log source ID assigned by Qradar to the MetaFlowsCEF log source (something like 400[1-9]).
    4. Edit the file mss.sh of all sensors and add the line export QRADAR=1.
    5. On one of the sensors you designate as the main Qradar updater, create the file /nsm/etc/qradar.ini to allow the sensor to communicate to the Qradar server (see an example below). Also add the line export QRADARLOGSOURCEID=<logsourceid>; where <logsourceid> is the number you noted in step 3. Probably something like 4001, or 4002, etc..
    6. Restart the sensors

Sample qradar.ini:

[DEFAULT]
certificate_file = /nsm/etc/qradar.pem
auth_token = f3f1201b-3562-46d1-9b8b-9a1623870000
server_ip = 123.52.215.20

Where:

/nsm/etc/qradar.pem is a copy of the file located at /etc/httpd/conf/certs/cert.cert on your Qradar box
auth_token is obtained from your QRADAR application
server_ip is the IP address of your Qradar box.

The Qradar updater sensor will automatically add to Qradar new IDS rules added by the sensor’s rule update  (which will be the same across all your sensors). This will happen through the Qradar API in the background as the sensor is running. The first time, the updater is run, it will have to catch up with about 50,000 definitions; so it will take many hours. Subsequent updates will take less time.

After each Qradar update, the email associated with the sensor owner will receive a summary of the update process.

Qradar integration is a bit complex; so do not hesitate to contact support@metaflows.com for any questions.

 

Search In Packet Logs

You can now search for arbitrary strings in the historical packet logs directly. The only requirements for this search is at least 1 IP address in addition to the search string.

For example in the search below we are looking for the IP address 139.182.44.203 in any packet either sent or received by the host 23.208.142.28. The search is also restricted to an hour worth of packets on 5/7/2018.

searchpayload

So why would you look for an IP address string in the packets? Well, this is normally done when there is more than one proxy and the system is not able to properly identify the proxy chain. In that case the offending IP will be recorded in the x-forwarded-for field of the http headers. Once you find the headers, you can find the real flows and then search again to get the data exchanged specifying the source and destination ports.

But this search feature is much more powerful than that; in fact you can also look retroactively in your packet history using full PERL regular expressions!

If you reached this far in this post, and you are an expert user, you will be wondering about the example above. The search string above would actually match more than  139.182.44.203 because the dots really mean any character (for example 139a182b44c203 would also match). To be more precise you would need to enter:

 139\.182\.44\.203

But suppose you wanted to match a specific set of IP addresses

139.182.44.203
139.182.44.205
139.182.44.206

Using a regular expression you could search for:

139\.182\.44\.20[356]

Just imagine what you could search for when you are hunting down specific strings or patterns.  So, this little new feature (also available through the CLI interface as the option -Q) should really expand the power of our historical packet logging system. It will let you easily dig in your network history for hidden clues of what happened in the past.

WannaCry Ransomware Advisory

It has been all over the news this weekend, a surge in Ransomware under the name ‘wannacry’ that has the potential to cripple large portions of networks due to the way that it spreads.

This is a pretty stealthy piece of malware at the network level, little to no CnC has been confirmed, but at an individual level it doesn’t behave much differently from any other Ransomware that we have seen in the past.

What distinguishes WannaCry is that it has a secondary infection vector that prior Ransomware variants lacked. Like any other, the primary infection vector appears to occur via email attachment (zipped javascript). However, once a machine is compromised it begins to behave more like a worm, able to exploit SMB (windows file sharing) on any systems that it can reach in order to spread its self.

This worm like behavior makes it particularly dangerous. While usually* smb (port 445) is not accessible from the outside world, it is often completely unrestricted within a local network, allowing one infected machine to spread the Ransomware across an entire site.

* This is your reminder to do double check firewall rules and run some external scans to make absolutely certain your windows file shares are not reachable from the outside world.

 

The following signatures are currently indicators to look out for:

2024218: ET EXPLOIT Possible ETERNALBLUE MS17 Echo Response
2024291: ET TROJAN Possible WannaCry DNS Lookup (trojan.rules)
2024292: ET INFO Bitcoin QR Code Generated via Btcfrog.com (info.rules)

MetaFlows has added 2024291 to our priority alerts category, and may also add 2023218 to add an extra level of alerting for these events.

 


Many of the windows related scan rules have been updated, and may be treated with greater suspicion, but are not alone indicators of this malware:

2001569 – ET SCAN Behavioral Unusual Port 445 traffic Potential Scan or Infection (scan.rules)
2001579 – ET SCAN Behavioral Unusual Port 139 traffic Potential Scan or Infection (scan.rules)
2001580 – ET SCAN Behavioral Unusual Port 137 traffic Potential Scan or Infection (scan.rules)
2001581 – ET SCAN Behavioral Unusual Port 135 traffic Potential Scan or Infection (scan.rules)
2001582 – ET SCAN Behavioral Unusual Port 1434 traffic Potential Scan or Infection (scan.rules)
2001583 – ET SCAN Behavioral Unusual Port 1433 traffic Potential Scan or Infection (scan.rules)

There are likely to be more updates and more information soon as researchers have time to study the samples collected so far.
Our primary signature provider, Emerging Threats, maintains a mailing list where these issues are discussed as they unfold.
https://lists.emergingthreats.net/pipermail/emerging-sigs/2017-May/028122.html
https://lists.emergingthreats.net/pipermail/emerging-sigs/2017-May/028113.html

Websockets Are Here

Adobe Flash is one of the original sins. It is everywhere and yet it is a huge security risk. Websockets is an HTML5 standard that, for us, provided an alternative to Adobe Flash.

For now we support both. The browser will try to use Adobe Flash first, and if it is not present or it is disabled, it will try using Websockets (which are hard-coded in your Browser). I you want to keep using Adobe Flash, you do not have to do anything; things should keep working as before.

If your sensors are configured as clients, and you do not want to use Flash anymore, just disable it and the Browser will do the rest. You will be using MetaFlows SSL certificate.

If your sensors are configured as servers, and you do not want to use Flash anymore, well, it’s a bit of work to use Websockets because current Browser implementations do not allow self-signed SSL certificates (this is probably a good thing). To use Websockets on sensors configured as servers:

  1. Add your sensors’ static IPs to the DNS (like: <sensorname1>@mydomain.com)
  2. Generate a valid SSL certificates  that matches the DNS name in step 1 (cannot be self-signed). If you do not want to generate a separate certificate for each sensor, you can also buy a *.mysensordomain.com certificate to share by all your sensors.
  3. Bundle the certificates with the command:

# cat my_certificate.crt my_certificate.key bundle_certificate.crt > sensorcert

  1. Copy sensorcert to /usr/local/etc/ntop/sensor-server.pem on your sensors’ hard disk.
  2. Go to nsm.metaflows.com and replace the static IP address of your sensors as a server with the names you setup in step 1
  3. Make sure your browser can reach the sensors on ports 3009 and 3010
  4. Restart your sensors as a server with the command:

#/nsm/etc/mss.sh restart

Adding Websockets support was a fairly extensive change in our system; so there could still be some issues. As always feel free to contact us if you have any questions or you see any problems.

Thank you for exploring the unknown with us!

The MetaFlows Team.

Feature Highlight: Snort Rule Editor

Recently, the Snort Rule Editor as a part of the Rules Management Interface has been updated.  This redesign allows for increased flexibility and provides the user with more of a handle on the IPS rules settings.

How It Works

Entering the Rules Management Interface is easy and can be accessed from two possible locations.  From the View Sensors page, the user can select the Edit Rules link to enter the Rules Management Interface. The user can also navigate to the Rules Management Interface from the Main Menu link.

Upon selecting the Rules Management Interface, the user will be prompted to select a sensor from the dropdown and then choosing the Load Rules button.  From there, the user will be able to manage the rule sets on a sensor-by-sensor basis.

Selection_003

After selecting a sensor, the user will have the option to modify properties of the rule sets, and issue commands to the sensors with the tools on the Menu Bar. In order for the changes made in the rules interface to take effect, commit the changes using the Save button.  (Please note that the Save/Cancel options only appear if the rules have been modified.) After selecting the Save option, a window will appear indicating that the changes are being verified. This process ensures that there are no issues with any changes that were made to the rule sets and that the sensor will correctly load the rules. Once the Save process is finished, the user will be prompted to reload the sensor.

To update the rule files, select the Get Updates button on the Snort Rules Controls. Most of the sensor controls buttons will be disabled/greyed out, and the Get Updates button will have a spinner icon until the update process finishes.  Next, the page will refresh to indicate if the updates were applied successfully. After the Get Updates process completes, select to Save the changes and Reload the sensor configuration for the changes to take effect.

Get_updates_1Get_updates_2

The Rule File List portion of the interface displays a complete listing of all the rule sets in the sensor configuration.  The local.rules file contains the pass rules (if any) that have been generated using the Tune IDS feature and any rules that were uploaded by the user.

Rules_listings_per_file

In the Rule List, each rule can be enabled or disabled by selecting the checkbox under the Active column next to each rule. If the checkbox under the Drop column next to a rule is checked, the sensor will drop flows that trigger the rule. Although we only recommend it for advanced users, each specific rule can also be edited by clicking on the rule itself. This will open the Rule Editor window, displaying the original rule, the editor, and some statistics that have been collected for that rule. Advanced Snort users can use the Rule Editor to make changes to the content of individual rules. When modifications are made, the Diff section will show changes to a rule since the most recent Save of the rule sets.

Manual_rule_editor

Pass Rules inform the IDS system that packet matching these rules should not generate alerts. These are helpful for eliminating false positives without having to disable the offending rule altogether. The Pass Rules have system-generated SIDs so that they do not conflict with the original rules that to which they refer. The user can utilize the tuning interface to add pass rules to the local.rules file. Please remember to Save any changes to the rule sets through the Rules Management Interface.

Selection_004

For users that want to run a reduced rule set for performance reasons, there is an Automatic Tuning option under the Bulk Edit menu. This option will disable rules that are unlikely to trigger based on our observations across all customer networks.  After clicking on the Automatic Tuning option, the changes will be merged with any prior changes that have made, meaning that rules that have specifically enabled will stay enabled. If the user wants to revert to a default minimal set, they should consider first using the Rules Defaults option.  Once the Automatic Tuning has processed the rules, it will then automatically fetch the latest updates and disable all of the appropriate rules.  To complete this process, the user will need to select Save so that the changes are committed.

Full color images and even more detailed instructions regarding The Rules Management Interface can found in the MetaFlows Wiki.

FireEye Foibles

On February 15th, Blue Frost Security released a statement regarding an analysis engine evasion that was identified in regards to FireEye’s virtualization-based dynamic analysis.  Their statement reads, “An analysis engine evasion was identified which allows an attacker to completely bypass FireEye’s virtualization-based dynamic analysis on Windows and whitelist arbitrary malicious binaries.”

We now have a signature to detect this method of evading FireEye:

2022554 || ET EXPLOIT FireEye Detection Evasion %temp% attempt – Inbound ||

Any customers who are also using the FireEye system may want to set up additional rank or email classes for this rule so that they can be alerted to malware that may be attempting to bypass their FireEye appliance. FireEye has released an update for this that users should apply immediately, if they have not done so already.  However, even once the issue has been patched, seeing the attempt of this bypass can be a valuable indicator of malicious activity on its own. This may be tried alongside future evasion attempts.

Predictive Global Correlation Feed

After months of data gathering, we turned on a new global correlation feature that complements the existing local multi-session correlation. The aim is to further tighten the net and catch more bad stuff while also decreasing false positives.

We now show the  ranking as total/global when we display an alert. When the global ranking is missing, it is because that event is only ranked locally and the global portion is unknown. When the total and global rank are the same (like 187/187 in the example below), it means that an event was ranked exclusively using global relevance and it would have been missed by the local analysis.

You can see the global ranks by going to the IDS rule management interface. IDS rules listed there will have the current global rank assigned to them for that day (if any).

blog

This additional information complements the local multi-session correlation analysis by trying to look at things from a global intra-domain prospective:

If a domain similar to yours has experienced a significant amounts of high-priority network security incidents involving a particular IDS signature, that signature will receive a positive global rank in your domain.

The key here is the word “similar”.  The events each customer generates are used to compute a similarity matrix that tells us how similar each network is to the others. Using this information, rather than recommending all high-priority signatures to all domains (we call this simple prediction), we only recommend what is most likely relevant to your domain (we call it predictive correlation).

 

Let us know how this works for you and if you have any questions.

Thanks!

The Skinny on CVE-2015-7547

While the DNS exploit CVE-2015-7547 was discovered a week ago, the code containing the flaw has been in use since May, 2008. CVE-2015-7547 works by allowing arbitrary code to execute on any system reliant on glibc by way of a malformed query response.  As discovered by Redhat Linux and Google, there are flaws in GNU C Library.  The GNU C Library connects to DNS to resolve names.  This problematic code effects all versions of glbc since 2.9 and allows for remote code execution.

We have seven signatures, the first of which was released the day after the exploit was discovered. We were able to push the beta version of the rule to our research partners immediately, and to all sensors during the normal daily signature update.

2022531 || ET EXPLOIT Possible 2015-7547 Malformed Server response || cve,2015-7547

2022542 || ET EXPLOIT Possible 2015-7547 PoC Server Response || cve,2015-7547

2022543 || ET EXPLOIT Possible CVE-2015-7547 Long Response to A lookup || cve,2015-7547

2022544 || ET EXPLOIT Possible CVE-2015-7547 Long Response to AAAA lookup || cve,2015-7547

2022545 || ET EXPLOIT Possible CVE-2015-7547 Malformed Server Response A/AAAA || cve,2015-7547

2022546 || ET EXPLOIT Possible CVE-2015-7547 A/AAAA Record Lookup Possible Forced FallBack(fb set) || cve,2015-7547

2022547 || ET EXPLOIT Possible CVE-2015-7547 Large Response to A/AAAA query || cve,2015-7547

          Signature 2022547 is currently triggering on multiple customer sites, but at least for now it is in low volume.  However, according to Dan Kaminsky, this is a threat that could swiftly escalate as more and more adversaries improve their attack strategies to increase the damage made possible by CVE-2015-7547.  Patching this particular bug is paramount, as well as continually monitoring your system for the exploit.

 

Measured Antivirus Effectiveness

I wanted to share with you some insight from the data that originated from our customers’ networks last week. This time, we wanted to provide some information on how different antivirus vendors perform on the .exe, .dll, .pdf, and .zip files seen around the world.

This table shows the relative hit ratio of all the antivirus vendors hosted by Virus Total on 697 confirmed bad files. You will notice that 43% of the time none of the antivirus products detected anything. The top performer is McAfee-GW-Edition with a 37% detection rate.

Looking at the types of samples detected, one can also consider which Antivirus Vendors were able catch the worst malicious code. We assigned an Average Priority of 1 to spyware or unwanted software and an Average Priority of 100 to known Trojans or unclassified malware.  Then, we multiplyed the Average Priority by the Detection Rate, giving rise to the Severity column. This column shows which Antivirus Vendors found the most dangerous code. This week Arcabit wins with a Detection Rate of 29%, an Average Priority of 30.17, and a Severity of 8.96.

Antivirus VendorTrue PositivesAverage Priority Detection RateSeverity
None3000.430416 (mss)
Arcabit20730.170.2969878.96
F-Secure19228.840.2754667.95
ESET-NOD3220524.180.2941187.11
AVG12937.070.1850796.86
Avast20023.770.2869446.82
Qihoo-36020722.520.2969876.69
GData22320.090.3199436.43
McAfee-GW-Edition26416.750.3787666.34
CAT-QuickHeal16227.280.2324256.34
VIPRE17223.450.2467725.79
Cyren20119.720.2883795.69
Panda8546.420.1219515.66
F-Prot16024.510.2295555.63
ClamAV6263.270.0889535.63
Fortinet10529.290.1506464.41
McAfee11725.540.1678624.29
Avira21012.790.3012913.85
Bkav8330.820.1190823.67
MicroWorld-eScan16215.060.2324253.50
BitDefender16115.140.2309903.50
Emsisoft16015.230.2295553.50
CMC24100.000.0344333.44
Kaspersky8627.480.1233863.39
TrendMicro6337.140.0903873.36
Ad-Aware14016.560.2008613.33
Ikarus20910.950.2998573.28
AVware9523.930.1362983.26
Comodo6926.830.0989962.66
Sophos7720.290.1104732.24
Rising1957.090.2797701.98
Tencent5024.760.0717361.78
ALYac1089.250.1549501.43
Microsoft2536.640.0358681.31
K7AntiVirus1095.540.1563850.87
DrWeb1343.960.1922530.76
Malwarebytes2221.890.3185080.60
K7GW1203.480.1721660.60
Antiy-AVL745.010.1061690.53
Symantec1611.610.2309900.37
VBA32534.740.0760400.36
nProtect1613.380.0229560.31
NANO-Antivirus762.300.1090390.25
SUPERAntiSpyware383.610.0545190.20
Jiangmin383.610.0545190.20
Zillya1311.000.1879480.19
ByteHero425.750.0057390.15
Baidu-International831.000.1190820.12
AhnLab-V3801.000.1147780.11
Agnitum571.000.0817790.08
ViRobot121.000.0172170.02
AegisLab91.000.0129120.01
TotalDefense21.000.0028690.00
Zoner11.000.0014350.00
Alibaba11.000.0014350.00

Our sandbox was able to detect the remaining samples (the missing 43%).

antivirus

The bubble graph above illusrates the Severity (Detection Rate * Average Priority) verses the Prevalence (Detection Rate * Total Priority). The detection rate is encoded in color and the size of the bubble is proportional to how many customers saw the malware.


If you are curious about more statistics like this, you can visit https://www.metaflows.com/stats/ (best viewed on a desktop) for a ton of additional information. If you want a quick fix, watch some of our videos at https://www.metaflows.com/saas/.