Being able to go back and look at the payloads or files transmitted on a network is extremely useful for several reasons:
- If you do not have the payload, you cannot really prove malicious intent, and legally you are on the hook.
- Payloads/Files are the ultimate forensic tool to decide if a particular incident is a false positive or a true positive.
- In more advanced systems payloads can also be used to find false negatives (things should have caused a security event but did not).
Obviously logging all data transferring on a network is challenging because disk space is limited and disks are relatively slow.
The MetaFlows Security System Logging Approach
Our overall approach to overcoming logging limitation is:
- We store Payloads/Files that are associated with a specific security alert (using the time and the source/destination addresses and ports for identification)
- When logging proactively (to also see Payloads/Files that do not involve a security alert), keep the disk at 90% utilization or below a certain number of Gigabytes by deleting the older logs.
This scheme gives you certainty of access if there is an incident and a time window to go back in time to look for certain things that might have been overlooked.
The Logging and File carving system has been vastly improved by the following:
- We now index the packets based on IP addresses using a proprietary approach. Instead of looking for particular packets in a big bucket full of files, the files are divided in smaller buckets each representing a subset of the addresses. This indexing scheme slows down packet logging a bit but makes looking for packets about 200 times faster!
- We added the ability to specify user-defined logging policies. Once a policy hits, the logging system prioritizes all packets for the matching policy and stores the Files/payloads in a separate high-priority repository which takes precedence over the normal logging. We will make a separate announcement on the policy specification because it is quite powerful and complex, and requires a dedicated post. For now, the only logging policy is to prioritize any packets involved in high priority events. In the future users will be able to customize more precise ad-hoc policies based on IP addresses, ports, and type of alerts.