PACKET CAPTURE FILE SYSTEMS
One of the interesting and non-obvious problems of a packet capture system is engineering the underlying filesystem to store all data. You might think writing to a standard EXT4 / XFS / JFS / pick your favorite file system is enough. However this is a huge huge red flag, as there is no guarantee on the sustained write performance, which can result in packet loss.
The above access pattern is for EXT3 file systems. Its scattered all over the place.
The above access pattern is for EXT4 file systems. The allocation is clearly far better and significant improvement over EXT3
For JFS file systems, its a some what bizarre scatter bomb allocator.
And the champion XFS file systems. This is the best allocation scheme out of all the standard file systems. If your doing DIY packet capture XFS is probably your best bet.
And finally, the above access pattern is for Reiser file systems. It appears to be maximizing the number of disk seeks, to minimize the performance! To be fair... its designed for maximum space efficiency for small files.
SEQUENTIAL WRITE PERFORMANCE
If the above hasn`t scared you enough about the dangers of using a standard file system, perhaps the raw sustained write throughput of non sequential write access will. (Data from our R&D work-in-progress 100Gbps packet capture system)
The graph below is testing the raw hardware performance by writing a single 1TB file. However to replicate a fragmented file system the write head seeks (jumps) every 1MB to 1TB of written data (X axis). The Y Axis is the final 1TB write throughput in Gbps. This shows maintaining sequential disk access is critical to guarantee sustained high performance.
In the chart above on the very lefthand side, we see barely 10Gbps throughput for extremely fragmented systems. It then then continues in a linear ramp to just under 70Gbps sustained throughput as the writes become less and less fragmented. After that it wobbles around and finally hits just under 90Gbps sustained throughput for a 1TB write with 0% fragmentation.
FILE SYSTEM FOR PACKET CAPTURE
For packet capture we are blessed with extremely simple requirements. Data is always written sequentially across the entire disk in a simple First-In-First-Out allocation system. There`s 0% fragmentation, 100% Sequential write access with no complex directory structure.
The above shows how our captures are allocated on the disk, its a very simple linear layout with a 100% guaranteed sequential access pattern. The only interesting point is when the storage eventually fills up (shown below).
When the storage system fills up our file system simply overwrites the oldest packet data, shown above. The newly captured data (Blue Arrow) overwrites the oldest data on the system (purple arrow). Eventually "CaptureA" will have no data and is fully removed from the system.
SUMMARY
Packet capture storage performance is entirely based on:
a) raw hardware Sequential Write performance
b) the applications disk access pattern.
Our 100Gbps, 40Gbps, 20Gbps, 10Gbps Packet capture system`s write directly to the raw hardware using a customized file system that guarantee`s sequential access in all situations. Coupled with excellent raw hardware performance this enables full 10Gbps to 100Gbps line rate packet capture to disk across the lifetime of the device, with no fragmentation.
Happy packet capturing!
