When is Packet Capture not just Packet Capture?
At FMADIO we build 10G 25G 40G 100G 200G line rate full packet capture systems. We have been building these systems for many years and through our valued customers, we have learnt there is a fundamental difference between what’s traditionally known as packet capture and what FMADIO provides.
Packets at Rest?
We started out targeting the highest capture to disk rates without loss, as at the time all other packet vendors were failing to achieve this.
"FMADIO is Lossless Packet Capture"
We got there ahead of all other vendors and even now, 5 years later our initial 100Gbps to disk @ 149Mpps without loss is something many vendors still have problems with.
“Packets at Rest” is your typical packet capture solution; a capture card, typically FPGA-based, with a large amount of attached storage. Packets at Rest means once the data has been captured it remains “at rest” where it was captured and it is only accessed now and again on an ad-hoc basis.
Doing this at high speed with no loss and fast retrieval is a difficult problem, particularly searching for and retrieving the proverbial “needle in a haystack”. However, the use cases for this are limited, typically,
- network troubleshooting
- cyber forensics (who talked to who at what time)
This is typically what the industry talks about when discussing "Packet Capture", Bob cannot connect to server X at time Y. His IP address is 192.168.1.100, send us the PCAP around 12 noon on this day, so we can load it into Wireshark and see what’s happening.
It’s a fantastic way to understand what’s happening on the network; if you have the skillset to understand Wireshark it’s a very effective tool.
Yet, FMADIO devices can do so much more.
Packets in Motion
Over the years we have gone through different messages to try explaining to our customers what FMADIO can do beyond traditional packet capture. It started out as,
"FMADIO is a Packet Stream"
Unfortunately, this original idea did not really work that well. The idea is "Stream Processing" technically is about taking a large block of data X, reducing it to Y. Then taking that new block of Y data, running more processing on it to reduce it to Z amount of data. All the machine learning algorithms do this in some shape or form.
Effectively it’s a form of lossy compression. For example, we take 1TB of raw PCAP data, process it into flow data, then sample that flow data every 1 second. It’s a good way to take a vast amount of raw PCAP data and convert it into something that’s digestible and usable.
FMADIO PCAP2JSON is a great example of this stream processing. It generates about a 100:1 compression ratio, lossy of course, yet you mostly don’t care about every bit in a packet, just how many packets there are and where they are going to/coming from.
"FMADIO is a Packet Cache"
This has been fairly effective; it certainly changes customers' mindsets on what FMADIO systems do. It means FMADIO is not primarily used for long term PCAP storage. Instead, it’s used as a cache where it can capture high speed line rate traffic, run some operations locally on the cache of packet data, then move the packets to a downstream system for further processing or long term storage.
It kind of works, customers understand "Packet Cache" fairly well, it brings up a different mental image than a 20G NIC attached to 100KG worth of HDDs. Yet it doesn’t really explain what FMADIO provides.
"FMADIO Capture.Filter.Push"
Our systems can also push PCAP data with filters, compression, and anything else you can think of. This typically writes PCAPs over NFS or similar protocol to systems downstream. It sounds like a basic task, but the use cases for this are numerous. Plus, we can utilize all CPUs on a 100CPU 1U 200G Capture/Analytics system just running BPF filters and your favourite compressor, to process packets at a phenomenal rate.
Think about the CPU cycles required to compress 1TB of data. Then think of the amount of network bandwidth. The disk storage and ultimately dollars you save by running that filter and compression task on the FMADIO device is enormous. If we send your downstream systems PCAPs of exactly only the data they’re interested in, they don’t have to waste cycles, bandwidth, and storage on data that’s unrelated to their task.
"FMADIO is Packets in Motion"
And finally, we are back to Packets in Motion. It summaries our system fairly well vs Packets at Rest. FMADIO Packet Capture systems go well beyond what we traditionally know as packet capture by taking care of the initial costly processing to ultimately save our customers money.
Contact us for a discussion.
