PCAP NETWORK FLOW PROCESSING SCALING

Network Flow Performance Scaling

We`ve been hard at work to scale our PCAP to Netflow JSON code to work with FMADIO 100G packet capture systems, hard hard actually, not insanely hard but definitely requires a lot of low level architecture and high-performance computing software skills. What’s interesting is the code and hardware architectures keep getting closer and closer to looking like HDL/Verilog code. In some ways it makes sense, multi-core / mutli-cpu deeply parallel code needs to be written very differently, it certainly ain’t C code from 2001.

In any case quick background, pcap2json pcap2json is our utility to convert raw PCAP data in netflows with 100ms to 1sec snapshots. The format of the NetFlow is JSON format which is sent directly to an Elastic Stack cluster without any need for Logstash or other utilities. Its designed to push the captured data into Elastic Stack with low latency so you can graph all sorts of interesting stats to monitor your network.

One of the biggest problems is performance, imagine running a 100G Packet capture on a link, that has 1,000,000 unique flows every 100msec. Thats alof of data and traffic to monitor and getting it into a form that’s manageable. Using our 20G capture system the performance scales with the number of CPUs like shown below.

The graph above is generated using 100K flows / 100msec with 64B packets. Its the kind of worst case scenario of lots and lots of tiny packets all with unique flow signatures. As you can see the performance platues out around the 8 CPU mark.

NETFLOW MONITORING "NETFLOW"

One of the basic ways to monitoring traffic is via Network Flows / IPFIX where you put a device onto your network that calculates statistics for each and every network flow. A network flow in this case is a unique 7-tuple IP Src/Dst, Protocol, Port Src/Dst, ingress/egress port. Usually the TOR switch generates the flows, however there are a number problems with this.

FINAL THOUGHTS

Don't think of packet capture as a PCAP on simple magnetic storage. True in the past, but with high speed SSD drives so many things can be achieved now. Netflow Snapshot generation is a nice example of using FMADIO 100G Packet Capture systems to buffer, process and send processed data downstream for further processing.

Best still, you have full packet capture to fall back on for those hard to find deep dive investigations!