100G Packet capture arrives

Posted by fmadio | 100G Ethernet

Its certainly taken longer than expected but we can finally announce our full 100G Packet capture system! Its an incredible combination of FPGA`s, SSD`s and slick 2U 48 bay chassis all for a modest ~ $100K USD.

  • multicast join
100G Packet Capture

Finally there is a 100Gbe Packet capture solution! Our FMADIO100 system is a stylish 2U chassis with 48 hot swap-able 7mm SSD`s at the front, 4 hot swap OS drives at the back, 800W dual hot swap-able power supplies, all to keep you running 24/7 non-stop without interruption. Packet capture uses the latest Virtex Ultrascale FPGA from Xilinx and powered by Intel`s sever grade motherboard with dual socket Intel 2600 CPU`s.

fmadio 100G architecture

Capturing at 100Gbps

Its no easy task to get full 100G capture to disk with RAID5 up and running, it pushes the current DDR4 and PCI Express Gen3 to its limits. The trick is extensive use of the FPGA to perform all packet processing, leaving the CPU free for storage and IO processing. We`re currently only using 30% of the FPGA`s resources with plenty of room for new advanced features. First new features on our schedule are advanced pre-capture full line rate 100G packet filtering and sustained 100Gbps hardware packet compression to maximize storage capacity and SSD life.

fmadio 100G fpga

Above pic is the current FPGA utilization (aqua parts are occupied logic cells). As you can see current utilization is less than 50%.

Why use an FPGA? FPGA is critical for 100G packet capture, there`s absolutely no way to do sustained full line rate 148.88M packets per second processing on a CPU. Many will argue its theoretically possible to crunch on a CPU, there`s even demos using 24 core monster`s showing impressive thoughput speeds. But in practice with hard real-time deadline`s and small FIFO`s, it only takes a few L1/L2 cache misses to throw the SW off its game. Those cache misses cascade and ripples throughout the software stack eventually resulting in packet drops. (Hint: its all about memory hierarchy not CPU cycles). For FMADIO 100G systems all packet processing is performed in RTL on the FPGA which guarantees full sustained 100G line rate processing, no matter what you throw at the system.

100Gbps Packet Generator

But wait.. yes there`s more! All 100G system`s includes a FPGA based 100Gbps packet generator. It is a Layer 1, Layer 2 and Layer 3 packet generator with all the features you expect for a high quality packet blaster. This includes a Layer 1 PRBS-7, PRBS-15, PRBS-31 bit generator/receiver for testing 100G fiber and transceiver. And a Layer 2/Layer 3 100G packet generator for load testing 100G switches and circuits. And best of all you can simultaneously generate and capture at the same time - after all, this is exactly how we test our capture system.

fmadio 100G packet generator
Enter the Ether at 100Gbps

100G Ethernet deployment is quickly ramping, its no longer the domain of teleco backbones, $50K transceiver`s and deep undersea cables. As of now May 2016 a 100G SR4 QSFP28 transceiver is only ~$1500 USD now (2016/5). And finally with FMADIO100, full packet capture is possible, affordable and critical for network troubleshooting and of course the elephant in the room, 100G network security.

fmadio 100G packet capture front fmadio 100G packet capture rear

The Specs

Capture Speed
Sustained 100Gbps
Sustained 148.88Mpps

Capture To Disk
Sustained 100Gbps

Capture Interface
x1 100G QSFP28 Port

Generator Speed
Sustained 100Gbps
Sustained 148.88Mpps

Generator Protocol
Layer 1, 2, 3 Packets

Storage Size
24TB/48TB/96TB solid state

                                        Packet HW Time Stamp
3.2ns resolution

Packet Size
min 64B packets
max 9200B packets

Time Synchronization

External IO
x2 USB Ports
x1 DSub VGA
x1 Serial Port

x1 10G SFP+
x1 1G RJ45
x1 IPMI RJ45

                                        Rack Profile
2U Chassis 650mm deep

432mm x 88mm x 650mm
19" x 3.5" x 25.5"

30.0Kg / 66.0lb

Tooless 31" Rails

3x Hot Swap 60x60mm fans

800W Dual Hotwap PSU
(DC48V on request)