PCAP Push Downstream
Our FMADIO 100Gbps does full line rate 100Gbps 149Mpps without loss, but so what? Capturing the data without loss is only the start, making network recorder traffic accessible beyond a Wireshark session is just as, if not more important.
One popular option is to Push PCAP data downstream, typically in 1 minute slices, or 1GB slices using our fully Open Source pcap_split utility, see here
https://github.com/fmadio/pcap_split/
A typical splitter will split the data in the following way. Example below is splitting capture data in pseudo-realtime (10s seconds of lag) into 3 seperate Multicast groups and a catchall TCP group
FMADIO Push PCAP Downstream
To dig bit deeper into this, we have setup split in the follow way. Using fmadiocli command “show push-pcap” to generate the display
The above show 4 separate pushes, each with its own FilterBPF. This is the filter used to generate the outputting pcap. Its standard BPF (“just like tcpdump”) enabling a very wide range of filtering options.
Push BPF Filter Example Hilighted
Next is the name of the output file. In this case we are writing to a locally mounted NFS server, thats mounted on /mnt/remote0 on the FMADIO Capture Appliance. The full path on where to store the data is specified in the “Path” options shown below.
Push Path location
While the above is an NFS target, FMADIO Packet Capture systems support almost any target, such as NFS, SFTP, SCP, SAMBA, S3, FTP, ….. and many more.
Next is how to split the data, the following its split by 1 minute e.g. 60e9 nanoseconds. How it can be split is highly configurable, 1sec, 1min, 1hour, 1 day or 1GB, 10GB it really depend on how your downstream application is ingesting data.
How to split the PCAP
Finally we have how to name the output file. This is also highly dependent on the downstream applications. In the above example we chose Hour Minute, meaning it will be 8:00 -> 0800.pcap 08:01 -> 0801.pcap etc
Full documentation on the filename formats supported is here
https://docs.fmad.io/fmadio-documentation/configuration/automatic-push-pcap#filename
Finally the output of all of this ends up looking like the following x 4 (one for each target).
Example PCAP files pushed
There are other options such as gzip zstd lz4 compression, also format conversion PCAP -> PCAPNG or PCAP -> ERF during the process. Generally there’s no limit on the number of separate PCAP push or the complexity of the BPF filter. As a practical matter keeping it under 100 is recommended.
Any questions of comments please contact us, thank you.
