DEBUGING ETHERNET CRC FCS ERRORS

One of the great things about 10GbE, 40GbE and 100GbE packet capture is it captures everything, included network corruption! Errors can take the form of Ethernet Frame Check Sequence (FCS) which is a 32bit CRC calculated across the entire packets payload. This post discuss how we capture and some of the tools to help debug errors on the network typically seen on WAN links.

ETHERNET FRAME CHECK SEQUENCE

A quick recap on Ethernet plumbing. The foundation is OSI Layer 1, the raw electrical/optical specification that is a constantly varying stream of bits with no packets or error correction, think of a raw pipe flowing with water. Layer 2 segments this bit stream into distinct individual packet`s with clearly defined payload data and a much needed CRC. Think about that pipe of flowing water (Layer 1) feeding a watermill that segments and buckets (Layer 2) the water into unique packets.

The structure of this segmentation is shown above, I`m sure you`ve seen some or all of these components. Typically, network cards process and hide the "Ethernet Preamble", "Frame Check Seq", "Ethernet Epilog" and "Inter Frame Gap" data and only present the "Packet Payload" section of the data to Linux / System for further processing. However a NIC will only present the data to the system, IF the the Frame Check Seq (CRC) is good, as it implies the data integrity is valid .

What is a network FCS/CRC ? Briefly, when a packet is transmitted from a Switch or NIC the sender calculates a 32bit CRC value of the packet`s data and attaches it to the end of the packet on the network. This enables the receiver to validate the packets data, by calculating the exact same CRC from the received data and comparing it to what the sender attached to the packet. If the numbers match the packet`s data is good, if it does not the packet has been corrupted somewhere/somehow and is typically discarded by the NIC.

CAPTURING ETHERNET FCS

One of the primary uses of our 100G packet capture device is debugging 100G LR4 WAN links. As 100G ethernet within the datacenter is not common (2015) but 100G between DC`s is spreading like wildfire! As such its critical our tools capture and process 10G, 40G, 100G Ethernet FCS`s correctly and precisely. However if your building some DIY packet capture project, very few NIC`s will forward the FCS to the host system and specialized network cards are required for real full packet capture devices.

But can`t I use wireshark builtin FCS check logic? Absolutely, tho you still need the original FCS value on the wire from the NIC to compare with. Wireshark is a fantastic tool and we highly recommended it. But even if you have the FCS of all packets, loading a 1TB PCAP into wireshark... is going to a very long long time...

Our PacketScope Microburst analyzer can filter and process invalid FCS`s enabling you to extract only invalid FCS packets, typically a hand full out of billions of packets. You can also narrow it down further to specific time ranges or MAC addresses and even protocols depending on the level of wire corruption.

OUR FCS CORRUPTION TESTSUITE

We have a regression test specifically for capturing and downloading packets with invalid FCS. It uses the FMADIO FPGA based packet generator as you can not intentionally corrupt the Ethernet FCS on a regular NIC. The specific test corrupts 1/10 packets or 10% of 1,000,000,000 total packets with a payload of 64B and running at full 10GbE line rate. After packet generation and downloading the capture into Wireshark it looks like the below screenshot.

You can clearly see the FCS errors at regular intervals (1 out of 10 packets) marked in Black by Wireshark. Its a good test but in practice these kinds of errors are typically 0.0000001% of the total packets. With say ~100 corrupt packets in a 100GB capture. A 100GB capture is certainly a problem for Wireshark but our builtin FMADIO PacketScope analyzer handles 100GB with ease.

PACKETSCOPE INVALID FCS EXTRACTION

In the screenshot below a capture is loaded into packet scope and a filter rule to select all invalid FCS packets (highlighted in green) is applied using the following:

ether.fcs == fail

The PCAP is then downloaded using the button highlighted in Blue (above). With the system extracting only invalid FCS packets and utilizing full IO throughput of 10Gbps, 20Gbps even 100Gbps of SSD bandwidth. Its certainly a hell of alot faster than processing in wireshark. After download is complete you then load the PCAP into wireshark to inspect only the problem packets (see below).

SUMMARY

Its important your packet sniffer device captures line rate good packets AND bad/corrupted network packet`s on the wire. Without both its hard to see the full picture. Our FMADIO packet sniffer`s captures the good, bad and ugly packets to enable simple fast extraction of invalid FCS packets so you can troubleshoot the problem ASAP.

If you have any further questions or have questions on bad (and good) packets, contact us at any time.

... alterantively if your at this page looking for how to get wireshark to display only FCS errors, see below!

eth.fcs_bad

Happy hunting!