What is 10Gbit Line Rate?

Posted by fmadio | 100G Ethernet

Often in networking technology circles we hear the term "line rate" such as "10Gbit line rate" which roughly translates to network traffic at the maximum capacity and in this case at 10Gbit/second. While it sounds clear, the actual data transfer rate can be a bit non-intuitive.

  • what is 10G

The first thing to know about 10Gbit line rate is, its not exactly 10,000,000,000 bits / second but slightly higher. How much higher? 10.3125Gbit`s to be precise which we discussed in a previous post HERE. The summary of that post is, to achieve 10.0000Gbps of actual data you need an extra 312.5Mbps worth of bandwidth for control data. The post goes into gory detail about how data is actually encoded onto the wire, which might surprise you.

The key point is, that 64B ICMP ping data you see in tcpdump "on the wire" is only part of what is actually going down the wire. We discussed how such a packet gets translated into XGMII in a previous post All about XGMII which goes into the details of the first 10G ethernet layer of abstraction. The summary is each data packet is surrounded by an ethernet preamble, frame check sequence, ethernet epilogue and the minimum inter-frame-gap, as shown below.

  • 10g explained packet

  • In this example each section is as follows:

    Preamble blue 8 bytes
    Payload green variable
    Frame Check Sequence yellow 4 bytes
    Epilogue purple 1 bytes
    Inter Frame Gap (+Epilogue its 12B)Red 11 bytes

    When a vendor (such as us) boasts about "line rate" performance, it means the device can process the maximum amount of data payload that can be serialized using the protocol shown above. So lets makes this more concrete and calculate what "10Gbit line rate 64B packets" actually means, as its the gold standard as far as network devices are concerned.

    10Gbit Line Rate 64Byte

    Starting with the 64B packets part, which means 64Bytes between Ethernet Preamble and Ethernet Epilogue and translates to 60B of data and 4B of FCS. This is not as typically assumed 64B worth of data payload!

    Preamble blue 8 bytes 8 B
    Payload green 60 bytes68 B
    Frame Check Sequence yellow 4 bytes72 B
    Epilogue purple 1 bytes73 B
    Inter Frame Gap (+Epilogue its 12B) Red 11 bytes84 B

    Which makes a 60 byte data packet, bloat out to 84B in total on the wire. Next we calculate total number of these packets that can be pushed at exactly 10.00Gbps. 10.00e9 bits / (8 bits * 84 bytes) = 14.88e6 packets And we see that familiar 14.88M packets per second number that everyone likes to write. That means on a 10 Gbit link the maximum number of packets with 60 bytes of data (plus 4B FCS) is 14.88M packets. But what is the actual data transfer rate ? 14.88e6 packets * 60 bytes * 8 bits = 7.14Gbps So while we`re sending 60 byte packets as fast as possible, consuming the full 10Gbps worth of bandwidth we`re only sending 7.14Gbps worth of data!

    10Gbit Line Rate 1500 Byte

    From the above is pretty clear using 64B packets to acheive maximum bandwidth aint such a good idea, so lets see what the numbers are for a 1500 byte packets.

    Preamble blue 8 bytes 8 B
    Payload green 1500 bytes1508 B
    Frame Check Sequence yellow 4 bytes1512 B
    Epilogue purple 1 bytes1513 B
    Inter Frame Gap (+Epilogue its 12B) Red 11 bytes1524 B

    .. and crunching the above to get the number of packets per second @ 10.0Gbps

    10.00e9 bits / (8bits * 1524 bytes) = 820,209 packets

    which results in the following amount of data bandwidth

    820,209 packets * 1500 bytes * 8 bits = 9.85 Gbps

    Meaning sending 1500B packets at line rate down a 10Gbps link results in 9.85 Gbps worth of actual payload data. Certainly alot better than 7 Gbps but even with 1500B packets you can`t hit the full 10.0Gbps worth of data.

    TCP 10Gbit Line Rate 1500 Byte Packets

    ... but above calculation of 9.85Gbps is kind of miss leading, as its only the raw ethernet frame bandwidth. In the real world packets use higher level protocols such as FTP, SSH, HTTP, HTTPS, etc etc and 99% of the time over a TCP connection. Thus a new question: using a 1500 byte ethernet frame, what is the real payload bandwidth of a TCP packet?

    10g explained packet

    Preamble blue 8 bytes 8 B
    Ethernet Header green14 bytes22 B
    IPv4 Header green 20 bytes42 B
    TCP Header green 20 bytes82 B
    Data Payload green 1426 bytes1508 B
    Frame Check Sequence yellow 4 bytes1512 B
    Epilogue purple 1 bytes1513B
    Inter Frame Gap (+Epilogue its 12B) Red11 bytes1524 B

    As you can see from the table above, its 1426 B of data payload wraped in 1524B worth of 10G bandwidth. If we do the math on this

    10.00e9 bits / (8bits * 1524 bytes) = 820,209 packets

    With the resulting effective TCP payload bandwidth

    820,209 packets * 1426 bytes * 8 bits = 9.36 Gbps

    Its not bad but we`ve effectively lost close to 1Gbps of bandwidth from the ethernet + IP + TCP encoding. So if your getting high 8.xx Gbps numbers in a TCP bandwidth test, your doing pretty dam well.


    Its nice to think your fancy new 10Gbps link can push 10Gbps worth of market data, porn and lol catz (in that priority) but it never can and never does. Even the above is an oversimplification as technically its an "average inter frame gap of 12 bytes". As almost all hardware has alignment restrictions on when a packet can start/stop within a XGMII cycle. For example our own FPGA based traffic generator outputs 8B IFG, then 16B IFG, then 8B IFC.. etc making the average IFG (8 + 16 + 8 + 16 ...) / Packet Count = 12B.

    Hope that clears up any confusion over what 10Gbit Line rate is and 10G ethernet != 10Gbps of data.