100PB 100G PACKET CAPTURE

As the final days of 2016 close, we take a look at the future of 10G to 100G Packet Capture in 2017

EXABYTE PACKET CAPTURE

It might sound outlandish but PetaByte 10G, 100G Packet Capture is already here and deployed begging the question of what does a 1 ExaByte Packet Capture system look like? To be clear 1 ExaByte is 1000 PetaBytes, which is 1,000,000 TB of storage. Or to put that in perspective, using the current maximum capacity 10TB (2016/12) 3.5" HDD`s results in:

1,000,000 TB / 10TB = 100,000 HDD drvies.

Using our latest 3U 320TB 2 x 10G Packet Capture system "fmad20F 320TB" which has 32 Drives results in a total of:

1,000,000 TB / 320 TB = 3,125 3U Packet Capture systems.
3,125 sytems * 3U = 9375 U
9375U / 42U = 233 42U Racks.

Where 233 Racks isnt that much space in your typical large 1000 - 3000 total racks data center. In total you could deploy 1 ExaByte packet capture system now if you have the cash.

NOTE: Not sure what the correct terminilogy, 1 EB (ExaByte) = 1000 PB, and 1 EiB = 1024 PiByte etc etc.

EXABYTE PACKET CAPTURE SYSTEM PRICE?

So great, cool, fanstaic a 1 ExaByte 10G Packet Capture system is possible, doing the math it works out 3,125 systems * $49K USD (fmad20 320TB) = $156M USD. Which is great if you`ve got a spare $156M free in your budget, hell if you make such an order an automatic 50% discount is applied (coupon code 31337CAPTURE) making it "only" a $78M Purchase Order! Pretty cool but that kind of budget is way outside the range of most companies and even goverments. Which leads us to the next question of for that 50% discount $78M USD order what do you get?

Firstly number of full 10G line rate ports is:

3125 * 2 = 6250 x 10G ports

With a total aggregate capture bandwidth of:

62.50 Tbps.

Yes, thats 62.56 TeraBits full line rate no drop packet capture! Thats pretty cool, actually way beyond cool but its certainly excessive for any sort of enterprise.

NETWORK CYBER SECURITY

The only reason for building out such a massive packet capture system is network security. In the words of Rob Joyce, who is head of the NSA`s "Tailored Access Operations", the US government’s top hacking team who are responsible for breaking into the systems of its foreign adversaries. He said one of their worst nightmares is an "out-of-band network tap". This means recording every single byte of data that moves on your network, and more importantly keeping that data around for as long as possible.

The annalogy is "Security Camera for your Network". What its doing is recording everyone (and every thing/device friendly and unfriendly) that enters and leaves your network. Its like a Security Camera recording 24/7 everyone who enters and leaves the building, for your network. Every device that transmits or receives data gets stored and logged in the 10G 100G Packet Capture system. So WHEN, not IF, there`s a network security breach, you have resources to understand what happened and how. Exactly like Security Camera footage of a bank robbery.

With that in mind our 1 ExaByte 100G packet capture system starts to look a bit different. No longer need 62.5Tbps of raw capture bandwidth and instead need only need a maximum of 200G packet capture bandwidth, but with huge amounts of storage. The breakdown is 2 x 100G ports capturing say all WAN traffic.

With a 30 day retention window, the numbers work out to be:

200Gbps * 24H : 2.160PB / day
30 Days * 2.16PB : 64.8PB / month
64.8PB / 160TB FMAD Storage Node : 405 Nodes
405 Nodes * 1.5U : 608U
608U / 42U = 15 racks

In total Duplex 100Gbps link full packet capture for 30 days, takes about 2 rows in a Datacenter and consume 64PB / month.

Pricing:

2 pcs : FMADIO 100G : $100K USD
405pcs : FMADIO Storage Node : $10K USD
12 pcs : FMADIO Storage Farm Switch : $25K USD

Total: $4.550M USD
(2 x 100 Gbps Packet Capture Full 30 days retnetion, 64PB Storage)

Not too bad all things considered. That kind of pricing is way outside the scope of most budgets. Its also way over specification, a typical network the 24H average link utilization can easily be 10% so lets crunch the numbers.

200Gbps * 24H @ 10% utilization : 216TB / day
30 Days * 216TB : 6.48PB / month
6.48PB / 160TB FMAD Storage Node : 41 Nodes
41 Nodes * 1.5U : 62U
62U / 42U = 1.5 racks

In total full Duplex 2 x 100G packet capture with 10% 24H average network utilization for 30 days, takes up around 1.5 racks in a Datacenter and consume 6.4PB / month.

Pricing:

2 pcs : FMADIO 100G : $100K USD
41pcs : FMADIO Storage Node : $10K USD
2 pcs : FMADIO Storage Farm Switch : $25K USD

Total: $660K USD
(2 x 100 Gbps Packet Capture 10% 24H Network Load, 30 days retnetion, 6.4PB Storage)

Fairly reasonable all things considered, particularly if your a high value target.

HOW ABOUT 2X40G OR 8X10G PACKET CAPTURE?

To bring it down to a more reasonable level, lets say we have 2 core switches running 40Gbps uplinks, each with a single 40Gbps SPAN/Mirror port to feed our our capture systems. That makes the theoretical maximum one day packet capture size 2 x 40Gbps * 60 sec * 60 Min * 24 hours = 864 TB of storage for a single day. Lets round it up to make the math simple and say its 1PB / day. If our target is 30 days of data, we need 30PB of storage.

Using our ultra dense 1.5U 16 bay storage node filled with 10TB drives each
1.5U 16 bay * 10TB = 160TB
30,000TB / 160TB = 187 nodes.
187 nodes = 280U
280U = 7 Racks

Pretty cool for 30 P-e-t-a-B-y-t-e-s of storage!

Pricing that out we need:
1 pcs : FMAD80 capture system : $100K USD
187 pcs : FMAD 1.5U Storage Node : $10K USD
6 pcs : FMAD Storage Farm Switch : $25K USD

Total: $3.5M USD
(2 x 40Gbps Packet Capture Full 30 days retnetion, 30PB Storage)

While its expensive, the total price is within the budgets of a large enterpris. Particually if your a high value target enterprise.

80GBPS 10% AVERAGE 24H NETWORK LOAD

While the above is the maximm cost for a full 80Gbps packet capture system with 30 days retnetion. In practice the actual 1 day packet capture size is significantly lower. Most networks when averaged over 24 hours the utilization is only a fraction of the theoretical maximum. Using a 10% average network utilization for the same example results in something far more reasonable.

2 x 40Gbps, 80Gbps total = 1 PB/day
2 x 40Gbps, 80Gbps 10% utilization = 100TB / day
30 days @ 100TB / day = 3PB total storage
3,000 TB / 160TB FMAD Storage Node = 18 Storage Nodes
18 Storage Nodes = 28U

Meaning a 80Gbps system @ 10% 24H average network utilization, with 30 days storage. Fits entirely within a single 42U rack.

Pricing:

1 pcs : FMAD80 capture system : $100K USD
18 pcs : FMAD 1.5U Storage Node : $10K USD
1 pcs : FMAD Storage Farm Switch : $25K USD

Total: $300K USD
(2 x 40Gbps Packet Capture, 10% 24H Average Network load, 30 days retention, 3PB Storage)