When I saw this article about CloudFlare's world-wide router meltdown, however, I immediately felt a bit smug about all those hours spent learning and teaching about packet-level trivia. If you don't want to read the article, here's the tl;dr:
- their automated DDoS detection tool detected an attack against a customer using packets sized in the 99,000 byte range.
- their ops staff pushed rules to their routers to drop those packets
- their routers crashed and burned
In order for this meltdown to happen, they had to have a compounded series of errors:
- the attack detection tool was coded to allow detection of packet sizes that can't actually occur: no bounds checking.
- the ops staff didn't retain the "trivia" that they learned in Networking 101, and thus couldn't see the problem with the output generated by the detection tool.
- the router OS didn't do input validation, and blew up when attempting to configure itself to do something crazy.