CCIE: Five Year Reflections

I passed the CCIE Routing & Switching lab five years ago today. Back then my number seemed enormous, but now five years later I'm already below the halfway point (as of this writing I believe the numbers are in the mid-to-high thirty-thousands). A lot has changed since then: it seems like "data center" and "cloud" have taken over almost completely as the hot topics in network engineering (with "software defined networking" hot on their heels), and it seems like Cisco has lost some of its shine due to its rapid diversification into markets outside pure networking and the rise of tough competitors in networking niche markets. We've gone through a huge economic contraction that we may or may not be exiting. In the certification world, Cisco has added several new tracks, and other companies have added their own coveted expert-level certifications.

I want to write about a few trends in certification and professional development that I've either observed personally, or that seem to be the subject of frequent discussion on the Internets.

Consolidation
One of the most interesting things I've noticed as a regular attendee at Cisco Live is that almost all CCIEs are in one of three categories:

1. Consultants working for Cisco resellers.
2. Employees of Cisco or one of its competitors.
3. Instructors working in the training and certification business.

Maybe this is sampling bias: perhaps it's just that the majority of CCIEs who attend Cisco Live also fall into one of those categories, and the ones who don't aren't attending in droves. Still, it seems comparatively rare to find CCIEs who are actually employed full time in network design or operations for a single company. I think one reason for this is that as IT employees in operational positions gain experience and seniority, their training and professional development opportunities decrease, possibly due to increasing costs, lack of availability of advanced training, and reluctance of employers to have their A-Teamers away from the office.

I think this is unfortunate, and it may be one cause behind the churn that companies tend to experience among high-level technical employees. The expense of maintaining training and professional development programs for these employees may also be a factor in the amount of outsourcing that we see in the network engineering field.

Track Proliferation and Specialization
I feel like the CCIE Routing & Switching track is kind of like a black belt in a legitimate martial art: it represents a thorough mastery of the basics, impresses novices who don't know any better, and hopefully impresses upon its recipients that they are really just at the beginning of the path. It still seems to me like it would be hard to pass the CCIE lab without understanding fundamental networking really well, but apparently it is possible; it's not uncommon to read about "paper CCIEs", and I've met at least one myself.

For me, the whole motivation behind studying for the lab was to confirm and exercise my understanding of the basics. I'm not a consultant or reseller, and I'm no longer a trainer; I actually work on the same network every day, and although my employer was very supportive of my studies, they certainly didn't require it. This motivation is one of the reasons that I haven't gone on to another track: they're too product specific for what I do. I work daily with Cisco security, voice, and wireless products, but I'm not intellectually driven by that kind of product specificity in anything resembling the way that I'm driven by the underlying theory and practice of general networking. The logical next step for me would probably be the CCDE, and indeed I was lucky enough to be invited as a beta participant in that program. I got spanked badly on the practical and haven't gone back, at least partly because the exam made it clear to me that even if I passed, I wouldn't have the real-world experience of working on multi-thousand router networks to go along with it.

Defining the Super-Generalist
None of this is meant to diminish the accomplishment inherent in the other CCIE tracks in any way: I remain extremely impressed by my friends who have passed the other tracks. However, for people working in mainstream IT networking my observation has been that the world could use more super-generalists. What skills should the super-generalist have? Here's my take on it:

[Edited to add: I'm not saying that this is a high-level skill set that substitutes for a CCIE. I'm saying this is a good base for working towards CCIE, and that if you find yourself missing big chunks of this while working on your second CCIE, you might consider re-prioritizing your learning.]

• Extremely solid IPv4 networking fundamentals. Certification programs are supposed to emphasize the basics, but I see CCNP-level people who haven't yet fully grokked ARP, STP, connection-oriented vs connectionless concepts, or why routing protocols work the way they do, even if they can explain how they work
• A growing familiarity with IPv6, and an appreciation of how protocols other than IPv4 have attempted to solve common problems.
• The ability to use Wireshark and tcpdump and interpret the resulting data.
• An understanding of the inner workings of common application-layer protocols, especially HTTP, DNS, and SMTP (yeah yeah, you can say email is dead but people still scream when it breaks). People can and do make entire careers out of each one, but understanding the basics is imperative. I am always amazed at how common it is to see server admins who don't understand HTTP response codes or how a recursive DNS query works.
• A familiarity with the internals of both Windows and Linux.
• Familiarity with common virtual machine platforms and how they affect networks.
• The basics of a scripting language and the common automation tools in the platforms with which you work most frequently.
• Fundamentals of network monitoring: SNMP, NetFlow, syslog, WMI, taps and mirror ports, considerations for asymmetric flows, etc.
• The basics of databases. This has long been one of my weakest areas, and something I've been working on fixing.
• The security considerations surrounding all of the above--and not just from a control standpoint. It's not enough to just know packet filtering and encryption; you also need to understand more than a little about the psychological aspects of security and privacy, and you should understand how your monitoring and diagnostic tools can be used both for good and ill.
• The big-picture of how the Internet works: what BGP is and the common ways that ISPs connect to customers and to each other, what CDNs are, the role of IANA and the RIRs, what the IETF and RFCs are, etc.
• A little respect for the ones who have gone before us, and some knowledge of Internet folklore. You damn well ought to know a little about the likes of Paul Baran, John Postel, Vint Cerf, Radia Perlman, and many others.
• The ability to write and speak coherently!

I'm sure I've left a few things out (add them in the comments), but even with just these you can iterate through them for years on end.