Thursday, December 31, 2009

predicting reload times on Catalyst 3560/3750

During a recent IOS upgrade on a Catalyst 3560, I was connected to the console and noticed that the reload was taking much longer than usual due to some operations by the "Front End Microcode IMG Mgr". The output looked like this:

POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed

front_end/ (directory)
extracting front_end/fe_type_1 (34760 bytes)
extracting front_end/front_end_ucode_info (86 bytes)
extracting front_end/fe_type_2 (73104 bytes)
extracting ucode_info (76 bytes)

Front-end Microcode IMG MGR: Installed 3 image(s) in cache:

Front-end Microcode IMG MGR: found microcode images for 3 devices.
Image for front-end 0: flash:/front_end_ucode_cache/ucode.1
Image for front-end 7: flash:/front_end_ucode_cache/ucode.1
Image for front-end 14: flash:/front_end_ucode_cache/ucode.1

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...26580 bytes.
Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrwsssspsssspsssspsss
[output truncated]

I opened a TAC case to find out what this is, since if you are relying on highly predictable reload times during a maintenance window, this could throw a wrench into your plans.

It turns out that the Catalyst switches have a special-purpose microcontroller that rarely needs to be upgraded. When it does need upgrading, however, the upgrade happens as a normal part of a new IOS image load. This upgrade makes the first reload to the new IOS take much longer than usual--I didn't time it, but I would guess 3-4 times longer than normal.

Microcontroller upgrades are not typically listed in the image release notes, so the only way to know for sure how long a particular upgrade is going to take is to test it in a lab, using the exact same before/after images that you will use in production.


ixfnx said...

nice, I ran across this today and was like WTF is taking this friggin' switch so long!!! thanks for the info. I also opened the TAC case just for informational purposes and can confirm, this is exactly the case! :)

Anonymous said...

The microcode programming on the 3560X takes MUCH longer than with other models. I did a 2960S recently and it took a few minutes; the 3560X PoE sitting next to me has been going for about 10 minutes now. That's progress!

Aman said...

thanks. this is the only site that mentioned this. thanks to op for creating this thread. saved me hassle to log a tac case :)


Anonymous said...

Thanks jswan! Saved me some heartache, I thought I borked the switch until I read this :)

Anonymous said...

I just had a 3750x take 30 minutes to boot. Thanks for the post.

Anonymous said...

I leave my switched turned off an hour before I get this error message...if I boot it up I get a black screen & the fans is going full what do I do...

1) when the device eventually boots up delete the old flash code
2) or...answers on a postcard please.

Anonymous said...

i had to update two of those today, the first one take a lot of time to boot, so i finally find this post... after that i clock the second one, just to post over here, jajjaja

the time was ..., from reload to boot and login: 25:59.4

for the first boot, after the upload took 21:50 and in a normal boot took 4:15

Anonymous said...

Puh, ok no panic then after reading this post. I'm doing a 3750X-PoE and I suspected something seriously wrong with my switch, so I reloaded it but here we go with rrrrrrwssssssssssssssssssspssss... again. This time I'll let it do whatever it does with the microcode and grab a cup of coffe. Thx for the post!

Anonymous said...

great post thanks for the info. thought the switch had major issues, was taking over 10 minutes to boot and is still going. Will let it do its thing.

AllanD said...

This was a huge help in satisfying my curiosity. Many thanks for posting it up!! :)

Anonymous said...

Tks a lot for posting this...i have the same case with a 3560X using for my ccie home lab

Anonymous said...

Another thank you for the post. 15 minutes into a 3750X and I am now just letting it do it's thing "ssssssssssssssssssspssssssssssssssssssspssssssssss"

Anonymous said...

thanks a lot - I just upgraded WS-C3560X-24P from 122-55.SE7 to 150-2.SE1 and it took 35 min

Anonymous said...

Hi All
I have just done the IOS upgrade on a C3750x stack.
From pressing reload until it was up again, 48 minuts
And thank you for the info, it was very helpfull

Anonymous said...

Hi All,

I had the same thing on both of my WS-C3560G-xxPS switches. 30+ minutes for the 1st reload after upgrading from 12.2(25)SED1 to 15.02.


Anonymous said...

I have foun this tread after I rebooted 3750X (upgrade from 12.2 to 15.0) It took around 26 minutes to boot ... amazing.