Bringing gateway from the dead...

At some point I was planning to explain how to bring “dead” Cisco gateway (or router, at that stage there is no difference in functionality) back to life and one of our AS5400 conveniently just “died” in the lab. How do I know that it’s “dead”? Well, a telnet session froze in the middle, since then I cannot ping this gateway anymore.

If this happens and you already tried to access to your gateway through every interface configured, then the only choice left is using the console. Connect the Console port on your gateway with the COM port on the PC, open Hyperterminal, use the default settings for speed, parity and so on, once finished, press enter and check the prompt that the gateway is replying with. If you see something like this:

rommon 1 >

then your gateway failed to load and is currently in the ROMMON. If you’re curious what cause your gateway to get into this mode, use “sysret” command.

The main goal at this stage is to find the right IOS image and boot it. Check the devices available:

rommon 2 > dev
Devices in device table:
id name
flash: flash
bootflash: boot flash

Check if flash: has any images already stored:
rommon 3 > dir flash:
File size Checksum File name
14423092 bytes (0xdc1434) 0x64cb5d98 c5400-is-mz.123-2.T5.bin
13769 bytes (0x35c9) 0x229ec3b1 tippi5400-confg
58095 bytes (0xe2ef) 0x8f9c2d6d crashinfo_20000101-002059

This gateway already has 5400-is-mz.123-2.T5.bin, so we just boot it:
rommon 4 > boot flash: c5400-is-mz.123-2.T5.bin

If the image is correct and finished the booting successfully, you’ll get a regular prompt:
tippi5400>

Normally, in case of an internal failure or a power outage, the router/gateway should reboot and start automatically loading the saved startup configuration. Some basic post-mortem diagnostics of why router went down can be done by looking at the “sysret” output in the ROMMON or checking the output from “sh ver” once loaded (if you’re running one of the latest versions and it keeps crashing in similar conditions, you probably want to let Cisco TAC know about it, including sending them the crash dumps). The next step is to fix the boot procedure so the router could come up automatically.

The most likely reason for this problem is that the router/gateway couldn’t find the IOS image to boot. In the router’s config you can specify image(s) to load. In my case they were pointing to the wrong files in the flash: and caused the boot procedure to fail. Remove the obsolete/incorrect references with “no boot system flash <name> ” and add the correct one with “boot system flash <name> ”:
boot system flash c5400-is-mz.123-2.T5.bin

Verify that your changes work. Save the configuration and reboot, this will help avoiding surprises when your gateway dies next time.

So, I enter “wr” then “reload” and... this gateway doesn’t boot again. :-( Moreover, it doesn’t look like it was actually trying to boot from anywhere at all. So, I manually specified where to boot from, loaded again and checked if anybody messed up with the register. Somebody definitely did:

tippi5400#sh ver
Cisco Internetwork Operating System Software
IOS (tm) 5400 Software (C5400-IS-M), Version 12.3(2)T5, RELEASE SOFTWARE (fc2)
<skipped>
Configuration register is 0x0

Well, it doesn’t load because it was specifically set not to. Fixed that by setting register to default with "config-register 0x2102". Again, saved and reloaded, and everything worked fine.

A couple of hints for getting rid of annoying messages:
“%Error opening tftp://255.255.255.255/tippi540.cfg (Timed out)” messages…
Enter “no service config” in the global config and these messages are gone.

“Translating "<something>"...domain server” when typing wrong command…
Enter “no ip domain-lookup” in the global config.

Finally, the gateway is up, so check that “no logging console” is set and forget about console terminal until next time you cannot telnet to your gateway.

There is a certain pain associated with downloading the IOS image from ROMMON via TFTP. In the lab gateway we laready had IOS image in the flash, so it was just the task of booting from it. Sometimes (especially after experiments with commands that erase the flash) your gateway may fail to boot because the flash is empty. Then you'll have to use ROMMON fro downloading IOS image as well. Use “set” command to specify IP_ADDRESS, IP_SUBNET_MASK, DEFAULT_GATEWAY, TFTP_SERVER, and TFTP_FILE, then enter “tftpdnld” which will download the image and place it into the flash (unless -r is specified). "reset" will cause gateway to reboot from ROMMON and verify if it can load successfully.