To: heanet-faults@listserv.heanet.ie
From: Garwin Liu <garwin.liu@heanet.ie>
Subject: HEA-NOC/20100204-8 [CLOSE] IOS Upgrades to Limerick PoP Devices
X-HEAnet-TicketID: 20100204-8
X-HEAnet-Ticket-Distribution: public
In-Reply-To: <20100204-8-7@mousetrap.hea.net>
References: <20100204-8-7@mousetrap.hea.net>
<20100204-8-6@mousetrap.hea.net> <20100204-8-5@mousetrap.hea.net>
<20100204-8-4@mousetrap.hea.net> <20100204-8-3@mousetrap.hea.net>
<20100204-8-2@mousetrap.hea.net> <20100204-8-1@mousetrap.hea.net>
Message-ID: <20100204-8-8@mousetrap.hea.net>

Ticket Number: HEA-NOC/20100204-8      Ticket Status: CLOSE
Ticket Type:   scheduled               Resolver:      HEA-NOC
Ticket Opened: 20100204 10:17 UTC      Problem Start: 20100218 07:00 UTC
Ticket Closed: 20100304 12:38 UTC      Problem End:   20100218 09:00 UTC

Site/line:     IT Tralee, LIT, SAD, UL, Limerick PoP

Problem Description:
Essential IOS upgrades to all devices (layer 2 and 3) connected to
HEAnet's Limerick PoP are to take place on February 18th at 07:00.
Window is 2 hours. 


Problem Effects:
The following clients will experience a brief (router/switch reload time)
loss of connectivity

UL CPE 
LIT 
Mary Immaculate 
Limerick School of Art and Design 
Limerick Outreach Centre  

Loss of Resiliency 
IT Tralee 

Problem History:
20100204 10:10 GL Opening ticket

20100217 17:53 GL REMINDER: IOS Upgrades tomorrow morning (18/2) to go
ahead at 07:00

20100218 08:14 GL Supervisor card in Limerick has failed during IOS
update. Circuits down. Investigating

20100218 08:16 GL Reverted back to old IOS image. Investigating

20100218 08:29 GL Circuits back up and stable. Ran MD5 check on image
prior to loading IOS onto device..checked out ok. Aborting IOS upgrade of
Limerick PE device until further investigation reveals reason for
redundant
supervisor card to fail. Also Cork School of Art and Design's switch's
flash memory was too small to upgrade IOS on this occasion. Both upgrades
will be rescheduled.

20100222 08:53 GL A Cisco bug was responsible for this. Too many boot
variable statements, which is used to select to the IOS loaded onto flash,
confuses the device and causes the boot loader to load from its basic
sup-bootflash image. On next reload, statements will be limited to 2.
Rescheduled for March 3rd at 07:30

20100303 07:50 GL IOS upgrade completed successfully. Monitor for
remainder of day.

20100304 12:38 GL Limerick circuits stable. Closing ticket

-- 
This ticket can be monitored at http://www.hea.net/tickets/20100204-8

HEAnet Limited. Registered in Ireland, No. 275301.