To: heanet-faults@listserv.heanet.ie
From: Garwin Liu <garwin.liu@heanet.ie>
Subject: HEA-NOC/20100225-11 [CLOSE] IOS Upgrades to Cork,TCD and Galway
PoP
X-HEAnet-TicketID: 20100225-11
X-HEAnet-Ticket-Distribution: public
In-Reply-To: <20100225-11-31@mousetrap.hea.net>
References: <20100225-11-31@mousetrap.hea.net>
<20100225-11-30@mousetrap.hea.net> <20100225-11-29@mousetrap.hea.net>
<20100225-11-28@mousetrap.hea.net> <20100225-11-27@mousetrap.hea.net>
<20100225-11-26@mousetrap.hea.net> <20100225-11-25@mousetrap.hea.net>
<20100225-11-24@mousetrap.hea.net> <20100225-11-23@mousetrap.hea.net>
<20100225-11-22@mousetrap.hea.net> <20100225-11-21@mousetrap.hea.net>
<20100225-11-20@mousetrap.hea.net> <20100225-11-19@mousetrap.hea.net>
<20100225-11-18@mousetrap.hea.net> <20100225-11-17@mousetrap.hea.net>
<20100225-11-16@mousetrap.hea.net> <20100225-11-15@mousetrap.hea.net>
<20100225-11-14@mousetrap.hea.net> <20100225-11-13@mousetrap.hea.net>
<20100225-11-12@mousetrap.hea.net> <20100225-11-11@mousetrap.hea.net>
<20100225-11-10@mousetrap.hea.net> <20100225-11-9@mousetrap.hea.net>
<20100225-11-8@mousetrap.hea.net> <20100225-11-7@mousetrap.hea.net>
<20100225-11-6@mousetrap.hea.net> <20100225-11-5@mousetrap.hea.net>
<20100225-11-4@mousetrap.hea.net> <20100225-11-3@mousetrap.hea.net>
<20100225-11-2@mousetrap.hea.net> <20100225-11-1@mousetrap.hea.net>
Message-ID: <20100225-11-32@mousetrap.hea.net>
Ticket Number: HEA-NOC/20100225-11 Ticket Status: CLOSE
Ticket Type: scheduled Resolver: HEA-NOC
Ticket Opened: 20100226 17:30 UTC Problem Start: 20100306 09:00 UTC
Ticket Closed: 20100413 12:09 UTC+1 Problem End: 20100306 17:00 UTC
Site/line: Cork PoP, Galway PoP, TCD-PoP PoP
Problem Description:
Essential IOS upgrades on layer 2 and layer 3 devices are to take place
across 3 PoPs, Cork, Galway and TCD.
This is scheduled for Saturday 6th March between the hours of 09:00 and
17:00. An 8 hour window is scheduled but clients will not be offline for
the full 8 hours. Both layer 2 and layer 3 devices require a reload.
Proposed Timeline:
TCD PoP: 09:00 - 12:00
Galway PoP: 12:00 - 14:00
Cork PoP: 14:00 - 15:00
These are estimate times which may or may not over run. If a PoP is
finished earlier than proposed, work will continue onto the next PoP.
Problem Effects:
NCI
DIT (All Campuses and links in between)
Mater Dei (DCU)
TCD (General IP, TCD eGov Network,Co-lo link from TCD to
Servecentric,D'Olier St,Foster Street)
Georges Dock (HEAnet Office)
St. Patricks, Drumcondra
RIA
DIAS Merrion - LOR
Oireachtas
RSCI (LoR)
IPA
NDRC
DIAS Burlington Road
IT Tallaght
ICHEC HQ
FEDERICA
Digital Depot
St Vincents University Hospital
RTE
UCD (Belfied (LoR) and Eccles St. Campuses)
UCC
CIT
Blackrock College
ITT (LoR)
ESBT & E-net fibre link to WIT
nmci
Crawford College
Col����¯�¿�½���¯���¿���½������¯������¿������½����¯�¿�½���¯���¿���½����¯�¿�½������¡iste
Stifioan Naofa
NCRI
Cork School of Music
Cork School of Architecture
UCC Hydraulics & Marine Research Centre
AIT
GTI
Akamai
LYIT (Main & Killybegs)
IT Sligo
GMIT (Main,Letterfrack, Castlebar)
Cluain Mhuire
Castlebar
PAC
STACS
Tipperary Institute (Clonmel and Thurles)
Marin Institute
NUIG (Main,Northern Computer Room,Open Learning Centre, Clinical
Science,DERI,Gweedore)
CAO
Trunk to Airspeed for Gaeltacht
-MHARS
-MRI
-ASNG
-Carraroe
-Sean Scoil Shailearna, Inverin
-Martin Ryan
NIRAN
Problem History:
20100225 15:39 GL Opening ticket.
20100305 15:37 GL REMINDER: This work is to take place tomorrow morning
Saturday 6ht March at 09:00
20100306 10:41 GL UPDATE: Upgrades to TCD PoP completed.
20100306 11:00 GL Commencing Galway PoP IOS UPgrades
20100306 12:59 GL Upgrades to Galway PoP completed
Commencing Cork PoP IOS Upgrades
20100306 14:01 GL IOS Upgrades to Cork PoP, Galway PoP and TCD PoP
completed succesfully. Monitoring for 48 hours.
20100306 14:35 GL Circuits on a head end switch in Cork failed to come
back up after upgrade. Investigating
20100306 14:58 GL Affected circuits:
Loss of resiliency: IT Tralee, CIT, UCC
Loss of connectivity: Blackrock College, Cork School of Music, HMRC, Cork
School of Architecture.
Rang fibre provider to see if any issues within thier network. None
reported. Continue to investigate
20100306 17:28 GL Above sites still down. Continuing to investigate.
Window has now been extended till 21:00 as it is the weekend.
20100306 20:01 GL No further developments for tonight. Circuits remain
unreachable. Updates to follow. Closing window
20100307 10:16 GL This will be looked into tomorrow, Monday. Monitoring
20100307 17:34 GL Emailed/called LANcomms. They have an engineer working
on it. Awaiting reply.
20100307 18:40 GL Reverting back to previous IOS image. This had been
attempted yesterday but returned an error. An added argument in a command
was needed to overcome the error. Updates to follow.
20100307 18:53 GL Circuits back up and reachable. Traffic is flowing just
like before. Investigating root cause. Updates to follow in the morning.
20100308 16:22 GL Sought update from LANcomms. Awaiting reply.
20100325 14:10 GL A retry of the failed IOS upgrade on core switch in Cork
is to take place on March 31st at 07:00. Window of 1 hour. Clients
notified.
20100331 08:00 GL Window to be extended for 10 minutes until 08:10.
Apologies for any inconvenience caused.
20100331 08:07 GL Service has resumed. No further outages will occur.
Awaiting test results from Cisco TAC
20100413 12:03 GL Cisco TAC replicated scenario and have filed a new bug.
Closing this ticket to avoid confusion over whether or not updates have
been carried out successfully on other devices. An internal ticket has
been
opened and a new ticket will be issued and clients notified when upgrade
to
switch is to be attempted again.
Time to Fix:
--
This ticket can be monitored at http://www.hea.net/tickets/20100225-11
HEAnet Limited. Registered in Ireland, No. 275301.
|
