To: heanet-faults@listserv.heanet.ie
From: Garwin Liu <garwin.liu@heanet.ie>
Subject: HEA-NOC/20100305-4 [CLOSE] IOS Upgrades to Citywest and
Kilcarbery PoP
X-HEAnet-TicketID: 20100305-4
X-HEAnet-Ticket-Distribution: public
In-Reply-To: <20100305-4-17@mousetrap.hea.net>
References: <20100305-4-17@mousetrap.hea.net>
<20100305-4-16@mousetrap.hea.net> <20100305-4-15@mousetrap.hea.net>
<20100305-4-14@mousetrap.hea.net> <20100305-4-13@mousetrap.hea.net>
<20100305-4-12@mousetrap.hea.net> <20100305-4-11@mousetrap.hea.net>
<20100305-4-10@mousetrap.hea.net> <20100305-4-9@mousetrap.hea.net>
<20100305-4-8@mousetrap.hea.net> <20100305-4-7@mousetrap.hea.net>
<20100305-4-6@mousetrap.hea.net> <20100305-4-5@mousetrap.hea.net>
<20100305-4-4@mousetrap.hea.net> <20100305-4-3@mousetrap.hea.net>
<20100305-4-2@mousetrap.hea.net> <20100305-4-1@mousetrap.hea.net>
Message-ID: <20100305-4-18@mousetrap.hea.net>

Ticket Number: HEA-NOC/20100305-4      Ticket Status: CLOSE
Ticket Type:   scheduled               Resolver:      HEA-NOC
Ticket Opened: 20100305 15:37 UTC      Problem Start: 20100313 09:00 UTC
Ticket Closed: 20100510 10:49 UTC+1    Problem End:   20100313 17:00 UTC

Site/line:     Citywest PoP, Kilcarbery Park PoP

Problem Description:
Essential IOS upgrades to devices in Citywest and Kilcarbery PoPs is to
take place Saturday 13th March between 09:00 and 17:00.

A window of 8 hours is scheduled but clients will be offline for no longer
than 10 minutes. 

Problem Effects:
This will affect ALL HEAnet clients. 

All clients will experience and loss of resiliency except for IADT and
Carlow College who will experience an outage. 

Full list to follow on Monday.

Problem History:
20100305 15:31 GL Opening ticket

20100308 16:23 GL This work has been postponed till a later date.

20100416 16:35 GL Kilcarbery PoP will be upgraded on Thursday 22nd April
at 07:00. 

This will affect ALL HEAnet clients. 

All clients will experience a loss of resiliency for all services

Loss of connectivity:

ESRI
IEDR
IT Carlow (Wexford Campus)
NUIM (Kilkenny Campus)
IEDRSN
IUA
DIT - Airport Link
HEA


Servcices:
Commodity IP
Government Network

20100422 08:58 GL One of line cards did not come back up after IOS
upgrade. Moved all client circuits to another line card and rebuilt
circuit
configs. Clients affected through loss of connectivity are back up.
Commodity IP and Government Networks also back online. Optical links to
Servecentric and Dundalk remain down as is connection to core router in
Kilcarbery meaning a loss of resiliency for all clients. Replacement card
sought. Lancomms Ticket Ref is 61852

20100422 13:25 BN Lancomms are due to replace this card between 14:00 -
15:00.  This work will not be service affecting to client connections.

20100422 14:35 COC LineCard has been replaced. All connections restored.
Kilcarbery Park IOS update complete.

20100422 15:16 GL Circuits need to be rebuilt back to replacement card.

When: Monday 26th April 08:00. 
Window: 30 minutes with 5 minute outage.

Loss of connectivity:
IEDR
IT Carlow (Wexford Campus)
NUIM (Kilkenny Campus)
IEDRSN
IUA
DIT - Airport Link
HEA

Services:
Commodity IP
Government Network

Loss of Resiliency to all other clients.

20100426 08:03 GL Work completed. Circuits rebuilt over replacement card.
Circuits up, passing traffic and stable. Monitoring

20100428 12:01 RH IOS Updates scheduled for 5th May at 06:00 (3 hour
window) until 09:00

Complete loss of connectivity to All Clients and Services:

cwt-pe1
cwt-pe2         
pw-pe2
cwt-sw1
amnch-sw1
an-cheim-sw1
tippthurles-sw1
iadt-sw1
ittallaght-sw1
belfield-sw2
nuim-sw2
tony-ryan-sw1
itc-sw2
iadt-sw2
carysfort-sw1
lyons-sw1
dias-sw2
rcsi-sw2

cpe1-epa
cpe1-nui
cpe1-bcfe
cpe1-hetac
cpe1-whitehall
cpe1-killester
cpe1-liberties
cpe1-hrb
cpe1-iadt
cpe1-ittallaght
cpe1-ac
cpe1-carlowc
cpe1-itc
cpe1-tippinst
cpe1-rcsi
cpe2-ichec
cpe2-ncad
mgt1-adva

Services:
An Cheim

20100504 11:32 GL REMINDER: IOS UPGRADES TO TAKE PLACE WEDNESDAY 5TH MAY
06:00 - 09:00

20100505 07:26 COC cwt-sw1 switch 2 failed after reloading. Contacted
LANCOMMS to RMA equipment. Arranging access for. RMA ordered for
WS-C3750G-12S-S. 
Loss of connectivity too

an-cheim-sw1
belfield-sw2
ittallaght-sw1:
amnch-sw1: 
NUIM - Bonded link 1 
NUIM - Bonded link 2 
IADT
cwt-ar2.net.edu.ie Schools cwt-ar2 to pw-br1)
cpe2-ncad 
Eircom Research Lab Citywest TSSG/WIT
tony-ryan-sw1

20100505 09:05 GL The window for this work has been extended due to
failure of an upgrade to a core switch. As a result of this there will be
an outage extending beyond this window. The clients affected are stated in
this ticket https://www.hea.net/mousetrap/update/20100505-1

20100505 09:47 GL Replacement switch installed. All services restored.

20100510 09:27 GL Circuits stable. As a core switch failed, some remote
switches could not be reached and therfore not upgraded. A separate ticket
will be opened for the remote switches upgrades. Closing ticket

-- 
This ticket can be monitored at http://www.hea.net/tickets/20100305-4

HEAnet Limited. Registered in Ireland, No. 275301.