To: heanet-faults@listserv.heanet.ie
From: Brian Nisbet <brian.nisbet@heanet.ie>
Subject: HEA-NOC/20100216-1 [CLOSE] NQAI connectivity lost              
X-HEAnet-TicketID: 20100216-1
X-HEAnet-Ticket-Distribution: public
In-Reply-To: <20100216-1-22@mousetrap.hea.net>
References: <20100216-1-22@mousetrap.hea.net>
<20100216-1-21@mousetrap.hea.net> <20100216-1-20@mousetrap.hea.net>
<20100216-1-19@mousetrap.hea.net> <20100216-1-18@mousetrap.hea.net>
<20100216-1-17@mousetrap.hea.net> <20100216-1-16@mousetrap.hea.net>
<20100216-1-15@mousetrap.hea.net> <20100216-1-14@mousetrap.hea.net>
<20100216-1-13@mousetrap.hea.net> <20100216-1-12@mousetrap.hea.net>
<20100216-1-11@mousetrap.hea.net> <20100216-1-10@mousetrap.hea.net>
<20100216-1-9@mousetrap.hea.net> <20100216-1-8@mousetrap.hea.net>
<20100216-1-7@mousetrap.hea.net> <20100216-1-6@mousetrap.hea.net>
<20100216-1-5@mousetrap.hea.net> <20100216-1-4@mousetrap.hea.net>
<20100216-1-3@mousetrap.hea.net> <20100216-1-2@mousetrap.hea.net>
<20100216-1-1@mousetrap.hea.net>
Message-ID: <20100216-1-23@mousetrap.hea.net>

Ticket Number: HEA-NOC/20100216-1      Ticket Status: CLOSE
Ticket Type:   unscheduled             Resolver:      Eircom
Ticket Opened: 20100216 08:07 UTC      Problem Start: 20100215 23:37 UTC
Ticket Closed: 20100303 17:26 UTC      Problem End:   20100224 17:50 UTC

Site/line:     NQAI

Problem Description:
Connectivity has been lost to NQAI

Problem Effects:
No access to NQAI

Problem History:
20100216 08:05 JB - Connectivity lost at 23:37 yesterday evening (15th
Feb). Opening ticket to track outage and investigating.

20100216 09:22 JB - NQAI confirm no issues on site. Reported possible line
fault to Eircom.

20100216 11:52 GM - Eircom have found no faults on the link and have
closed their ticket.

20100216 12:06 GM - Bouncing the interface on the CRS had no affect on the
line. CRS cannot see MAC address of neighbour device either.

20100216 12:36 GM - NQAI power cycled their 2801 CPE and service was
restored. Need to investigate the cause of the outage.

20100216 12:40 GM - Time on NQAI is out by approximately 15-20 mins, this
was noticed when logging to central syslog server and correlating logs
during outage. This has been rectified by updating the NTP config on the
box and the addition of 3 further NTP servers.

20100219 16:10 JB - Unable to determine reason for the CPE becoming
unreachable as there were no logs sent during the outage and non logs were
retained for after a reboot. If the CPE fails again a site visit will be
required to get the logs. Circuit has been stable, closing ticket.

20100223 06:51 JB - Connection has gone down again at 21:04 on 20100222.

20100223 09:31 GL This is related to a switch in our core in Citywest
malfunctioning. It has also come to light that when NQAI's CPE router
loses
connectivity, for it to re-establish connectivity, it needs to be manually
rebooted. This iwll be investigated further when switch in Citywest has
been fully restored

20100223 13:30 COC Eircom Ref for this is 1056. Went onsite to NQAI and
verified that cpe1-nqai was functioning correctly. Waiting for Eircom to
supply an RFO and to re-establish this link

20100223 14:44 COC Eircom have checked this circuit and believe this to be
a local issue onsite in NQAI. Informed them that there was no link local
light on their equipment. Also informed them that the issue would seem to
be with port 13 on the Extreme switch not showing any activity. Eircom are
going to try and power cycle their equipment.

20100223 15:19 COC Eircom checked the wrong circuit, new fault opened
1074. Eircom asked that their NTU be rebooted by on site support in NQAI.
This was done but it did not restore connectivity.

20100223 16:43 COC Eircom Engineer has arrived onsite in NQAI

20100224 09:15 COC Eircom have determined that the fault lies with a
faulty power card in Custom Dock. They are replacing it this morning.
Contacting NQAI.

20100224 11:56 BN Current Eircom ref 1074.

20100224 13:44 BN Eircom have, as yet, failed to resolve this issue as the
card replacement did not work.  An Eircom engingeer is expected on-site in
NQAI early this afternoon.

20100224 15:29 COC Eircom have informed us that there is a problem now
with their line which they are attempting to fix. ETA still unknown.

20100224 17:19 BN NTU has been replaced on-site in NQAI.  However further
configuration is required to bring the link back into service.  This work
cannot be completed by Eircom until 08:00 on 20100225.

20100224 17:51 COC Connectivity has been restored. Monitoring

20100224 18:04 GL Informed client via voicmail. Will contact in the
morning.

20100225 17:19 GL Circuit up and stable. RFO requested from Eircom account
manager by BN. Awaiting reply

20100303 17:25 BN Circuit has remained stable for nearly one week, ticket
is being closed.  Outage to be discussed, at length, with Eircom.

Time to Fix:
21:04 on 20100222 to 17:50 on 20100224 = 44 hrs 46 mins

-- 
This ticket can be monitored at http://www.hea.net/tickets/20100216-1

HEAnet Limited. Registered in Ireland, No. 275301.