[IGSMAIL-7013] Outages and delays impacting IGS Ultra-rapid this weekend (wk1822_6_18 to 1823_0_06)

[Kevin Choi - NOAA Federal]

Author: Analysis Center Coordinator (ACC)

Dear IGS Colleagues:

The computer servers performing the IGS combinations experienced a network
outage that started sometime after 20:01 (GMT) on Saturday, December 13,
2014
for about a few hours.  The cause of the network outage remains unknown,
but
it seems to have been associated with an event affecting our offsite
facility
hosting the ACC servers.

The network outage impacted the Ultra-rapid combinations in the following
ways:

   * file retrieval processes were unable to get Analysis Center (AC) and
other
     critical products via the internet for the duration of the network
outage

   * the timing of the network outage spanned the duration of the
Ultra-rapid
     combination for GPS Wk 1822, day 6, hour 18, and this had several
adverse
     consequences:
     + the 1822_6_18 combination used only a single AC
     + it failed to complete properly
     + and without manual intervention, combination processes that fail to
       complete properly prevent future combinations from occurring--thus,
       all subsequent Ultra-rapid combinations failed through 1823_0_06
     + thus, there were no Ultra-rapid products delivered for 1822_6_18
       through 1823_0_06 (18 hours)

   * the timing of the network outage did not span the duration of the Rapid
     combination for Wk 1823, day 0:
     + However, PPN comparison with the Rapid solution was failed due to
failure
       of the RINEX file retrieval of BRUX, WILL, and TOW2 for DOY 347.

IGS Ultra-rapid products for 1822_6_18 through 1823_0_06 were restored and
resubmitted today between 14:52 and 15:05 (EST). The subsequent Ultra-rapid
combination for
1823_00_12 happened automatically without a problem. We expect the
Ultra-rapid
products to continue normally.

I am very sorry for the unfortunate events of this weekend. Please feel
free to
send any comments, suggestions, or concerns to Kevin and me (igs.acc at
noaa.gov).


Best regards,

Kevin  (NOAA/NGS)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.igs.org/pipermail/igsmail/attachments/20141214/26846d9a/attachment.html>