[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NEXRAD Level II Maintenance Notification - Today August 23th, 2016

Radar Operations Center and OPSnet engineering continue to troubleshoot the ROC latency issue.  Current latency is near 1.8 seconds for the last 24 hours, but the actual issue has not been resolved.

Networking diagnostic tools have been loaded on each server to localize issues with a standardized application.  Testing may effect the link between Silver Spring & the ROC but it is not anticipated to effect the TDS output.  (But If I didn't send out this e-mail Murphy's Law would bite me.)

Please be aware we are monitoring the line and will ensure a stable system once the testing is complete.  (I don't anticipate an issue, and continue to watch the Critical Weather status.)

I will send out a notice once we suspend testing today.


Thank you for your patience as we investigate this issue.


     ...Don
On 8/17/2016 15:19, Donald J Horvat - NOAA Affiliate wrote:

Radar Operations Center continues troubleshooting the ROC latency issue with legacy OPSNet engineering.  The symptoms have changed during the course of the day.  The ROC latency started dropping this morning and caught up to within a second of the TDS for MOST of the day.

Right now ROC Latency is at 440 seconds and growing so the issue persists.

If the latency problem persists further, the ROC will fall about 30 minutes behind.  The ROC & Legacy OPSNet Engineering will continue to use that configuration as an opportunity to examine and resolve the issue.

It is our intent to restore BOTH paths back to normal operation.

If weather looks like it will become a factor, we will refrain from troubleshooting and maintain a data monitoring function.

Thank you for your understanding for these short notices in this situation...


     ...Don


On 8/16/2016 11:02, Donald J Horvat - NOAA Affiliate wrote:

Radar Operations Center continues troubleshooting the ROC latency issue with legacy OPSNet engineering.  Moving the aggregate function to the ROC yesterday brought the latency of the alternate path down from over 30 minutes to under 2 seconds as the "acting primary" path.  Data transmission via Silver Spring as an "acting Alternate" path also remained under 2 seconds giving external users the impression of restoration of service.

Today we will roll back to Silver Spring as our Primary aggregate. (the system default).  If successful, it will restore the system back to normal operation.  Rolling the aggregate will cause temporary system latency issues

Right now both streams are up and there is no Critical Weather Day in effect.

1) At 1:00 PM (Central time) - the TDS will be rebooted.  It will stop data flow via that path for about 5 minutes.  When it comes back on line, it's entire queue will still be available allowing users to "catch up" on that stream.  (THE RDS PATH WILL BE UNTOUCHED AT THAT TIME.)

2) about an hour later (2:00 PM Central) - NL2 should be "running normal" once again. and we will roll the aggregate function back to Silver Spring.

a) At Silver Spring (as aggregate) - they will stop & restart LDM as they take over.  The aggregate rebuilds connections to all the NEXRAD sites and starts sending it's queue to the TDS.  Users will see data stop then "quickly" catch up.

b) At the ROC (as Alternate) - once the aggregate starts sending data the ROC severs will start forwarding.  (BUT THERE WILL BE A SHORT PERIOD AROUND 2 PM WHERE BOTH MOMENTARILY STOP)

If the latency problem from last week persists, the ROC will fall about 30 minutes behind.  The ROC & Legacy OPSNet Engineering will use that configuration as an opportunity to examine and resolve the issue.

It is our intent to restore BOTH paths back to normal operation.

If weather looks like it will become a factor, we will refrain from troubleshooting and maintain a data monitoring function.

Thank you for your understanding for the short notices in this situation...


     ...Don



-- 
**********************************************************************
*Donald J. Horvat               *  ____                              *
*                               * /    \     _____     _______  _____*
*Sr. Systems Engineer           *(      )   / ___  )  / ___  / / ___/*
* Support Contractor            * \____/   / /__/ /  / /  / / / /    *
*WSR-88D Radar Operations Center*  |\/|   / ___  <  / /  / / / /     *
*                               *  |/\|  / /   / / / /__/ / / /___   *
*1313 Halley Circle             *  |\/| /_/   /_/ /______/ /_____/   *
*Norman, OK 73069               *  |/\|                              *
*                               *   WSR-88D RADAR OPERATIONS CENTER  *
**********************************************************************
*Phone: (405) 573-3418          * e-mail: address@hidden   *
*FAX:   (405) 573-3340          *                                    *
**********************************************************************

-- 
**********************************************************************
*Donald J. Horvat               *  ____                              *
*                               * /    \     _____     _______  _____*
*Sr. Systems Engineer           *(      )   / ___  )  / ___  / / ___/*
* Support Contractor            * \____/   / /__/ /  / /  / / / /    *
*WSR-88D Radar Operations Center*  |\/|   / ___  <  / /  / / / /     *
*                               *  |/\|  / /   / / / /__/ / / /___   *
*1313 Halley Circle             *  |\/| /_/   /_/ /______/ /_____/   *
*Norman, OK 73069               *  |/\|                              *
*                               *   WSR-88D RADAR OPERATIONS CENTER  *
**********************************************************************
*Phone: (405) 573-3418          * e-mail: address@hidden   *
*FAX:   (405) 573-3340          *                                    *
**********************************************************************

-- 
**********************************************************************
*Donald J. Horvat               *  ____                              *
*                               * /    \     _____     _______  _____*
*Sr. Systems Engineer           *(      )   / ___  )  / ___  / / ___/*
* Support Contractor            * \____/   / /__/ /  / /  / / / /    *
*WSR-88D Radar Operations Center*  |\/|   / ___  <  / /  / / / /     *
*                               *  |/\|  / /   / / / /__/ / / /___   *
*1313 Halley Circle             *  |\/| /_/   /_/ /______/ /_____/   *
*Norman, OK 73069               *  |/\|                              *
*                               *   WSR-88D RADAR OPERATIONS CENTER  *
**********************************************************************
*Phone: (405) 573-3418          * e-mail: address@hidden   *
*FAX:   (405) 573-3340          *                                    *
**********************************************************************
begin:vcard
fn:Donald J Horvat
n:Horvat;Donald
org:Radar Operations Center;Systems Engineering Group
adr:;;1313 Halley Circle;Norman;OK;73069;USA
email;internet:address@hidden
title:Senior Systems Engineer
tel;work:405-573-3418
url:www.roc.noaa.gov
version:2.1
end:vcard