[NOAAPORT #BHX-452315]: LDM weirdness

Hi Paul,

This email is designed to try and catch you up on a variety
of things that have been happening over the past couple of
days...

On 20130301.1519 MST Steve asked:
> Did you receive data during the rtstats(1) outages?

On: 20130301.1522 MST you replied:

> It appears we did today. Other days we struggled. Today things were
> good as far as I can tell. Satellite, nexrad3 and surface data all
> flowed without a problem.

I just logged onto climate.cod.edu as 'mcidas' so I could see if you
had a continuous data record of decoded surface observations in McIDAS
format.  SFCLISTs of surface stations over the past 48 hours show that
you have been receiving and decoding data into McIDAS surface MD files
each hour (I checked reports for KDEN, KBOS and KLAX only).  This implies
that the real-time stats outages being seen in the IDS|DDPLUS volume
or products plots for climate.cod.edu were either artifacts associated
with real-time stats displays, or hiccups in reporting of real-time
stats on climate.  I am feeling more and more confident that the
real-time stats data gaps seen in the climate.cod.edu record were
artifacts caused by some as yet unknown cause on our machine that
is producing the plots.  I say this because all rtstats records stopped
updating last night (I happened to notice exactly when it happened
since I was working on a problem when it happened).  The situation
was not resolved until this morning when our system administrator,
Mike Schmidt, rebooted our machine that products the real-time
stats plots.

Also:

Because we got very concerned that the version of LDM-6.11.x that
was running on your NOAAPort ingest machine, noaaport.cod.edu,
may have somehow been causing some of the problems you were
reporting, I took the liberty of installing LDM-6.10.1 on
noaaport.cod.edu and switching to its use.  Because your
decoded record of surface observations in McIDAS appears to
be complete (see above), I am left with the feeling that
dropping back from LDM-6.11.3 to LDM-6.10.1 may not have been
called for on noaaport.cod.edu (or climate.cod.edu where I
also reverted the LDM release being used).

I want to leave things running as they are now at COD (meaning
running LDM-6.10.1 on both noaaport.cod.edu and climate.cod.edu).
We will be reviewing the sequence of events that transpired
over the past couple of days on your machines before making the
decision of whether or not to go back to using LDM-6.11.x on
your machines.  I hope you are OK with this!

A couple of questions for you:

- is there any "packet shaping" (deep packet inspection and/or
  artificially-imposed bandwidth limiting) being done either in
  your department or in COD at the moment?

- are you aware of any maintenance having recently been done on
  routers at COD that would be in the path of data flowing into
  our out of whatever machines you are using for LDM ingest or
  relay?

We are trying to trace down the cause of some very strange events
related to LDM use recently, and our experience working with network
folks at LSU (where one of the problems was experienced) makes us
suspicious that the problems seen could have been caused by "packet
shaping" software or router mis-configurations.

re:
> Could we have been backed up via an IDD?

The REQUEST lines I see on climate.cod.edu show that you are REQUESTing
NOAAPort-originated data (i.e., IDS|DDPLUS, HDS, NIMAGE, NGRID, and NOTHER)
only from internal COD machines:

request NIMAGE  .*      10.21.48.16     == noaaport.cod.edu
request NIMAGE  .*      10.21.48.17     == noaaportold.cod.edu
request NGRID   .*      10.21.48.16
request NGRID   .*      10.21.48.17
request NOTHER  .*      10.21.48.16
request NOTHER  .*      10.21.48.17
request NGRAPH  .*      10.21.48.16
request NGRAPH  .*      10.21.48.17
request IDS|DDPLUS|HDS .* 10.21.48.16
request IDS|DDPLUS|HDS .* 10.21.48.17

The only possible exception to this is your set of REQUESTS for
NEXRAD Level III data:

request NNEXRAD "SDUS(2|3|5|7|8)[1-6] (K*)"     10.21.48.16
request NNEXRAD "SDUS(2|3|5|7|8)[1-6] (K*)"     10.21.48.17
request NNEXRAD "SDUS(2|3|5|7|8)[1-6] (K*)"     weather.cod.edu

Given this, the answer to your question about being backed up via
an IDD would be no (unless weather.cod.edu is REQUESTing NEXRAD3
(ake NNEXRAD) from an external machine.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: BHX-452315
Department: Support NOAAPORT
Priority: Normal
Status: Closed