[LDM #QXS-738090]: ldm questions

Hi Mike,

re:
> I'm noticing a couple of things with image data sets and I haven't been
> able to figure out what is going wrong.  I'm sure that this is because I don't
> really understand how ldm works, but since I'm basically the person supporting
> ldm at Purdue, I'd better start learning...

:-)

re:
> We've only been getting NEXRCOMP images sporadically for the past week or
> so (?) for example, today we have the 1km/n0r only for three times
> (n0r_20121017_0026, n0r_20121017_0939, n0r_20121017_1354).  This was working
> fine just a week ago I believe.

The problem is not your LDM, but, rather, your Internet connection.  This can
be seen in the FNEXRAD latency plot of real-time statistics for the
Purdue LDM/IDD machine that is supporting statistics:

Unidata HomePage
http://www.unidata.uxar.edu

  Projects -> Internet Data Distribution
  http://www.unidata.ucar.edu/projects/index.html#idd

    IDD Current Operational Status
    http://www.unidata.ucar.edu/software/idd/rtstats/

      Statistics by Host
      http://www.unidata.ucar.edu/software/idd/rtstats/

weather2.eas.purdue.edu [6.9.7]
http://www.unidata.ucar.edu/cgi-bin/rtstats/siteindex?weather2.eas.purdue.edu

Click on the 'latency' link for the FNEXRAD feed and you will see that
the latencies are frequently and routinely exceeding what is likely
the age of the oldest product in the LDM queue from your upstream
host.  When this happens, you will miss products.

re:
> For a while now, we've been getting duplicate GOES images that have
> different names ending up in the same directory.

The LDM will detect and eliminate truly duplicate products from
its queue (actually, duplicates never get added to the queue),
so ending up with two of the same product in the local LDM queue
at the same time should never be possible.  What is more likely
is that you have more than one pattern-action file action that
is processing the same image in the same or different ways.

re:
> This messes up NAWIPS when
> we're trying to plot a loop. For example, we're getting 4km/IR:
> GOES-15_IR_20121017_1430 and IR_20121017_1430

The fact that the names of the files are different strongly suggests
the conjecture in the previous paragraph: you have at least two
pattern-action file actions that are processing the same product(s)
but naming them differently.

re:
> We haven't getting anything in the composite satellite images (?) like
> EAST-CONUS for quite a while now, I thought this might have something to do
> with GOES-13 being offline, but I'm wondering if we have some other issue.

The EAST-CONUS images are part of the NIMAGE datastream (unless you have
a pattern-action file action that stores UNIWISC (aka MCIDAS) images
in a directory/subdirectory named EAST-CONUS.  I do _not_ see the NIMAGE
feedtype listed in the set of feeds that weather2.eas.purdue.edu is receiving:

http://www.unidata.ucar.edu/cgi-bin/rtstats/siteindex?weather2.eas.purdue.edu

Perhaps someone mucked with your LDM configuration file, ~ldm/etc/ldmd.conf?

re:
> Thanks for your help!

Please send us your LDM configuration file and any/all of your pattern-action
files (as attachments to a reply to this email).  We will review the files
to see if there is anything obviously wrong.

The other thing you need to do is figure out why the latencies are so high
for some (not all) of the feeds you are REQUESTing.  A quick look at
latency plots for weather2.eas.purdue.edu shows the following:

Acceptable latencies:

CONDUIT
EXP
LIGHTNING
NEXRAD2

High/unacceptable latencies:

FNEXRAD
FSL2
IDS|DDPLUS
NEXRAD3
UNIWISC

The fact that the CONDUIT and NEXRAD2 feeds are OK while FSL2 and IDS|DDPLUS are
not is very strange since CONDUIT and NEXRAD2 have the highest volumes while 
FNEXRAD
and IDS|DDPLUS are two of the lower ones (LIGHTNING and UNIWISC are less).

Here is a snapshot of the data volumes/numbers of products from your machine:

Cumulative Volume Summary
http://www.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume?weather2.eas.purdue.edu

Data Volume Summary for weather2.eas.purdue.edu

Maximum hourly volume  13523.160 M bytes/hour
Average hourly volume  10221.263 M bytes/hour

Average products per hour     243129 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
CONDUIT                3520.328    [ 34.441%]     5673.115    72569.927
NEXRAD2                2926.339    [ 28.630%]     3812.826    44816.220
FSL2                   1201.772    [ 11.758%]     1524.888     1477.561
NEXRAD3                1174.291    [ 11.489%]     1544.065    64198.439
EXP                     985.933    [  9.646%]     1141.516     1940.122
HDS                     341.039    [  3.337%]      515.925    18822.659
IDS|DDPLUS               46.985    [  0.460%]       59.787    39231.463
UNIWISC                  22.706    [  0.222%]       31.088       21.268
FNEXRAD                   1.862    [  0.018%]       11.218        2.610
LIGHTNING                 0.011    [  0.000%]        0.036       48.390

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: QXS-738090
Department: Support IDD
Priority: Normal
Status: Closed