20000308: data losses? at STC

>From: alan anderson <address@hidden>
>Organization: St. Cloud State
>Keywords: 200003081631.JAA28926 IDD

Alan,

>A recent concern here has been missing data.  I have watched my email
>on the ldm-user list and the other postings from various sites, and
>have not seen the same concern, so am assuming it is a problem local
>to us.  I know there have been some outages, but those are not the ones
>we have seen.
>
>Our problem has been sporadic, and has been noticed as gaps
>in data found in MD files.  e.g. some upper air sites just not there
>on McIDAS, where they are plotted on the difax maps and can be found
>on various web sites.  Problem is not continuous, and on some days 
>does not occur at all.

The fact that on some days you do not see data loss tells me that the
problem is not likely to be with your LDM or McIDAS-X setup/installation.

>Could you comment on
>
>1. Possible causes for such missing data; 

Network congestion, or your clock being off so that your LDM is requesting
data that is in the future.

>2. Any specific checks we can do to identify the problem;  I seem to recall
>   there are some programs in the ldm that help with this, but I need some
>   advice on their use, and the interpretation of results.

One can look at latencies for your site.  For instance, if you were concerned
with products in DDPLUS, you could run a notifyme and interpret the
results from it:

notifyme -vxl - -f DDPLUS -o 3600 -h waldo.stcloudstate.edu
Mar 08 22:54:30 notifyme[7383]: Starting Up: waldo.stcloudstate.edu: 
20000308215430.530 TS_ENDT {{DDPLUS,  ".*"}}
        NOTIFYME(waldo.stcloudstate.edu) returns OK
Mar 08 22:54:30 notifyme[7383]: NOTIFYME(waldo.stcloudstate.edu): OK
Mar 08 22:54:31 notifyme[7383]: 281dae90d408c255c9b99abfc8465280      160 
20000308215433.592 IDS|DDPLUS 756  SAUS44 KFWD 082154 /pMTRSPS
Mar 08 22:54:31 notifyme[7383]: 1fbe85b6ad0df5e363d1ec2ff86b3ef2      127 
20000308215438.496 IDS|DDPLUS 758  FXUS43 KGLD 082154 /pMTTCBK
Mar 08 22:54:31 notifyme[7383]: 9603a9a8aaad5c92213d0bd230ca9bac      123 
20000308215438.508 IDS|DDPLUS 759  SAUS41 KCLE 082154 /pMTRTOL
Mar 08 22:54:31 notifyme[7383]: d4835a0931d244ca201b5a36370d4a82     1006 
20000308215438.788 IDS|DDPLUS 760  WWUS45 KFSD 082154 CCA /pNPWFSD
Mar 08 22:54:36 notifyme[7383]: 22b32efa47b1571bbe4208f8822717cb      109 
20000308215439.246 IDS|DDPLUS 769  SAUS44 KLZK 082154 /pMTRELD
Mar 08 22:54:36 notifyme[7383]: 1b7b4f0316261baf185f64eae9093058      302 
20000308215449.635 IDS|DDPLUS 794  SXUS01 KNYC 082154 /pOSONYC
Mar 08 22:54:36 notifyme[7383]: 3be56c1c656ea6eb5ab0f19ecbcd1953     3382 
20000308215449.962 IDS|DDPLUS 797  FPUS54 KSJT 082154 /pZFPSJT

There are two times in the listing:

o the time a product was received
o the time the product was originally injected into the IDD

From the last line in the listing above, we can see that the product
'FPUS54 KSJT 082154 /pZFPSJT' was injected at 20000308215449.962 and
received at Mar 08 22:54:36.  So, there is almost an entire hour lag
in your receipt of the data.  This is the extreme of the ragged edge.

At this point you may want to do pings and traceroutes to your upstream
feed site to see how long they are taking.

>3. Comments I can pass to my campus system people about the type of 
>   communications our ingesting machine has with our upstream site
>   to receive data.
>   I have talked some with them, mainly to remind them that we want good
>   connectivity  and bandwidth both within and off the campus.  They asked
>   me whether our data was received via ftp, and I said no, but was not
>   sure of the type of link we have with our upstream site.

The LDM uses TCP transport.

>4. Our ingesting machine is waldo, and I think you have already been
>introduced, so you can snoop if you like.  I have looked at our log files
>and don't see anything that seems bad to me, but I may not recognize what
>is happening.

The notifyme tells all.  For some reason the data are just not making it
to your machine in a timely fashion.  Have you tried switching to your
backup site?

>Again, I thank you for your help.

You are welcome.  If you continue to have problems, I will turn this over
to Robb, Jeff, or Anne.

>I do have a joke.  Seems that this garden snail was out one evening and was
>mugged by a turtle.  The police were called, and asked the snail how it
>happened. The snail replied that he didn't recall anything, as it all
>happened too fast.

Groan!

Tom