[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Support #WSQ-597130]: Missing data on LDM

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [Support #WSQ-597130]: Missing data on LDM
  • Date: Wed, 22 Feb 2012 09:35:58 -0700
Stacey,

[I've driven your email into our support tracking system.]

> My name is Stacey Simmons and I work for a subscription based weather site
> called weatherTAP.com.  We use LDM to ingest most of our data and have
> recently noticed that we are missing some files from a certain set of data.
> We ingest the files on one computer and use LDM to send those files to
> another computer.  We are getting the files on the ingesting computer, but
> some of them are not making it down to the storage computer.  If you have a
> little time, we would really appreciate it if you could look at our
> ldmd.conf and pqact.conf files that are on our storage computer.  I've
> attached both of these files.  The files are pretty short because we do not
> process a lot of data on this computer.  Could you look at them and see if
> it looks like we are doing something wrong that would cause some of the
> files to not be stored on this machine?  The files in question are the ones
> in the ldmd.conf file that come in as "weathertap/wrf/noram_rr/*".  We get
> 9 of these files per hour and we are missing anywhere from 2 to 8 files per
> day.  We have verified that they are coming in on our ingester machine and
> we have not noticed anything in the ldm log file that would suggest a
> problem.  Any help or direction you could provide about what could cause
> this behavior would be very much appreciated. We are using LDM 6.8.1 on
> both machines. Thanks for your time.

The "ldmd.conf" contained the following entries:

    exec "pqbinstats"

The pqbinstats(1) program is deprecated. You probably don't need it.

    request EXP "weathertap/wrf/noram_rr/*" ingester
    request EXP "weathertap/wrf/noram_rr/*" ingester_alt

The "/*" in the above is probably a mistake. The 3rd field is interpreted as an 
extended regular expression (ERE); the string "/*" means zero or more instances 
of the backslash character. You probably want the ERE 
"weathertap/wrf/noram_rr/.*", which means zero or more characters after the 
"noram_rr/" substring.

The "pqact.conf" file is syntactically correct. I don't know if its ERE-s match 
the intended product-identifiers, however. You could use the notifyme(1) 
utility on the storage system to verify these ERE-s.

You might also look in the LDM log files for any entries that indicate a 
problem.

> Sincerely,
> Mr. Stacey Simmons
> WeatherTAP.com

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: WSQ-597130
Department: Support LDM
Priority: Normal
Status: Closed