[LDM #SJZ-870882]: Problem ingesting archive files

Hi Jim,

Long time no hear...

re:
> Our usually reliable server had some problems (not LDM related) late 
> yesterday afternoon
> into this morning. In the process, LDM stopped ingesting even though it still 
> appeared
> to be running. I could not get it to stop gracefully with an "ldmadmin stop" 
> and had to
> do a hard kill to stop it. I was then able to get it back and running 
> smoothly.

Were you forced/did you decide to delete and remake the LDM queue?  This is not 
a bad
thing to do after OS problems.

re:
> In trying to recover data, I went to motherload.ucar.edu and downloaded the 
> DDPLUS (e.g.
> SURFACE_DDPLUS_20120720_0000.txt) files and then tried to insert them into 
> the ldm by
> using:
> 
> pqing -l - -q /usr/local/ldm/data/ldm.pq -f DDPLUS 
> SURFACE_DDPLUS_20120720_0000.txt
> 
> to insert them into the LDM. This has always worked well in the past. 
> However, only some
> of the data seemed to have been processed.

Question:

- did you check the exit status of 'pqing' after the insertion attempt

  If yes, was the indication one of success or failure?

The reason I ask is twofold:

- if the 'pqing' attempt failed, it would be logical that the product(s) would
  not end up in the LDM queue

- if the exit status of 'pqing' indicated success, it may mean that there is
  already a product in the LDM queue that has the same MD5 signature
  as the product you are trying to ingest.  'pqinsert' does not indicate
  a failure if the attempt to insert a product into the queue was not
  successful because of an existing product with the same MD5 signature
  already being in the queue.  I _assume_ that 'pqing' works the same
  way.

re:
> The missing upper air and MDR radar files
> were processed, however the surface metars and RCM radar files were not, yet 
> I can
> certainly see the surface metars in the motherlode file.

The fact that some products were processed successfully and others were not
suggests, but does not prove, that the LDM queue is OK.

re:
> Any ideas?

The simplest thing is if there is already a product with the same MD5 signature
in the queue, so 'pqing' does not insert the product.  This is really not
a failure per se; it is a situation that occurs a lot when LDMs are REQUESTing
the same data from more than one upstream host.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: SJZ-870882
Department: Support LDM
Priority: Normal
Status: Closed