[LDM #BKT-485550]: Problems with profiler data

Hi Laurie,

re:
> Thanks for checking this. It looks like our MD5's match yours.
> I guess this points to a problem from ESRL/GSD?

I would think so, yes.

> Here is the output from our MD5 checks:
> MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062281718) =
> e485cbfa5d47ac98502ee03faf973498
> MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062291618) =
> 0a401fdd3f3adaa04b445ff2ba199fa1
> MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062292000) =
> e275f790f6fc76a43c0c4482bce62c11
> MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062292012) =
> b836c229686986bb27af64b01a3f5515
> MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062292048) =
> 6278f6af82dd21e1ea3068bf7d95d74e
> MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062292054) =
> 9a3123d6d05ee6b56a161c12ef87cfc1
> MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062300018) =
> 73b02941676bfeea2c08f5d834cf84a8
> 
> On Friday 18 August 2006 04:32 pm, Unidata LDM Support wrote:
> > e485cbfa5d47ac98502ee03faf973498  20062281718.nc
> > 0a401fdd3f3adaa04b445ff2ba199fa1  20062291618.nc
> > e275f790f6fc76a43c0c4482bce62c11  20062292000.nc
> > b836c229686986bb27af64b01a3f5515  20062292012.nc
> > 6278f6af82dd21e1ea3068bf7d95d74e  20062292048.nc
> > 9a3123d6d05ee6b56a161c12ef87cfc1  20062292054.nc
> > 73b02941676bfeea2c08f5d834cf84a8  20062300018.nc

Yup, they are the same.

Here is some more information.  It appears that ESRL/GSD is sending
two files that have the exact same product ID about 1.5 seconds apart.
Here is one example:

notifyme -vxl- -o 3600 -f FSL2 -p RASS
 ...
Aug 18 22:39:31 notifyme[4803] INFO: 7ba6c854b39e81d31b38cd89749e4416     8080 
20060818213840.994    FSL2 000  FSL.NetCDF.NOAAnet.RASS.06min.20062302124
Aug 18 22:39:31 notifyme[4803] INFO: 582443d0a3b3a2c88e7cb4e253b15fcb     8080 
20060818213901.871    FSL2 000  FSL.NetCDF.NOAAnet.RASS.06min.20062302124
Aug 18 22:39:31 notifyme[4803] INFO: ec2c8ecfd4420c1e9ea0927d2f9de897     8080 
20060818214400.244    FSL2 000  FSL.NetCDF.NOAAnet.RASS.06min.20062302130
Aug 18 22:39:31 notifyme[4803] INFO: 0b393608f94cfdfe4067ce9405fbe49b     9824 
20060818214401.747    FSL2 000  FSL.NetCDF.NOAAnet.RASS.06min.20062302130

Notice that in the size of the product is different in the second pair of this 
example; interesting...

Now, if your pqact.conf processing entry looks like:

# Profiler RASS data from FSL in NetCDF format
FSL2    ^FSL\.NetCDF\.NOAAnet\.(RASS)\.(01hr|06min)\.(.*)
        FILE    data/fsl/\1/\2/\3.nc

and if the first FILE action does not finish before the second product is 
received, then
the second process can step on the first one.  Also, if the first process 
_does_ finish
before the second is kicked off, the second copy of the product will be 
appended to
the end of the first.  We verified that this was happening on our system by 
doing
an octal dump of the file and seeing that there were two netCDF files in the 
same
output file.  Because of this, we changed our pqact.conf action to include the
'-overwrite' flag to FILE:

# Profiler RASS data from FSL in NetCDF format
FSL2    ^FSL\.NetCDF\.NOAAnet\.(RASS)\.(01hr|06min)\.(.*)
        FILE    -overwrite
        data/fsl/\1/\2/\3.nc

This will insure that the second invocation does not append to the file that the
first invocation wrote to.  It does not, however, prevent the short, damaged 
files
(the ones I referenced in my first reply).

So, it is our opinion that there is a problem in the IDD insertion logic of
RASS data by ESRL/GSD.  We will need to contact them to let them know of our
opinion.

> Thanks,

No worries.


Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: BKT-485550
Department: Support LDM
Priority: Normal
Status: Closed