[LDM #BKT-485550]: Problems with profiler data

Hi Laurie,

re:
> As I mentioned in a previous e-mail we are having some problems with
> corrupted profiler data from our IDD LDM feed. The problems started on
> 7/20/06. At first we thought it may have been a result of changes we made to
> our configuration around that time frame. However, we moved back to our
> original configuration yesterday and are still getting errors. Perhaps you
> can help us figure out where the problem is.
> 
> We pick up the FSL profiler data. We get the 6 minute and hourly RASS and wind
> data matching the following patterns: FSL.NetCDF.NOAAnet.RASS.06min*,
> FSL.NetCDF.NOAAnet.windprofiler.06min*,FSL.NetCDF.NOAAnet.RASS.01hr* and
> FSL.NetCDF.NOAAnet.windprofiler.01hr*.
> 
> Some of the files are corrupted and can not be read using ncdump. Others have
> bad data in them. For example, the levels are missing, duplicated or not in
> order.
> 
> Here is an example list of corrupt files from today.
> 
> This is the list of 6 minute RASS data unreadable by ncdump:
> FSL.NetCDF.NOAAnet.RASS.06min.20062281718
> FSL.NetCDF.NOAAnet.RASS.06min.20062291618
> FSL.NetCDF.NOAAnet.RASS.06min.20062292000
> FSL.NetCDF.NOAAnet.RASS.06min.20062292012.
> FSL.NetCDF.NOAAnet.RASS.06min.20062292048
> FSL.NetCDF.NOAAnet.RASS.06min.20062292054
> FSL.NetCDF.NOAAnet.RASS.06min.20062300018

I just looked at the same files on one of our data ingest/decode machines and
see that the data we FILEd for the same times were also bad.  I notice that
the files on our system are of the same, exceedingly small size:

-rw-rw-r--  1 ldm  ustaff  3284 Aug 16 11:31 20062281718.nc
-rw-rw-r--  1 ldm  ustaff  3284 Aug 17 10:32 20062291618.nc
-rw-rw-r--  1 ldm  ustaff  3284 Aug 17 14:14 20062292000.nc
-rw-rw-r--  1 ldm  ustaff  3284 Aug 17 14:25 20062292012.nc
-rw-rw-r--  1 ldm  ustaff  3284 Aug 17 15:01 20062292048.nc
-rw-rw-r--  1 ldm  ustaff  3284 Aug 17 15:07 20062292054.nc
-rw-rw-r--  1 ldm  ustaff  3284 Aug 17 18:31 20062300018.nc

What are the size of these files on your disk?  If they are the same
as ours, it suggests that the files could have been bad when they were
first injected into the IDD by ESRL/GSD (formerly FSL).

As a sanity check, here are the MD5 checksums for each of the files on
our disk:

e485cbfa5d47ac98502ee03faf973498  20062281718.nc
0a401fdd3f3adaa04b445ff2ba199fa1  20062291618.nc
e275f790f6fc76a43c0c4482bce62c11  20062292000.nc
b836c229686986bb27af64b01a3f5515  20062292012.nc
6278f6af82dd21e1ea3068bf7d95d74e  20062292048.nc
9a3123d6d05ee6b56a161c12ef87cfc1  20062292054.nc
73b02941676bfeea2c08f5d834cf84a8  20062300018.nc

Please calculate the MD5 checksum for the same files on your system and
compare them against this list.  If the MD5 checksums are the same, then
there is nothing wrong with your ingestion or processing; the files were
likely bad when injected into the IDD.

> This is a list where the levels are duplicated, out of order, etc...
> They should be:  level = 500, 750, 1000, 1250, 1500, 1750, 2000, 2250, 2500,
> 2750, 3000, 3250, 3500, 3750, 4000, 4250, 4500, 4750, 5000, 5250, 5500,
> 5750 ;
> 
> FSL.NetCDF.NOAAnet.RASS.06min.20062281818.levdiff:<  level = 2500, 2750, 3000,
> 500, 750, 1000, 1250, 1500, 1750, 2000, 2250,

I get the following levels for the 20062281818 6-minute RASS data
that we ingested:

 level = 2500, 2750, 3000, 1250, 1500, 1750, 2000, 2250, 2500, 2750, 3000,
    3250, 3500, 3750, 4000, 4250, 4500, 4750, 5000, 5250, 5500, 5750 ;

Since I am not sure what your 'levdiff' output is supposed to represent, I
can not say that the set of levels in our file is different from the set of
levels in your file.  I can say, however, that the set of levels in our
file does not match the list of levels that should be in the file.

 ...
> FSL.NetCDF.NOAAnet.RASS.06min.20062301030.levdiff:<  level = _, _, _, _, _, _,
> _, _, _, _, _, 3250, 3500, 3750, 4000, 4250, 4500

It sounds like there may be some processing problems at ESRL/GSD.  We will 
better
know if this is the case after you do the MD5 checksum comparisons I suggest
above.

> Thank you,

No worries.

> Laurie
> --
> address@hidden
> ARM External Data Center
> Brookhaven National Laboratory
> www.xdc.arm.gov, www.arm.gov

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: BKT-485550
Department: Support LDM
Priority: Normal
Status: Closed