[IDD #MJH-494207]: Prolems in the generation of the IDD FSL2 feed RASS data

Hi,

Please pass the following along to the appropriate person(s).

Unidata User Support was contacted by a user who is ingesting the FSL2 6-minute
RASS data asking about two problems that she is seeing:

  "Some of the files are corrupted and can not be read using ncdump. Others have
   bad data in them. For example, the levels are missing, duplicated or not in
   order."

We verified that we have received a number of RASS products that could not be
dumped using 'ncdump'.  One interesting feature of these files is that they
appear to be of the same small size.  Here is an example listing of a set of
bad files:

-rw-rw-r-- 1 ldm ustaff 3284 Aug 16 11:31 20062281718.nc
-rw-rw-r-- 1 ldm ustaff 3284 Aug 17 10:32 20062291618.nc
-rw-rw-r-- 1 ldm ustaff 3284 Aug 17 14:14 20062292000.nc
-rw-rw-r-- 1 ldm ustaff 3284 Aug 17 14:25 20062292012.nc
-rw-rw-r-- 1 ldm ustaff 3284 Aug 17 15:01 20062292048.nc
-rw-rw-r-- 1 ldm ustaff 3284 Aug 17 15:07 20062292054.nc
-rw-rw-r-- 1 ldm ustaff 3284 Aug 17 18:31 20062300018.nc

We verified that the products we received and FILEd had the same MD5
checksum as the products that the user received and FILEd.  This gives
us confidence that there is no problem in the LDM relaying the data.

We were also able to verify that some of the products we receive have levels
that were not complete and/or in the "correct" order.  Here is a snippit
of an 'ncdump' output showing the value of the 'level' parameter in the
20062281718.nc file written on our LDM ingest/decode machine:

 level = 2500, 2750, 3000, 1250, 1500, 1750, 2000, 2250, 2500, 2750, 3000,
3250, 3500, 3750, 4000, 4250, 4500, 4750, 5000, 5250, 5500, 5750 ;

We believe that the levels that should be included in the file are:

 level = 500, 750, 1000, 1250, 1500, 1750, 2000, 2250, 2500,
2750, 3000, 3250, 3500, 3750, 4000, 4250, 4500, 4750, 5000, 5250, 5500,
5750 ;

While investigating these problems, we noticed that the FSL2 datastream
contains pairs of products with the same product IDs a few seconds
apart.  Here is one example that illustrates this:

notifyme -vxl- -o 3600 -f FSL2 -p RASS -h idd.unidata.ucar.edu
...
Aug 18 22:39:31 notifyme[4803] INFO: 7ba6c854b39e81d31b38cd89749e4416 8080 
20060818213840.994 FSL2 000 FSL.NetCDF.NOAAnet.RASS.06min.20062302124
Aug 18 22:39:31 notifyme[4803] INFO: 582443d0a3b3a2c88e7cb4e253b15fcb 8080 
20060818213901.871 FSL2 000 FSL.NetCDF.NOAAnet.RASS.06min.20062302124
Aug 18 22:39:31 notifyme[4803] INFO: ec2c8ecfd4420c1e9ea0927d2f9de897 8080 
20060818214400.244 FSL2 000 FSL.NetCDF.NOAAnet.RASS.06min.20062302130
Aug 18 22:39:31 notifyme[4803] INFO: 0b393608f94cfdfe4067ce9405fbe49b 9824 
20060818214401.747 FSL2 000 FSL.NetCDF.NOAAnet.RASS.06min.20062302130

Notice that in the first pair the file sizes are identical (8080 bytes) while 
in the second they
are different (8080 versus 9824 bytes).

We are wondering if there is a problem in the logic that is injecting the 
6-minute RASS data
into the originating LDM queue?  Perhaps the insertion process is being kicked 
off prematurely
(the insertion of the first product) and then again when the product is 
actually ready?

Any help you can provide in this matter would be greatly appreciated!

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************