This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hi Laurie, re: > Thanks for checking this. It looks like our MD5's match yours. > I guess this points to a problem from ESRL/GSD? I would think so, yes. > Here is the output from our MD5 checks: > MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062281718) = > e485cbfa5d47ac98502ee03faf973498 > MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062291618) = > 0a401fdd3f3adaa04b445ff2ba199fa1 > MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062292000) = > e275f790f6fc76a43c0c4482bce62c11 > MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062292012) = > b836c229686986bb27af64b01a3f5515 > MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062292048) = > 6278f6af82dd21e1ea3068bf7d95d74e > MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062292054) = > 9a3123d6d05ee6b56a161c12ef87cfc1 > MD5 (FSL.NetCDF.NOAAnet.RASS.06min.20062300018) = > 73b02941676bfeea2c08f5d834cf84a8 > > On Friday 18 August 2006 04:32 pm, Unidata LDM Support wrote: > > e485cbfa5d47ac98502ee03faf973498 20062281718.nc > > 0a401fdd3f3adaa04b445ff2ba199fa1 20062291618.nc > > e275f790f6fc76a43c0c4482bce62c11 20062292000.nc > > b836c229686986bb27af64b01a3f5515 20062292012.nc > > 6278f6af82dd21e1ea3068bf7d95d74e 20062292048.nc > > 9a3123d6d05ee6b56a161c12ef87cfc1 20062292054.nc > > 73b02941676bfeea2c08f5d834cf84a8 20062300018.nc Yup, they are the same. Here is some more information. It appears that ESRL/GSD is sending two files that have the exact same product ID about 1.5 seconds apart. Here is one example: notifyme -vxl- -o 3600 -f FSL2 -p RASS ... Aug 18 22:39:31 notifyme[4803] INFO: 7ba6c854b39e81d31b38cd89749e4416 8080 20060818213840.994 FSL2 000 FSL.NetCDF.NOAAnet.RASS.06min.20062302124 Aug 18 22:39:31 notifyme[4803] INFO: 582443d0a3b3a2c88e7cb4e253b15fcb 8080 20060818213901.871 FSL2 000 FSL.NetCDF.NOAAnet.RASS.06min.20062302124 Aug 18 22:39:31 notifyme[4803] INFO: ec2c8ecfd4420c1e9ea0927d2f9de897 8080 20060818214400.244 FSL2 000 FSL.NetCDF.NOAAnet.RASS.06min.20062302130 Aug 18 22:39:31 notifyme[4803] INFO: 0b393608f94cfdfe4067ce9405fbe49b 9824 20060818214401.747 FSL2 000 FSL.NetCDF.NOAAnet.RASS.06min.20062302130 Notice that in the size of the product is different in the second pair of this example; interesting... Now, if your pqact.conf processing entry looks like: # Profiler RASS data from FSL in NetCDF format FSL2 ^FSL\.NetCDF\.NOAAnet\.(RASS)\.(01hr|06min)\.(.*) FILE data/fsl/\1/\2/\3.nc and if the first FILE action does not finish before the second product is received, then the second process can step on the first one. Also, if the first process _does_ finish before the second is kicked off, the second copy of the product will be appended to the end of the first. We verified that this was happening on our system by doing an octal dump of the file and seeing that there were two netCDF files in the same output file. Because of this, we changed our pqact.conf action to include the '-overwrite' flag to FILE: # Profiler RASS data from FSL in NetCDF format FSL2 ^FSL\.NetCDF\.NOAAnet\.(RASS)\.(01hr|06min)\.(.*) FILE -overwrite data/fsl/\1/\2/\3.nc This will insure that the second invocation does not append to the file that the first invocation wrote to. It does not, however, prevent the short, damaged files (the ones I referenced in my first reply). So, it is our opinion that there is a problem in the IDD insertion logic of RASS data by ESRL/GSD. We will need to contact them to let them know of our opinion. > Thanks, No worries. Cheers, Tom **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: BKT-485550 Department: Support LDM Priority: Normal Status: Closed