19990514: dchrly

>
>Hello,
>
>In just the last few weeks, it seems as though data enters our
>_sao.gem file at some point during the day and corrupts the file.
>Sometimes, this has shown up as enormous gempak surface data files
>(500 Meg+) that cannot be opened by programs like SFLIST.
>Today, what is happening is if I use SFLIST and happen to set DATTIM
>to an hour that doesn't contain "bad" data, it lists the data
>correctly.  However, if I pick a different hour, it stops showing
>the data and prints " [FL 168] " on the screen.....and after
>this point, no matter what hour I use in DATTIM (even hours that
>had worked previously), I get an error message saying:
> [SF -14]  There are too many times in file.
>
>This problem is causing havoc with some good research scripts we
>use at the end of each day that must look at the full 24 hours
>of data each day.  The gempak surface data file becomes essentially
>unreadable.  
>
>Have you ever encountered this type of problem before?
>
>Bill
>
>**********************************************
>*  Bill Gallus                               *
>*  Dept. of Geological and Atmospheric Sci.  *
>*  3025 Agronomy Hall                        *
>*  Iowa State University                     *
>*  Ames, IA 50011                            *
>*  (phone) 515-294-2270                      *
>*  (fax)   515-294-3163                      *
>**********************************************
>
>------- End of Forwarded Message
>

Bill,

Two reasons I can think of the above situation happening-

1) A corrupt SA bulleting is being received and is trying to be stored
   in the TEXT field of the file.

   Note that if this is the case, the problem should be reproduceable.
   I have checked and reingested and rechecked our raw data files here
   as well as the resultant files produced from dchrly and have not 
   found any corruption. I did notice this problem almost 2 years ago but
   since then provided a patch. If your version is lacking the patch,
   this could still be a problem... and we can rectify this by updating your 
   decoder or gempak distribution. Are you running under IRIX 6.5.x?

2) If your decoding machine is having a hard time keeping up with the 
   amount of incoming data, then its possible that the LDM would get
   bogged down and overflow the PIPE output to dchrly, and as a result,
   a second copy of dchrly could be started up. When two running dchrly
   programs try to write to the same file, data corruption is almost
   guaranteed. If this is the problem, you would probably notice the machine 
   running very slow, and you would probably see dchrly invocations in the data 
logs
   where dchrly is started several times without a shutting down log message.
  

I can install my current dchrly executable on your machine to see if that
solves your problem if any patches atc are missing from your distribution.
If the problem is with the machine load, then we should see if anything has 
recently started consuming a large portion of your resources.

Steve Chiswell
Unidata User Support