Re: survey results....LDM data recovery (part 1)

David,

It seems you got a good handle on the problem. I was wondering if you got
a response/contacted Harry Edmond at Washingtion to see if a consenses
could be reached about the format of  the archived data?  The problem I
see would be if a change was needed, then one of the archives would have
to change formats.  But, if the archives are for short ranges then exact
duplication could be accomplished within months. Just a thought.

Robb...



On Tue, 11 May 1999, David Wojtowicz wrote:

> 
> [Sorry, this is long.]
> 
> Thanks to all who responded to my recent questions concerning their use of
> the LDM data recovery archive.  It seems that there's a fairly strong
> consensus of opinion. 
> 
> Nearly all of the responses indicated that such a service is a useful and
> desirable thing to have.  (Presumably those who weren't interested,
> simply didn't respond)
> 
> Of those that had used our ftp site at one time or other, most responded
> that they had indeed found it useful in recovering lost data.
> 
> I asked whether it would be better to continue providing files of the raw
> feeds that can be reingested locally or if it would be better to provided
> files in a more readily useful format such as GEMPAK. 
> 
> Although a few would prefer GEMPAK files, most responded that the raw
> files where better, especially if they could be refed to pqact for local
> processing.  This is because it seems that every site does different
> things with the data they receive.  Looking at our pqact.conf file with
> its several dozen special purpose actions, I easily can believe that
> everyone else has similarly specific local customizations and processing
> steps that they need to do. So having the ability to reprocess the
> recovered data in exactly the same way that new data comes in, is the
> desired goal. 
> 
> Some other items:
> 
> Long term vs Short Term archive:
> 
>   Because the term "archive" has been used with this service, some have
> been misled to believe that this is a long term archive.  (i.e.  If you
> wanted to recover data for a case study from the big Midwest blizzard at
> the start of this year (now several months distant) you could go to this
> archive and find it). 
> 
> Although that would be really nice to be able to do,it was never the
> intention of the data recovery archive....basically due to limitations of
> online storage space required.  (Although you could keep about a year of
> compressed DDPLUS data on a 9GB disk, the storage requirements are much,
> much larger when you start adding in satellite imagery and model output,
> both of which we consider necessary for a decent case study)
> 
> Instead, it was intended to be a recovery source for recent
> (last week or so) of IDD feeds.   So, if your machine
> crashed or your network went out for a while (as seems
> more likely during interesting weather events), you could
> use it to recover the data you missed while down a few days ago.
> 
> [At the moment, the archive is very short term, which is
> why I'm working on redoing it]
> 
> The difference between long and short term is largely a question of the
> amount of online storage available.  Disk space becoming cheaper and more
> available all the time, however this is counterbalanced by the ever
> increasing volume of the data feeds.  One must also
> consider that large amounts of data require more complicated
> management and access methods.    There's also a problem with
> reingesting data older than one month that I'll mention later.
> 
> Because there are already others looking at the long term archive aspect,
> I'm mainly concentrating on the short term "recovery" of lost recent data
> for the moment. 
> 
> Redundancy:
> 
>  It was suggested that more one top tier IDD site implement such a
> recovery archive (preferably the exact same way for consistency).
> This is necessary because here at UIUC, we're not always up
> %100 of the time (as Gilbert soooo often reminds me :-)   So when
> we go down and the sites below us lose data, they can have
> someplace else to recover it from since we won't have it.
> 
> 
> This is getting pretty long so I'll stop here for the moment
> and summarize.
> 
> 
> Conclusions:  
> 
>  Yes, the data recovery archive is indeed useful and should be continued.
>  Reingestable raw data is the best format.
>  This particular archive is meant for short term recovery.
>   (although it would be nice to use the same methods for
>   long term recovery)
>  It would be good to have at least one redundant site.
> 
> Comments and discussion are most welcome.
> 
> In part 2 I hope to discuss some issues involved
> in the "reingestion" of older data and propose
> some ideas for solving these problems.
> 
> --------------------------------------------------------
>  David Wojtowicz, Research Programmer/Systems Manager
>  Department of Atmospheric Sciences Computer Services
>  University of Illinois at Urbana-Champaign
>  email: address@hidden  phone: (217)333-8390
> --------------------------------------------------------
> 
> 
> 
> 
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================