Re: Fwd: FW: Re: NCEP North American Reanalysis (fwd)

hiya,

i'm a little confused about how a 1 mg file gets expanded to 1 gig. the
usual expansion from grib 1 to netcdf is about 10x, grib 2 can be 40x. i
could see a larger file if navigation information is added, but not more
than 15x.  But Glenn did state the resulting file was only about 3-4
times, that sounds reasonable. Could you point me to a file so i can take
a look?

robb...

>
> ---------- Forwarded message ----------
> From: Glenn Rutledge <address@hidden>
> Date: Tue, 19 Oct 2004 09:51:47 -0400
> Subject: Re: FW: Re: NCEP North American Reanalysis (fwd)
> To: address@hidden
> Cc: address@hidden, address@hidden, address@hidden
>
>
>  Hi Jon,
>  Just to clarify- we were trying to provide "a service" for users that
> converts these data to NetCDF, then compress it.  The mg to gb was an
> intermediary file that the routines apparently constructs to do the
> conversion - then it pares the data into NetCDF with is only about 2-
> 4 times bigger.  Not several orders of magnitude.
>
>  So- we are now just developing an informational page that points to
> Unidata web sites that users can download the utilities to do this
> conversion themselves.  So- it's just not that bad.  Hope this helps.
> Glenn
>
>  Jon Goodall wrote:
>
>
>
> Russ and Ben,
>
>  We are working with the NCDC to convert NARR reanalysis data from
> Grib to netCDF.  They reported that initial attempts to convert a 1mb
> NARR file in grib resulted in a 1gb NetCDF file.  Dr. Madment wanted
> me check with you to see if this sounds right.  I'm sure it depends on
> the specifics of the conversion routine, which Glenn Rutledge would
> know.  We want to use netCDF to store the NARR data, but can not
> justify a factor of 1,000 storage increase.
>
>  Jon
>
>  Jon Goodall
>  Center for Research in Water Resources
>  University of Texas at Austin
>
>  ----Original Message Follows----
>  From: "Glenn Rutledge" <address@hidden>
>  To: address@hidden
>  Subject: Re: NCEP North American Reanalysis (fwd)
>  Date: Tue, 12 Oct 2004 09:46:05 -0400
>
>  Hi Jon,
>  We just developed the capability to do this and find that it will not
> be wise for me to include this service for users since it takes a 1mb
> file and turns it into a 1gb file!  We have a meeting tomorrow to
> discuss some alternatives- but  did you try using grads-dods to access
> in NetCDF?  Please see COLA docs for info.
>
>  I'll let you know what we come up with on this.   Glenn
>
>  Jon Goodall wrote:
>
>  Hi Glenn,
>
>  I wanted to follow-up on the NARR in netCDF conversion you were
> working on a few weeks ago.  How's it going?
>
>  Thanks,
>  Jon Goodall
>  Center for Research in Water Resources
>  University of Texas at Austin
>
>
>  ________________________________
>
>
>
> Subject: Re: NCEP North American Reanalysis (fwd)
>
> From: "Glenn Rutledge" <address@hidden>
>
> Date: Tue, 12 Oct 2004 09:46:05 -0400
>
> To: address@hidden
>  Hi Jon,
>  We just developed the capability to do this and find that it will not
> be wise for me to include this service for users since it takes a 1mb
> file and turns it into a 1gb file!  We have a meeting tomorrow to
> discuss some alternatives- but  did you try using grads-dods to access
> in NetCDF?  Please see COLA docs for info.
>
>  I'll let you know what we come up with on this.   Glenn
>
>  Jon Goodall wrote:
>
> Hi Glenn,
>
>  I wanted to follow-up on the NARR in netCDF conversion you were
> working on a few weeks ago.  How's it going?
>
>  Thanks,
>  Jon Goodall
>  Center for Research in Water Resources
>  University of Texas at Austin
>
>  ----Original Message Follows----
>  From: "Glenn Rutledge" <address@hidden>
>  To: address@hidden
>  CC: address@hidden,  address@hidden,
> address@hidden,  address@hidden,
> address@hidden,  address@hidden,
> address@hidden,  address@hidden,
> address@hidden,  address@hidden,
> address@hidden,  address@hidden
>  Subject: Re: NCEP North American Reanalysis (fwd)
>  Date: Fri, 01 Oct 2004 10:28:12 -0400
>
>
>  ________________________________
>
>
>
> Subject: Re: NCEP North American Reanalysis (fwd)
>
> From: "Glenn Rutledge" <address@hidden>
>
> Date: Fri, 01 Oct 2004 10:28:12 -0400
>
> To: address@hidden
>  Thanks Jon,
>  I stand corrected- it does work, but for a limited volume only (as in
> your example!).  The conversion is rather straightforward, and w/o
> issues it could be by next week. Glenn
>
>  Jon Goodall wrote:
>
> Hi All,
>
>  I've been able to download a subset of the NARR data (only
> surface-level values from a week at the 3 hour time step) from the
> NCDC website using the ftp4u link.  It works great and I had no
> problems.  If we also have the  grib2nc conversion, we will be all
> set. Glenn, when do you hope to has this conversion available?
>
>  Jon Goodall
>  Center for Research in Water Resources
>  University of Texas at Austin
>
>
>  ----Original Message Follows----
>  From: "Glenn Rutledge" <address@hidden>
>  To: Steve Worley <address@hidden>
>  CC: Ben Domenico <address@hidden>,  David Maidment
> <address@hidden>, Jon Goodall <address@hidden>,
> "Dennis P. Lettenmaier" <address@hidden>, support-ldm
> <address@hidden>,  Martin Benno Blumenthal
> <address@hidden>, Don Hooper <address@hidden>,
> Roland Schweitzer <address@hidden>, Robb Kambic
> <address@hidden>,  Sharon Leduc <address@hidden>,
> Russell Vose <address@hidden>, Dan Swank <address@hidden>
>  Subject: Re: NCEP North American Reanalysis (fwd)
>  Date: Fri, 01 Oct 2004 09:04:49 -0400
>
>
>  ________________________________
>
>
>
> Subject: Re: NCEP North American Reanalysis (fwd)
>
> From: "Glenn Rutledge" <address@hidden>
>
> Date: Fri, 01 Oct 2004 09:04:49 -0400
>
> To: Steve Worley <address@hidden>
>  Thanks Steve and Ben for your forward.
>
>  We are now ingesting and have avbl the NARR.  We have 1987 thru 2003
> (of the 1979-2004) as requested below.  NCAR and SDSC will also have
> copies.  Here at NCDC, Murphy got in the way as I usually have grib
> sub-seting capabilities,  however the RAID I expected in for this
> operation has not yet arrived!  I'm told the check is in the mail.  To
> get around this, the NOMADS team (all 2.5 of us) put together direct
> access http to (m)get at the data due to this hardware delay.
>
>  So, users may access the data as per Steve's url below.  Use the http
> link to get at the data directories.  Depending on your internet
> connection, it may be a slow process (maybe open 2 ports).
>
>  By next week (or so), please point users to the "ftp4u" capability
> that can subset this large (5TB) dataset by parameter space and time
> (and by variable).   There is dataset documentation also avbl at the
> site.
>
>  As I write, we are developing a grib2nc capability for users to
> retrieve in NetCDF format.  We will also compress the NetCDF for
> transmission (GFDL, Kerr) found that these NetCDF files compress very
> well.   Please let me know your experiences with this site and do try
> to subset when that's avbl.  Thanks, Glenn
>
>  Steve Worley wrote:
>
> All,
>
>  North American Regional Reanalysis is just coming available from
>  two location I know of:
>
>  the NCDC NOMADS server (Glenn's work) at
>  http://nomads.ncdc.noaa.gov/data-access.html#narr_datasets
>
>  and NCAR through the Data Support Section.
>  We have 1987-2003 archived, but not fully ready for service.  However,
>  if you wish to let us know what you require we might be able to
>  accomodate your needs without too much problem.  Chi-Fan Shih
>  (address@hidden) can address your further questions about NARR
>  from NCAR.
>
>  sjw
>
>
>  Ben Domenico wrote:
>
> Hi David, Jon and Dennis,
>
>  Your questions are flattering in terms of the assumed breadth and
> depth of knowledge about data collections here at the Unidata Program
> Center (UPC), but we should be honest and offer a few clarifications
> that will help you find what you are looking for more quickly.
>
>  The main clarification is that the staff at the UPC mainly focuses on
> infrastructure for real-time data systems for weather data.   I
> understand that archived climate information is closely related, but
> your questions about the details of specific archived climate datasets
> are skirting pretty much at and beyond the edges of our UPC expertise.
>  Those sorts of questions are better posed to the folk who create and
> maintain the climate data archives at places like the  NCDC, NCAR,
> Lamont-Doherty, and CDC.  I plan to send a copy of this note to Glenn
> Rutledge at NCDC, Benno Blumenthal at LDEO, Steve Worley at NCAR, and
> Don Hooper & Roland Schweitzer at CDC who should be able to help you
> find what you need in terms of the North American Reanalysis datasets
> -- in particular a netCDF version.
>
>  On the other hand, we do have experience at the UPC with decoders
> that transform certain GRIB datasets into netCDF form (indeed with
> considerably expanded size) and I'll ask Robb to follow up on that
> issue to see if there is a version of that (GRIBTONC) decoder for the
> North American Reanalysis data.
>
>  I don't mean to be passing the buck, but there are many groups who
> have more detailed expertise than the UPC about archives of climate
> datasets.
>
>  --Ben
>  ******************************************************
>    Ben Domenico                Unidata Deputy Director
>    http://my.unidata.ucar.edu            P.O. Box 3000
>    address@hidden              Boulder, CO 80307
>    (303)497-8631                    FAX: (303)497-8690
>
>  --On Thursday, September 30, 2004 3:45 PM -0500 David Maidment
> <address@hidden> wrote:
>
>
> Robb:
>
>  Dennis Lettenmaier is correct here.  What you are pointing us to is
>  the global NCEP reanalysis, which is on a 2.5 degree grid and not
>  very informative for the hydrology of the US.  What we are seeking
>  to work with is the reanalysis for North America alone, which is
>  on a 32km grid and is in Grib format, not netCDF.   The global
>  reanalysis is available in netCDF.
>
>  David
>
>
>  -----Original Message-----
>  From: Robb Kambic [mailto:address@hidden]
>  Sent: Thursday, September 30, 2004 11:25 AM
>  To: Ben Domenico
>  Cc: David Maidment; support-ldm
>  Subject: Re: NCEP North American Reanalysis (fwd)
>
>
>
>  look at url:
>
>  http://www.cdc.noaa.gov/cdc/data.ncep.reanalysis.html#surface_gauss
>
>  parameters  water runoff, soil moisture, etc.  these files are in netcdf
>  so they should be usuable for your analysis. i downloaded a couple of
>  files and dumped them using ncdump.
>
>  there are proably other hydrologic fluxes data sets that can be found by
>  searching the web.
>
>  robb...
>
>  On Thu, 30 Sep 2004, Ben Domenico wrote:
>
>
> Hi all,
>
>  The note below from David Maidment asks about getting the NCAR
>  reanalysis products into netCDF form.  Is there something special
>  about these datasets or would our grib to netcdf decoder work with
>  them.  Regarding the grads part of it, I'm assuming that refers to the
>
>
>
> fact that they are available via the GDS protocol, but let me know if
>  there's more to it than that.
>
>  And how about the CDC archives?  I thought they published these data
>  in netCDF form.
>
>  Thanks.
>
>  -- Ben
>
>  ---------- Forwarded Message ----------
>  Date: Thursday, September 30, 2004 8:18 AM -0500
>  From: David Maidment <address@hidden>
>  To: Ben Domenico <address@hidden>
>  Subject: NCEP North American Reanalysis
>
>
>  Ben:
>
>  In CUAHSI HIS, we are thinking that a good "coverage" of hydrologic
>  fluxes from the atmosphere could be obtained from the NCEP North
>  American Reanalysis. More info is at
>  http://www.emc.ncep.noaa.gov/mmb/rreanl/
>
>  It seems like the data are all stored in one of your sister
>  institutions in Boulder.
>
>  Could you please help us to interpret what the data available are?
>
>  The data are in Grib/Grads format.   We have invested a lot of effort
>
>  in
>
>
> working
>  with netCDF and I don't want Jon Goodall or others in our CUAHSI
>
>  community
>
>
> having to tussle with Grib/Grads.   We have another other data formats
>  already
>  with GIS, point observations and remote sensing to consider.
>
>  Would it be possible to translate these reanalysis data layers into
>  netCDF format?
>
>  David
>
>
>  ---------- End Forwarded Message ----------
>
>
>
>
>  ========================================================================
>  =======
>  Robb Kambic                       Unidata Program Center
>  Software Engineer III               Univ. Corp for Atmospheric
>  Research
>  address@hidden           WWW:
>  http://www.unidata.ucar.edu/
>  ========================================================================
>  =======
>

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================