[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Fwd: FW: Re: NCEP North American Reanalysis (fwd)
- Subject: Re: Fwd: FW: Re: NCEP North American Reanalysis (fwd)
- Date: Tue, 19 Oct 2004 11:51:10 -0600 (MDT)
hiya,
i'm a little confused about how a 1 mg file gets expanded to 1 gig. the
usual expansion from grib 1 to netcdf is about 10x, grib 2 can be 40x. i
could see a larger file if navigation information is added, but not more
than 15x. But Glenn did state the resulting file was only about 3-4
times, that sounds reasonable. Could you point me to a file so i can take
a look?
robb...
>
> ---------- Forwarded message ----------
> From: Glenn Rutledge <address@hidden>
> Date: Tue, 19 Oct 2004 09:51:47 -0400
> Subject: Re: FW: Re: NCEP North American Reanalysis (fwd)
> To: address@hidden
> Cc: address@hidden, address@hidden, address@hidden
>
>
> Hi Jon,
> Just to clarify- we were trying to provide "a service" for users that
> converts these data to NetCDF, then compress it. The mg to gb was an
> intermediary file that the routines apparently constructs to do the
> conversion - then it pares the data into NetCDF with is only about 2-
> 4 times bigger. Not several orders of magnitude.
>
> So- we are now just developing an informational page that points to
> Unidata web sites that users can download the utilities to do this
> conversion themselves. So- it's just not that bad. Hope this helps.
> Glenn
>
> Jon Goodall wrote:
>
>
>
> Russ and Ben,
>
> We are working with the NCDC to convert NARR reanalysis data from
> Grib to netCDF. They reported that initial attempts to convert a 1mb
> NARR file in grib resulted in a 1gb NetCDF file. Dr. Madment wanted
> me check with you to see if this sounds right. I'm sure it depends on
> the specifics of the conversion routine, which Glenn Rutledge would
> know. We want to use netCDF to store the NARR data, but can not
> justify a factor of 1,000 storage increase.
>
> Jon
>
> Jon Goodall
> Center for Research in Water Resources
> University of Texas at Austin
>
> ----Original Message Follows----
> From: "Glenn Rutledge" <address@hidden>
> To: address@hidden
> Subject: Re: NCEP North American Reanalysis (fwd)
> Date: Tue, 12 Oct 2004 09:46:05 -0400
>
> Hi Jon,
> We just developed the capability to do this and find that it will not
> be wise for me to include this service for users since it takes a 1mb
> file and turns it into a 1gb file! We have a meeting tomorrow to
> discuss some alternatives- but did you try using grads-dods to access
> in NetCDF? Please see COLA docs for info.
>
> I'll let you know what we come up with on this. Glenn
>
> Jon Goodall wrote:
>
> Hi Glenn,
>
> I wanted to follow-up on the NARR in netCDF conversion you were
> working on a few weeks ago. How's it going?
>
> Thanks,
> Jon Goodall
> Center for Research in Water Resources
> University of Texas at Austin
>
>
> ________________________________
>
>
>
> Subject: Re: NCEP North American Reanalysis (fwd)
>
> From: "Glenn Rutledge" <address@hidden>
>
> Date: Tue, 12 Oct 2004 09:46:05 -0400
>
> To: address@hidden
> Hi Jon,
> We just developed the capability to do this and find that it will not
> be wise for me to include this service for users since it takes a 1mb
> file and turns it into a 1gb file! We have a meeting tomorrow to
> discuss some alternatives- but did you try using grads-dods to access
> in NetCDF? Please see COLA docs for info.
>
> I'll let you know what we come up with on this. Glenn
>
> Jon Goodall wrote:
>
> Hi Glenn,
>
> I wanted to follow-up on the NARR in netCDF conversion you were
> working on a few weeks ago. How's it going?
>
> Thanks,
> Jon Goodall
> Center for Research in Water Resources
> University of Texas at Austin
>
> ----Original Message Follows----
> From: "Glenn Rutledge" <address@hidden>
> To: address@hidden
> CC: address@hidden, address@hidden,
> address@hidden, address@hidden,
> address@hidden, address@hidden,
> address@hidden, address@hidden,
> address@hidden, address@hidden,
> address@hidden, address@hidden
> Subject: Re: NCEP North American Reanalysis (fwd)
> Date: Fri, 01 Oct 2004 10:28:12 -0400
>
>
> ________________________________
>
>
>
> Subject: Re: NCEP North American Reanalysis (fwd)
>
> From: "Glenn Rutledge" <address@hidden>
>
> Date: Fri, 01 Oct 2004 10:28:12 -0400
>
> To: address@hidden
> Thanks Jon,
> I stand corrected- it does work, but for a limited volume only (as in
> your example!). The conversion is rather straightforward, and w/o
> issues it could be by next week. Glenn
>
> Jon Goodall wrote:
>
> Hi All,
>
> I've been able to download a subset of the NARR data (only
> surface-level values from a week at the 3 hour time step) from the
> NCDC website using the ftp4u link. It works great and I had no
> problems. If we also have the grib2nc conversion, we will be all
> set. Glenn, when do you hope to has this conversion available?
>
> Jon Goodall
> Center for Research in Water Resources
> University of Texas at Austin
>
>
> ----Original Message Follows----
> From: "Glenn Rutledge" <address@hidden>
> To: Steve Worley <address@hidden>
> CC: Ben Domenico <address@hidden>, David Maidment
> <address@hidden>, Jon Goodall <address@hidden>,
> "Dennis P. Lettenmaier" <address@hidden>, support-ldm
> <address@hidden>, Martin Benno Blumenthal
> <address@hidden>, Don Hooper <address@hidden>,
> Roland Schweitzer <address@hidden>, Robb Kambic
> <address@hidden>, Sharon Leduc <address@hidden>,
> Russell Vose <address@hidden>, Dan Swank <address@hidden>
> Subject: Re: NCEP North American Reanalysis (fwd)
> Date: Fri, 01 Oct 2004 09:04:49 -0400
>
>
> ________________________________
>
>
>
> Subject: Re: NCEP North American Reanalysis (fwd)
>
> From: "Glenn Rutledge" <address@hidden>
>
> Date: Fri, 01 Oct 2004 09:04:49 -0400
>
> To: Steve Worley <address@hidden>
> Thanks Steve and Ben for your forward.
>
> We are now ingesting and have avbl the NARR. We have 1987 thru 2003
> (of the 1979-2004) as requested below. NCAR and SDSC will also have
> copies. Here at NCDC, Murphy got in the way as I usually have grib
> sub-seting capabilities, however the RAID I expected in for this
> operation has not yet arrived! I'm told the check is in the mail. To
> get around this, the NOMADS team (all 2.5 of us) put together direct
> access http to (m)get at the data due to this hardware delay.
>
> So, users may access the data as per Steve's url below. Use the http
> link to get at the data directories. Depending on your internet
> connection, it may be a slow process (maybe open 2 ports).
>
> By next week (or so), please point users to the "ftp4u" capability
> that can subset this large (5TB) dataset by parameter space and time
> (and by variable). There is dataset documentation also avbl at the
> site.
>
> As I write, we are developing a grib2nc capability for users to
> retrieve in NetCDF format. We will also compress the NetCDF for
> transmission (GFDL, Kerr) found that these NetCDF files compress very
> well. Please let me know your experiences with this site and do try
> to subset when that's avbl. Thanks, Glenn
>
> Steve Worley wrote:
>
> All,
>
> North American Regional Reanalysis is just coming available from
> two location I know of:
>
> the NCDC NOMADS server (Glenn's work) at
> http://nomads.ncdc.noaa.gov/data-access.html#narr_datasets
>
> and NCAR through the Data Support Section.
> We have 1987-2003 archived, but not fully ready for service. However,
> if you wish to let us know what you require we might be able to
> accomodate your needs without too much problem. Chi-Fan Shih
> (address@hidden) can address your further questions about NARR
> from NCAR.
>
> sjw
>
>
> Ben Domenico wrote:
>
> Hi David, Jon and Dennis,
>
> Your questions are flattering in terms of the assumed breadth and
> depth of knowledge about data collections here at the Unidata Program
> Center (UPC), but we should be honest and offer a few clarifications
> that will help you find what you are looking for more quickly.
>
> The main clarification is that the staff at the UPC mainly focuses on
> infrastructure for real-time data systems for weather data. I
> understand that archived climate information is closely related, but
> your questions about the details of specific archived climate datasets
> are skirting pretty much at and beyond the edges of our UPC expertise.
> Those sorts of questions are better posed to the folk who create and
> maintain the climate data archives at places like the NCDC, NCAR,
> Lamont-Doherty, and CDC. I plan to send a copy of this note to Glenn
> Rutledge at NCDC, Benno Blumenthal at LDEO, Steve Worley at NCAR, and
> Don Hooper & Roland Schweitzer at CDC who should be able to help you
> find what you need in terms of the North American Reanalysis datasets
> -- in particular a netCDF version.
>
> On the other hand, we do have experience at the UPC with decoders
> that transform certain GRIB datasets into netCDF form (indeed with
> considerably expanded size) and I'll ask Robb to follow up on that
> issue to see if there is a version of that (GRIBTONC) decoder for the
> North American Reanalysis data.
>
> I don't mean to be passing the buck, but there are many groups who
> have more detailed expertise than the UPC about archives of climate
> datasets.
>
> --Ben
> ******************************************************
> Ben Domenico Unidata Deputy Director
> http://my.unidata.ucar.edu P.O. Box 3000
> address@hidden Boulder, CO 80307
> (303)497-8631 FAX: (303)497-8690
>
> --On Thursday, September 30, 2004 3:45 PM -0500 David Maidment
> <address@hidden> wrote:
>
>
> Robb:
>
> Dennis Lettenmaier is correct here. What you are pointing us to is
> the global NCEP reanalysis, which is on a 2.5 degree grid and not
> very informative for the hydrology of the US. What we are seeking
> to work with is the reanalysis for North America alone, which is
> on a 32km grid and is in Grib format, not netCDF. The global
> reanalysis is available in netCDF.
>
> David
>
>
> -----Original Message-----
> From: Robb Kambic [mailto:address@hidden]
> Sent: Thursday, September 30, 2004 11:25 AM
> To: Ben Domenico
> Cc: David Maidment; support-ldm
> Subject: Re: NCEP North American Reanalysis (fwd)
>
>
>
> look at url:
>
> http://www.cdc.noaa.gov/cdc/data.ncep.reanalysis.html#surface_gauss
>
> parameters water runoff, soil moisture, etc. these files are in netcdf
> so they should be usuable for your analysis. i downloaded a couple of
> files and dumped them using ncdump.
>
> there are proably other hydrologic fluxes data sets that can be found by
> searching the web.
>
> robb...
>
> On Thu, 30 Sep 2004, Ben Domenico wrote:
>
>
> Hi all,
>
> The note below from David Maidment asks about getting the NCAR
> reanalysis products into netCDF form. Is there something special
> about these datasets or would our grib to netcdf decoder work with
> them. Regarding the grads part of it, I'm assuming that refers to the
>
>
>
> fact that they are available via the GDS protocol, but let me know if
> there's more to it than that.
>
> And how about the CDC archives? I thought they published these data
> in netCDF form.
>
> Thanks.
>
> -- Ben
>
> ---------- Forwarded Message ----------
> Date: Thursday, September 30, 2004 8:18 AM -0500
> From: David Maidment <address@hidden>
> To: Ben Domenico <address@hidden>
> Subject: NCEP North American Reanalysis
>
>
> Ben:
>
> In CUAHSI HIS, we are thinking that a good "coverage" of hydrologic
> fluxes from the atmosphere could be obtained from the NCEP North
> American Reanalysis. More info is at
> http://www.emc.ncep.noaa.gov/mmb/rreanl/
>
> It seems like the data are all stored in one of your sister
> institutions in Boulder.
>
> Could you please help us to interpret what the data available are?
>
> The data are in Grib/Grads format. We have invested a lot of effort
>
> in
>
>
> working
> with netCDF and I don't want Jon Goodall or others in our CUAHSI
>
> community
>
>
> having to tussle with Grib/Grads. We have another other data formats
> already
> with GIS, point observations and remote sensing to consider.
>
> Would it be possible to translate these reanalysis data layers into
> netCDF format?
>
> David
>
>
> ---------- End Forwarded Message ----------
>
>
>
>
> ========================================================================
> =======
> Robb Kambic Unidata Program Center
> Software Engineer III Univ. Corp for Atmospheric
> Research
> address@hidden WWW:
> http://www.unidata.ucar.edu/
> ========================================================================
> =======
>
===============================================================================
Robb Kambic Unidata Program Center
Software Engineer III Univ. Corp for Atmospheric Research
address@hidden WWW: http://www.unidata.ucar.edu/
===============================================================================