Re: Grads GDS / Re: GRIB Aggregation

Title: Re: Grads GDS / Re: GRIB Aggregation

TDS in fact allows for just what you are asking ('If it is possible to create hooks that allow TDS to read and import data/metadata from a source OPeNDAP URL') and we have a test of this implemented  (go to http://oceanwatch.pfeg.noaa.gov:8081/thredds/catalog.html and look at  'Test remote AUV data TDS'.  These data are served by OPeNDAP at MBARI, but we both aggregate and serve them through TDS at our sie.

If you want info on how the THREDDS catalogs and NcML files are set up to do this, email Jeorme King at address@hidden.  I would apprecaiae being cc:'ed ; also.

Hope this helps.

-Roy

At 10:07 AM -0400 6/15/06, dan.swank wrote:

> We have had the NARR on GDS for quite
> some time :
> http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY
> GDS works very well with GRIB data in general.
> Our particular goal is to get it on TDS due to its suporior
> metadata
> handling and WCS service.
> If it is possible to create hooks that
> allow TDS to read and import
> data/metadata from a source OPeNDAP URL,
> similar to what the Live Access
> Server does... that would be outstanding.  May pose an easier
> task to
> implement as well.
> -Dan
> Rob Cermak wrote the following on 6/14/2006 6:50 PM:
> > Dan,
> >
> > The Grads GDS OPeNDAP server is suppose to work with GRIB files. 
> The
> > large unknown is if TDS can then
> aggregate the data across the Grads GDS
> > server.  To help answer that is to see if TDS can aggregate
> across maybe
> > one of the NOMADs GFS servers?  I'm pretty sure they are
> using the Grads
> > GDS server.
> >
> > http://www.iges.org/grads/gds/
> >
> > Rob
> >
> > On Tue, June 13, 2006 2:28 pm, dan.swank wrote:
> >
> >>Ethan:
> >>Understood.  consider this a wish-list then :)
> >>-Dan
> >>
> >>Ethan Davis wrote the following on 6/13/2006 6:22 PM:
> >>
> >>>Hi Dan,
> >>>
> >>>Evidently, aggregation of GRIB data may not be working. I
> thought since
> >>>GRIB files are read by the netCDF-java library it would
> just work but I
> >>>just heard that there may be some issues with aggregating
> GRIB involving
> >>>how the underlying code writes indexes into the GRIB
> files. We'll have
> >>>to wait till John is back from vacation for a more
> detailed
> >>>answer/explanation.
> >>>
> >>>Sorry for the mistake.
> >>>
> >>>There currently isn't a way to have a single scan element
> scan a set of
> >>>directories.
> >>>
> >>>I believe the underlying netCDF-java code that reads GRIB
> files should
> >>>result in fairly well-formed, CF (I think) compliant
> netCDF views of the
> >>>dataset. Not sure how this would be affected by
> aggregation.
> >>>
> >>>Ethan
> >>>
> >>>dan.swank wrote:
> >>>
> >>>
> >>>>Ethan:
> >>>>
> >>>>The NARR is a reanalysis, so it don't have forecast
> times.  I would be
> >>>>a
> >>>>simple 03 hr chain (00 hr fct time) spanning 26
> years.
> >>>>
> >>>>See an existing GDS subset aggregation:
> >>>>http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_tmpprs.subset.info
> >>>>
> >>>>This will give a sense for the nature of the
> beast.
> >>>>
> >>>>The directory structure is set up as such:
> >>>>http://nomads.ncdc.noaa.gov/data/narr/
> >>>>
> >>>>
> >>>>Heres the TDS aggregation I set up while experimenting
> yesterday, on a
> >>>>non-related dataset:
> >>>>
> >>>>  <dataset name="OceanWinds Test Daily
> Aggregation"
> >>>>      ID="test/dailyagg"
> urlPath="test/agg">
> >>>>   
> <serviceName>allTest</serviceName>
> >>>>    <netcdf
> >>>>xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
> >>>>     <aggregation
> dimName="time" type="joinNew">
> >>>>       <variableAgg
> name="wind" />
> >>>>       <scan
> dateFormatMark="#yyyyMMdd"
> >>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
> >>>>suffix=".nc" />
> >>>>       <scan
> dateFormatMark="#yyyyMMdd"
> >>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
> >>>>suffix=".nc" />
> >>>>       <scan
> dateFormatMark="#yyyyMMdd"
> >>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
> >>>>suffix=".nc" />
> >>>>     </aggregation>
> >>>>     <variable name="time"
> orgName="time">
> >>>>       <attribute
> name="long_name" value="Days"/>
> >>>>      
> <attribute name="units" value="days since
> 1987-07-09" />
> >>>>     </variable>
> >>>>    </netcdf>
> >>>>  </dataset>
> >>>>
> >>>>
> >>>>Would this automatically detect the source of data
> were GRIB rather
> >>>>than
> >>>>NetCDF?  and it seems like you need to set the
> <scan> on each
> >>>>individual
> >>>>directory... Doing so the way NARR is set up would
> create one chunky
> >>>>configuration file.  Is there anyway to have this
> scan a pattern
> >>>>(YYYYMM/YYYYMMDD) of directories?
> >>>>
> >>>>I understand GRIB requires a certain amount of
> "supplemented" metadata
> >>>>for complience.  Where do you enter this?
> >>>>
> >>>>-Dan
> >>>>
> >>>>
> >>>>Ethan Davis wrote the following on 6/13/2006 1:21
> PM:
> >>>>
> >>>>
> >>>>
> >>>>>Hi Dan,
> >>>>>
> >>>>>Aggregation should work the same for GRIB as for
> netCDF files. The
> >>>>>issue
> >>>>>would be how your GRIB files are structured and
> how you want to
> >>>>>aggregate them. Our GRIB files each contain one
> full model run (all
> >>>>>parameters, all forecast times). We haven't tried
> aggregating beyond
> >>>>>that.
> >>>>>
> >>>>>We have started tracking what is available for the
> NCEP models on our
> >>>>>server. This is from the TDS 3.8 announcement
> (with links updated):
> >>>>>
> >>>>>   We also are now tracking detailed
> inventory of NCEP model output,
> >>>>>eg:
> >>>>>
> >>>>>http://motherlode.ucar.edu:8080/thredds/modelInventory/model/NCEP/NAM/CONUS_12km/
> >>>>>
> >>>>>
> >>>>>   These are all linked from the
> "collection dataset" pages; For
> >>>>>   example from
> >>>>>
> >>>>>http://motherlode.ucar.edu:8080/thredds/catalog/model/NCEP/NAM/CONUS_12km/catalog.html
> >>>>>
> >>>>>
> >>>>>   choose the top "CONUS_12_km"
> link, then choose "Available
> >>>>>Inventory"
> >>>>>   Documentation.
> >>>>>
> >>>>>One idea for this work is to eventually provide
> access to alternate
> >>>>>datasets, for instance, a dataset that contains
> all the 3hr forecast
> >>>>>times from the different runs, or one that
> contained all the 12Z valid
> >>>>>times from the different runs. Tracking these
> detailed inventories is
> >>>>>just the first step but aggregation and alternate
> groupings of the
> >>>>>data
> >>>>>is pretty interesting to think about.
> >>>>>
> >>>>>How are your GRIB files structured and what kind
> of aggregation where
> >>>>>you thinking about?
> >>>>>
> >>>>>Ethan
> >>>>>
> >>>>>dan.swank wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Hello,
> >>>>>>
> >>>>>>I've been tinkering with the TDS aggregation
> capabilities and they
> >>>>>>work
> >>>>>>quite well for NetCDF data, however, I can't
> seem to find anything in
> >>>>>>the docs regarding aggregating GRIB.
> >>>>>>We want to get The NARR dataset which we have
> here at NCDC-NOMADS on
> >>>>>>the
> >>>>>>TDS.  It consists of hundreds of
> thousands of 50 Mb + GRIB files in a
> >>>>>>YYYYMM/YYYYMMDD tree.
> >>>>>>  Just scouting for a quick answer
> here:
> >>>>>>Is aggregating the NARR GRIB currently
> feasable with the current
> >>>>>>release
> >>>>>>of TDS?  If so, do any docs exist which
> could give me a starting
> >>>>>>point?
> >>>>>>Converting it to NetCDF will not be possible
> (volume).
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>--
> >>Dan Swank <address@hidden>
> >>NOMADS Project:  Software & Data Management
> >>Contractor - STG, Incorporated
> >>Veach-Baley Federal Building
> >>151 Patton Avenue
> >>Asheville, NC 28801-5001
> >>Phone: 828-271-4007
> >>
> >>===============================================================================
> >>To unsubscribe thredds, visit:
> >>http://www.unidata.ucar.edu/mailing-list-delete-form.html
> >>===============================================================================
> >>
> >
> >
> >
>
> --
> Dan Swank <address@hidden>
> NOMADS Project:  Software & Data Management
> Contractor - STG, Incorporated
> Veach-Baley Federal Building
> 151 Patton Avenue
> Asheville, NC 28801-5001
> Phone: 828-271-4007
>
> ===============================================================================
> To unsubscribe thredds, visit:
> http://www.unidata.ucar.edu/mailing-list-delete-form.html
> ===============================================================================

-- 

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn

Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."