Re: Grads GDS / Re: GRIB Aggregation

TDS support :

I have tried to get 3 seperate aggregations working during the few hours
I've had to work on this during the past week.

In our TDS test area, Please take a look at the following 3 entries:
http://nomads.ncdc.noaa.gov:8085/thredds/testDatasets.html

  o  Blended Ocean Winds Test Daily Aggregation
  o  NARR-A Test Daily Aggregation
  o  Test Narr hgtprs Subset

The configuration file entries for these follow:

===============================================

        <!-- Test BOW Aggregation -->

  <dataset name="Blended Ocean Winds Test Daily Aggregation"
      ID="test/BOWdailyagg" urlPath="test/BOWdailyagg">
    <serviceName>allTest</serviceName>
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>

     <dimension name="time" length="6688" />

     <variable name="time" type="int" shape="time">
       <attribute name="long_name" value="Aggregated Time axis in days" />
       <attribute name="units" value="days since 1987-07-09" />
       <attribute name="_CoordinateAxisType" value="Time" />
       <values start="0" incr="1" npts="6688" />
     </variable>

     <aggregation dimName="time" type="joinNew">
       <variableAgg name="wind" />
       <scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
suffix=".nc" />
       <scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
suffix=".nc" />
       <scan dateFormatMark="#yyyyMMdd"
location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
suffix=".nc" />
     </aggregation>
    </netcdf>
  </dataset>

        <!-- Test NARR Aggregation (local) -->

  <dataset name="NARR-A Test Daily Aggregation"
      ID="test/narragg" urlPath="test/narragg">
    <access serviceName="allTest" urlPath="test/narragg"
dataFormat="GRIB-1" />
    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>

     <dimension name="time" length="8" />
     <variable name="time" type="int" shape="time">
       <attribute name="long_name" value="Aggregated Time axis in 3hr" />
       <attribute name="units" value="hours since 2006-01-01 00:00" />
       <attribute name="_CoordinateAxisType" value="Time" />
       <values start="0" incr="3" npts="8" />
     </variable>

     <aggregation dimName="time" type="joinNew">
       <variableAgg/>
       <scan dateFormatMark="#narr-a_211_yyyyMMdd_hhmm_000"
location="/nomads3_data/raid5a/data/merged_AWIP32/200601/20060101"
suffix=".grb" />
     </aggregation>
    </netcdf>
  </dataset>

        <!-- NARR hgtprs test agg (remote) -->

  <dataset
        name="Test Narr hgtprs Subset" ID="NARR/hgtprs"
        urlPath="NCEP_NARR_DAILY/narr-a_221_hgtprs.subset"
        serviceName="gds" >
    <catalogRef xlink:title="Test NARR Subset hgtprs"

xlink:href="http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_hgtprs.subset.thredds";
/>
    <serviceName>gds</serviceName>

    <variable name="Geopotential Height" orgName="hgtprs">
       <attribute name="long_name" value="Geopotential Height on
Pressure Levels" />
       <attribute name="units" value="gpm" />
    </variable>
  </dataset>

===============================================
Services defined as follows:
===============================================
 <service name="allTest" serviceType="Compound" base="">
   <service name="ncdods" serviceType="OpenDAP" base="/thredds/dodsC/"/>
   <service name="HTTPServer" serviceType="HTTPServer"
base="/thredds/fileServer/"/>
   <service name="WCS" serviceType="WCS" base="/thredds/wcs/"/>
 </service>
  <service name="gds" serviceType="DODS"
base="http://nomads.ncdc.noaa.gov:9091/dods/"; />
===============================================

The Blended Ocean winds almost work,
The OpENDAP interface comes up and allows the subsetting functions, but
whenever I submit a request for anything that spans time steps it
returns an error.

The other two, I may not even have in the same ballpark.
I thought it would be a good idea to just try a single day of NARR in an
aggregation with a scan element, but even that ~ so far ~ gives nothing
but a blank page for OpENDAP service.

Roy and Jerome :
I'd like to take up you offer to see your NcML configuration.
So far I've had no luck with this on my own.

Need some pointers on what I need in the configuration.
Thanks in advance for any assistance.

-Dan



Roy Mendelssohn wrote the following on 6/15/2006 10:44 AM:
> TDS in fact allows for just what you are asking ('If it is possible to
> create hooks that allow TDS to read and import data/metadata from a
> source OPeNDAP URL') and we have a test of this implemented  (go to
> http://oceanwatch.pfeg.noaa.gov:8081/thredds/catalog.html and look at 
> 'Test remote AUV data TDS'.  These data are served by OPeNDAP at MBARI,
> but we both aggregate and serve them through TDS at our sie.
> 
> If you want info on how the THREDDS catalogs and NcML files are set up
> to do this, email Jeorme King at address@hidden.  I would
> apprecaiae being cc:'ed ; also.
> 
> Hope this helps.
> 
> -Roy
> 
> 
> At 10:07 AM -0400 6/15/06, dan.swank wrote:
> 
>> We have had the NARR on GDS for quite some time :
>> http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY
>> GDS works very well with GRIB data in general.
>> Our particular goal is to get it on TDS due to its suporior metadata
>> handling and WCS service.
> 
>> If it is possible to create hooks that allow TDS to read and import
> 
>> data/metadata from a source OPeNDAP URL, similar to what the Live Access
>> Server does... that would be outstanding.  May pose an easier task to
>> implement as well.
> 
>> -Dan
> 
>>
>>
>> Rob Cermak wrote the following on 6/14/2006 6:50 PM:
>> > Dan,
>> >
>> > The Grads GDS OPeNDAP server is suppose to work with GRIB files.  The
> 
>> > large unknown is if TDS can then aggregate the data across the Grads GDS
>> > server.  To help answer that is to see if TDS can aggregate across maybe
>> > one of the NOMADs GFS servers?  I'm pretty sure they are using the Grads
>> > GDS server.
>> >
>> > http://www.iges.org/grads/gds/
>> >
>> > Rob
>> >
>> > On Tue, June 13, 2006 2:28 pm, dan.swank wrote:
>> >
>> >>Ethan:
>> >>Understood.  consider this a wish-list then :)
>> >>-Dan
>> >>
>> >>Ethan Davis wrote the following on 6/13/2006 6:22 PM:
>> >>
>> >>>Hi Dan,
>> >>>
>> >>>Evidently, aggregation of GRIB data may not be working. I thought since
>> >>>GRIB files are read by the netCDF-java library it would just work but I
>> >>>just heard that there may be some issues with aggregating GRIB
>> involving
>> >>>how the underlying code writes indexes into the GRIB files. We'll have
>> >>>to wait till John is back from vacation for a more detailed
>> >>>answer/explanation.
>> >>>
>> >>>Sorry for the mistake.
>> >>>
>> >>>There currently isn't a way to have a single scan element scan a set of
>> >>>directories.
>> >>>
>> >>>I believe the underlying netCDF-java code that reads GRIB files should
>> >>>result in fairly well-formed, CF (I think) compliant netCDF views
>> of the
>> >>>dataset. Not sure how this would be affected by aggregation.
>> >>>
>> >>>Ethan
>> >>>
>> >>>dan.swank wrote:
>> >>>
>> >>>
>> >>>>Ethan:
>> >>>>
>> >>>>The NARR is a reanalysis, so it don't have forecast times.  I would be
>> >>>>a
>> >>>>simple 03 hr chain (00 hr fct time) spanning 26 years.
>> >>>>
>> >>>>See an existing GDS subset aggregation:
>> >>>>http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_tmpprs.subset.info
>> >>>>
>> >>>>This will give a sense for the nature of the beast.
>> >>>>
>> >>>>The directory structure is set up as such:
>> >>>>http://nomads.ncdc.noaa.gov/data/narr/
>> >>>>
>> >>>>
>> >>>>Heres the TDS aggregation I set up while experimenting yesterday, on a
>> >>>>non-related dataset:
>> >>>>
>> >>>>  <dataset name="OceanWinds Test Daily Aggregation"
>> >>>>      ID="test/dailyagg" urlPath="test/agg">
>> >>>>    <serviceName>allTest</serviceName>
>> >>>>    <netcdf
>> >>>>xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>> >>>>     <aggregation dimName="time" type="joinNew">
>> >>>>       <variableAgg name="wind" />
>> >>>>       <scan dateFormatMark="#yyyyMMdd"
>> >>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
>> >>>>suffix=".nc" />
>> >>>>       <scan dateFormatMark="#yyyyMMdd"
>> >>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
>> >>>>suffix=".nc" />
>> >>>>       <scan dateFormatMark="#yyyyMMdd"
>> >>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
>> >>>>suffix=".nc" />
>> >>>>     </aggregation>
>> >>>>     <variable name="time" orgName="time">
>> >>>>       <attribute name="long_name" value="Days"/>
> 
>> >>>>       <attribute name="units" value="days since 1987-07-09" />
>> >>>>     </variable>
>> >>>>    </netcdf>
>> >>>>  </dataset>
>> >>>>
>> >>>>
>> >>>>Would this automatically detect the source of data were GRIB rather
>> >>>>than
>> >>>>NetCDF?  and it seems like you need to set the <scan> on each
>> >>>>individual
>> >>>>directory... Doing so the way NARR is set up would create one chunky
>> >>>>configuration file.  Is there anyway to have this scan a pattern
>> >>>>(YYYYMM/YYYYMMDD) of directories?
>> >>>>
>> >>>>I understand GRIB requires a certain amount of "supplemented" metadata
>> >>>>for complience.  Where do you enter this?
>> >>>>
>> >>>>-Dan
>> >>>>
>> >>>>
>> >>>>Ethan Davis wrote the following on 6/13/2006 1:21 PM:
>> >>>>
>> >>>>
>> >>>>
>> >>>>>Hi Dan,
>> >>>>>
>> >>>>>Aggregation should work the same for GRIB as for netCDF files. The
>> >>>>>issue
>> >>>>>would be how your GRIB files are structured and how you want to
>> >>>>>aggregate them. Our GRIB files each contain one full model run (all
>> >>>>>parameters, all forecast times). We haven't tried aggregating beyond
>> >>>>>that.
>> >>>>>
>> >>>>>We have started tracking what is available for the NCEP models on our
>> >>>>>server. This is from the TDS 3.8 announcement (with links updated):
>> >>>>>
>> >>>>>   We also are now tracking detailed inventory of NCEP model output,
>> >>>>>eg:
>> >>>>>
>> >>>>>http://motherlode.ucar.edu:8080/thredds/modelInventory/model/NCEP/NAM/CONUS_12km/
>> >>>>>
>> >>>>>
>> >>>>>   These are all linked from the "collection dataset" pages; For
>> >>>>>   example from
>> >>>>>
>> >>>>>http://motherlode.ucar.edu:8080/thredds/catalog/model/NCEP/NAM/CONUS_12km/catalog.html
>> >>>>>
>> >>>>>
>> >>>>>   choose the top "CONUS_12_km" link, then choose "Available
>> >>>>>Inventory"
>> >>>>>   Documentation.
>> >>>>>
>> >>>>>One idea for this work is to eventually provide access to alternate
>> >>>>>datasets, for instance, a dataset that contains all the 3hr forecast
>> >>>>>times from the different runs, or one that contained all the 12Z
>> valid
>> >>>>>times from the different runs. Tracking these detailed inventories is
>> >>>>>just the first step but aggregation and alternate groupings of the
>> >>>>>data
>> >>>>>is pretty interesting to think about.
>> >>>>>
>> >>>>>How are your GRIB files structured and what kind of aggregation where
>> >>>>>you thinking about?
>> >>>>>
>> >>>>>Ethan
>> >>>>>
>> >>>>>dan.swank wrote:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>>Hello,
>> >>>>>>
>> >>>>>>I've been tinkering with the TDS aggregation capabilities and they
>> >>>>>>work
>> >>>>>>quite well for NetCDF data, however, I can't seem to find
>> anything in
>> >>>>>>the docs regarding aggregating GRIB.
>> >>>>>>We want to get The NARR dataset which we have here at NCDC-NOMADS on
>> >>>>>>the
>> >>>>>>TDS.  It consists of hundreds of thousands of 50 Mb + GRIB files
>> in a
>> >>>>>>YYYYMM/YYYYMMDD tree.
>> >>>>>>  Just scouting for a quick answer here:
>> >>>>>>Is aggregating the NARR GRIB currently feasable with the current
>> >>>>>>release
>> >>>>>>of TDS?  If so, do any docs exist which could give me a starting
>> >>>>>>point?
>> >>>>>>Converting it to NetCDF will not be possible (volume).
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>--
>> >>Dan Swank <address@hidden>
>> >>NOMADS Project:  Software & Data Management
>> >>Contractor - STG, Incorporated
>> >>Veach-Baley Federal Building
>> >>151 Patton Avenue
>> >>Asheville, NC 28801-5001
>> >>Phone: 828-271-4007
>> >>
>> >>===============================================================================
>> >>To unsubscribe thredds, visit:
>> >>http://www.unidata.ucar.edu/mailing-list-delete-form.html
>> >>===============================================================================
>> >>
>> >
>> >
>> >
>>
>> --
>> Dan Swank <address@hidden>
>> NOMADS Project:  Software & Data Management
>> Contractor - STG, Incorporated
>> Veach-Baley Federal Building
>> 151 Patton Avenue
>> Asheville, NC 28801-5001
>> Phone: 828-271-4007
>>
>> ===============================================================================
>> To unsubscribe thredds, visit:
>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>> ===============================================================================
> 
> 
> 
> -- 
> 
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> 1352 Lighthouse Avenue
> Pacific Grove, CA 93950-2097
> 
> e-mail: address@hidden (Note new e-mail address)
> voice: (831)-648-9029
> fax: (831)-648-8440
> www: http://www.pfeg.noaa.gov/
> 
> "Old age and treachery will overcome youth and skill."

-- 
Dan Swank <address@hidden>
NOMADS Project:  Software & Data Management
Contractor - STG, Incorporated
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Phone: 828-271-4007

===============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
===============================================================================