Re: GRIB Aggregation

Ethan:
Understood.  consider this a wish-list then :)
-Dan

Ethan Davis wrote the following on 6/13/2006 6:22 PM:
> Hi Dan,
> 
> Evidently, aggregation of GRIB data may not be working. I thought since
> GRIB files are read by the netCDF-java library it would just work but I
> just heard that there may be some issues with aggregating GRIB involving
> how the underlying code writes indexes into the GRIB files. We'll have
> to wait till John is back from vacation for a more detailed
> answer/explanation.
> 
> Sorry for the mistake.
> 
> There currently isn't a way to have a single scan element scan a set of
> directories.
> 
> I believe the underlying netCDF-java code that reads GRIB files should
> result in fairly well-formed, CF (I think) compliant netCDF views of the
> dataset. Not sure how this would be affected by aggregation.
> 
> Ethan
> 
> dan.swank wrote:
> 
>> Ethan:
>>
>> The NARR is a reanalysis, so it don't have forecast times.  I would be a
>> simple 03 hr chain (00 hr fct time) spanning 26 years.
>>
>> See an existing GDS subset aggregation:
>> http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_tmpprs.subset.info
>>
>> This will give a sense for the nature of the beast.
>>
>> The directory structure is set up as such:
>> http://nomads.ncdc.noaa.gov/data/narr/
>>
>>
>> Heres the TDS aggregation I set up while experimenting yesterday, on a
>> non-related dataset:
>>
>>   <dataset name="OceanWinds Test Daily Aggregation"
>>       ID="test/dailyagg" urlPath="test/agg">
>>     <serviceName>allTest</serviceName>
>>     <netcdf
>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>      <aggregation dimName="time" type="joinNew">
>>        <variableAgg name="wind" />
>>        <scan dateFormatMark="#yyyyMMdd"
>> location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
>> suffix=".nc" />
>>        <scan dateFormatMark="#yyyyMMdd"
>> location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
>> suffix=".nc" />
>>        <scan dateFormatMark="#yyyyMMdd"
>> location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
>> suffix=".nc" />
>>      </aggregation>
>>      <variable name="time" orgName="time">
>>        <attribute name="long_name" value="Days"/>
>>        <attribute name="units" value="days since 1987-07-09" />
>>      </variable>
>>     </netcdf>
>>   </dataset>
>>
>>
>> Would this automatically detect the source of data were GRIB rather than
>> NetCDF?  and it seems like you need to set the <scan> on each individual
>> directory... Doing so the way NARR is set up would create one chunky
>> configuration file.  Is there anyway to have this scan a pattern
>> (YYYYMM/YYYYMMDD) of directories?
>>
>> I understand GRIB requires a certain amount of "supplemented" metadata
>> for complience.  Where do you enter this?
>>
>> -Dan
>>
>>
>> Ethan Davis wrote the following on 6/13/2006 1:21 PM:
>>  
>>
>>> Hi Dan,
>>>
>>> Aggregation should work the same for GRIB as for netCDF files. The issue
>>> would be how your GRIB files are structured and how you want to
>>> aggregate them. Our GRIB files each contain one full model run (all
>>> parameters, all forecast times). We haven't tried aggregating beyond
>>> that.
>>>
>>> We have started tracking what is available for the NCEP models on our
>>> server. This is from the TDS 3.8 announcement (with links updated):
>>>
>>>    We also are now tracking detailed inventory of NCEP model output, eg:
>>>  
>>> http://motherlode.ucar.edu:8080/thredds/modelInventory/model/NCEP/NAM/CONUS_12km/
>>>
>>>
>>>    These are all linked from the "collection dataset" pages; For
>>>    example from
>>>  
>>> http://motherlode.ucar.edu:8080/thredds/catalog/model/NCEP/NAM/CONUS_12km/catalog.html
>>>
>>>
>>>    choose the top "CONUS_12_km" link, then choose "Available Inventory"
>>>    Documentation.
>>>
>>> One idea for this work is to eventually provide access to alternate
>>> datasets, for instance, a dataset that contains all the 3hr forecast
>>> times from the different runs, or one that contained all the 12Z valid
>>> times from the different runs. Tracking these detailed inventories is
>>> just the first step but aggregation and alternate groupings of the data
>>> is pretty interesting to think about.
>>>
>>> How are your GRIB files structured and what kind of aggregation where
>>> you thinking about?
>>>
>>> Ethan
>>>
>>> dan.swank wrote:
>>>
>>>    
>>>
>>>> Hello,
>>>>
>>>> I've been tinkering with the TDS aggregation capabilities and they work
>>>> quite well for NetCDF data, however, I can't seem to find anything in
>>>> the docs regarding aggregating GRIB.
>>>> We want to get The NARR dataset which we have here at NCDC-NOMADS on
>>>> the
>>>> TDS.  It consists of hundreds of thousands of 50 Mb + GRIB files in a
>>>> YYYYMM/YYYYMMDD tree.
>>>>   Just scouting for a quick answer here:
>>>> Is aggregating the NARR GRIB currently feasable with the current
>>>> release
>>>> of TDS?  If so, do any docs exist which could give me a starting point?
>>>> Converting it to NetCDF will not be possible (volume).
>>>>
>>>>         
>>>
>>>     
>>
>>
>>   
> 
> 

-- 
Dan Swank <address@hidden>
NOMADS Project:  Software & Data Management
Contractor - STG, Incorporated
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Phone: 828-271-4007

===============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
===============================================================================