[THREDDS #GJF-200148]: TDS 4.3 GRIB names different from TDS 4.2

Hi Fan:

We have a bug when a datasetScan has NcML that has an iospParam. We are working 
on a fix for version 4.4.
At the moment, the only workaround i can think of is to add the table into our 
standard tables, see

  
http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/formats/GribTables.html

thanks for reporting the problem,
John


> Hi Lansing,
> 
> Thanks and we just installed the 4.3.19 version from your link.  Seems my 
> problem remains - the variables in the grib table are not showing up with 
> their correct names.  Wonder if you checked the fix.  The data and the grib 
> table are still at my ftp site:
> 
> ftp://hydro1.sci.gsfc.nasa.gov/private/ffang/GLDAS_CLM10SUBP_3H.A1979002.0000.001.2008086151108.grb
> ftp://hydro1.sci.gsfc.nasa.gov/private/ffang/gribtab_clm.tab
> 
> and my TDS config has these lines:
> 
> <datasetScan name="GLDAS_CLM10SUBP_3H" ID="GLDAS_CLM10SUBP_3H" 
> path="GLDAS_CLM10SUBP_3H" 
> location="/ftp/data/s4pa_TS2/TEST_OPENDAP/GLDAS_CLM10SUBP_3H/">
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; 
> iospParam="gribParameterTable=/home/ffang/gribtab_clm.tab">
> </netcdf>
> <metadata inherited="true">
> <serviceName>odapncss</serviceName>
> <dataType>Grid</dataType>
> </metadata>
> <filter>
> <include wildcard="*.grb"/>
> </filter>
> </datasetScan>
> 
> It is possible we did something wrong.  If it works for you, could you send 
> us the configuration lines?
> 
> Thanks.
> 
> -Fan
> 
> ________________________________________
> From: Unidata THREDDS Support [address@hidden]
> Sent: Tuesday, September 03, 2013 2:14 PM
> To: Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> Cc: address@hidden; Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]; 
> address@hidden; Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> Subject: [THREDDS #GJF-200148]: TDS 4.3 GRIB names different from TDS 4.2
> 
> Fan,
> 
> You can download a 4.3.19 snapshot here:
> 
> https://docs.google.com/file/d/0B7jP7ooGP2ZzdFh5NUprdEowMkE/edit?usp=sharing
> 
> Enjoy!
> 
> -Lansing
> 
> On 9/3/2013 11:24 AM, Fan Fang wrote:
> > New Client Reply: TDS 4.3 GRIB names different from TDS 4.2
> >
> > Hi Lansing,
> >
> > I notice that the bug is fixed in version 4.3.18.  Is it downloadable
> > from somewhere?
> >
> > Thanks.
> >
> > -Fan
> >
> > On 05/23/2013 04:37 PM, Unidata THREDDS Support wrote:
> >> Fan,
> >>
> >> I reproduced the behavior locally, and it is a bug.  Thanks for finding 
> >> and reporting it!  I've set up bug-ticket in jira, our issue tracking 
> >> system, which is here:
> >>
> >> https://bugtracking.unidata.ucar.edu/browse/TDS-435
> >>
> >> The developer who really knows grib is out for a long weekend, but I'll 
> >> make sure he sees it next week.
> >>
> >> -Lansing
> >>
> >>> Hi Lansing,
> >>>
> >>> You can get as many as you want from here:
> >>>
> >>> ftp://hydro1.sci.gsfc.nasa.gov/data/s4pa/GLDAS_V1/GLDAS_CLM10SUBP_3H/1979/002/
> >>>
> >>> My sample was the first file (at top) of that day.  The rest .grb files 
> >>> represent the hourly data for that day.  For more than one-day worth of 
> >>> data, go to 1979/003, 1979/004, etc. for year 1979.
> >>>
> >>> -Fan
> >>> ________________________________________
> >>> From: Unidata THREDDS Support [address@hidden]
> >>> Sent: Thursday, May 23, 2013 1:37 PM
> >>> To: Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> >>> Cc: address@hidden; Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]; 
> >>> address@hidden; Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> >>> Subject: [THREDDS #GJF-200148]: TDS 4.3 GRIB names different from TDS 4.2
> >>>
> >>> Fan,
> >>>
> >>> Can you put a few more of the grb files up?
> >>>
> >>> -Lansing
> >>>
> >>>> Hello Lansing,
> >>>>
> >>>> Thanks for the tip.  You can get the sample data and grib table from:
> >>>>
> >>>> ftp://hydro1.sci.gsfc.nasa.gov/private/ffang/GLDAS_CLM10SUBP_3H.A1979002.0000.001.2008086151108.grb
> >>>> ftp://hydro1.sci.gsfc.nasa.gov/private/ffang/gribtab_clm.tab
> >>>>
> >>>> My TDS configuration for the dataset is:
> >>>>
> >>>> <datasetScan name="GLDAS_CLM10SUBP_3H" ID="GLDAS_CLM10SUBP_3H"
> >>>> path="GLDAS_CLM10SUBP_3H"
> >>>> location="/ftp/data/s4pa_TS2/TEST_OPENDAP/GLDAS_CLM10SUBP_3H/">
> >>>> <netcdf
> >>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";
> >>>> iospParam="gribParameterTable=/home/ffang/gribtab_clm.tab">
> >>>> </netcdf>
> >>>> <metadata inherited="true">
> >>>> <serviceName>odapncss</serviceName>
> >>>> <dataType>Grid</dataType>
> >>>> </metadata>
> >>>> <filter>
> >>>> <include wildcard="*.grb"/>
> >>>> </filter>
> >>>> </datasetScan>
> >>>>
> >>>> -Fan
> >>>>
> >>>> On 05/22/2013 03:55 PM, Unidata THREDDS Support wrote:
> >>>>> Fan,
> >>>>>
> >>>>> Two things - first, don't serve the ncx files.  It doesn't matter where 
> >>>>> they are located (in the data directory or elsewhere), but it's not 
> >>>>> meant to be served as data.
> >>>>>
> >>>>> Second, it sounds like you are describing a bug in how user tables are 
> >>>>> applied.  I'd like to set up a tds locally in a debugger with your 
> >>>>> files, collection spec, and local grib table.  Please bundle them up 
> >>>>> and send them to me, or point me to where I can get them.
> >>>>>
> >>>>> Thanks,
> >>>>>    Lansing
> >>>>>
> >>>>>> Hello Lansing,
> >>>>>>
> >>>>>> Any comments/findings about my questions?
> >>>>>>
> >>>>>> With or without featureCollection we need to serve grib data in their 
> >>>>>> granule form, so how to make grib table work for individual granules 
> >>>>>> is remaining question number one.  So far our grib table works with 
> >>>>>> aggregated featureCollection dataset, but not for granules under 
> >>>>>> 'files' folder.  I also tried the 'iospParam' attribute of the 
> >>>>>> 'netcdf' element outside featureCollection, like
> >>>>>>
> >>>>>> <datasetScan name="GLDAS_CLM10SUBP_3H" ID="GLDAS_CLM10SUBP_3H" 
> >>>>>> path="GLDAS_CLM10SUBP_3H" 
> >>>>>> location="/ftp/data/s4pa_TS2/GLDAS_CLM10SUBP_3H/">
> >>>>>> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; 
> >>>>>> iospParam="gribParameterTable=/home/ffang/gribtab_clm.tab">
> >>>>>> </netcdf>
> >>>>>> ...
> >>>>>>
> >>>>>> and TDS didn't seem to read the grib table.
> >>>>>>
> >>>>>> The other main question is about the collection .ncx file usage.  I 
> >>>>>> found accessing that file very speedy in TDS and wonder if it should 
> >>>>>> be used to serve our extremely long time series datasets (~30 years of 
> >>>>>> hourly data), i.e. shall we let TDS make the file at the data location 
> >>>>>> instead of under cache/cdm, and treat it as the timeseries dataset?
> >>>>>>
> >>>>>> -Fan
> >>>>>>
> >>>>>> ________________________________________
> >>>>>> From: Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> >>>>>> Sent: Friday, May 17, 2013 2:50 PM
> >>>>>> To: Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]; address@hidden
> >>>>>> Cc: address@hidden
> >>>>>> Subject: RE: [THREDDS #GJF-200148]: TDS 4.3 GRIB names different from 
> >>>>>> TDS 4.2
> >>>>>>
> >>>>>> I take it back - the .tab extension apparently works.  We previously 
> >>>>>> made an error renaming the grib table.
> >>>>>>
> >>>>>> I also changed '/s' to '.s' but both came out the same.  For example, 
> >>>>>> kg/m^2/s or kg/m^2.s would show up as kgm^2s, which is wrong.  I 
> >>>>>> suppose we have to change it to kgm^-2s^-1 in the table?  I hope it is 
> >>>>>> consistent with software like 'wgrib'.
> >>>>>>
> >>>>>> It seems the variable names are essentially the long names and units 
> >>>>>> in grib table plus something like '_surface', with space chars 
> >>>>>> replaced by underscores.  I guess we can survive with that.
> >>>>>>
> >>>>>> These show up in the collection 'Best Timeseries' (is there a way to 
> >>>>>> re-configure this collection name?), but not for individual files 
> >>>>>> under 'files' folder in TDS (again, is there a way to configure for 
> >>>>>> the folder name?).  Is this expected?
> >>>>>>
> >>>>>> One final question: if I drop the collection index file .ncx at the 
> >>>>>> data location, TDS seems to be able to serve it like a dataset, and 
> >>>>>> for time series it is surprisingly fast to serve the .ncx file 
> >>>>>> instead.  Is this intended - in other words we shall direct user to 
> >>>>>> use .ncx, especially for long time series?  I know in default it's 
> >>>>>> under cache/cdm and not exposed to users.  What's the catch here?
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>> -Fan
> >>>>>> ________________________________________
> >>>>>> From: Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> >>>>>> Sent: Thursday, May 16, 2013 6:13 PM
> >>>>>> To: address@hidden for
> >>>>>> Cc: address@hidden
> >>>>>> Subject: RE: [THREDDS #GJF-200148]: TDS 4.3 GRIB names different from 
> >>>>>> TDS 4.2
> >>>>>>
> >>>>>> I can confirm changing the grib table name to have .tab as extension 
> >>>>>> still does not work.
> >>>>>>
> >>>>>> I'll try the modifications in the units, but wonder why it fails for 
> >>>>>> all since some of them have legit units, like 'K' for temperature.
> >>>>>>
> >>>>>> -Fan
> >>>>>> ________________________________________
> >>>>>> From: Unidata THREDDS Support [address@hidden]
> >>>>>> Sent: Thursday, May 16, 2013 2:37 PM
> >>>>>> To: Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]
> >>>>>> Cc: address@hidden; Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]; 
> >>>>>> address@hidden
> >>>>>> Subject: [THREDDS #GJF-200148]: TDS 4.3 GRIB names different from TDS 
> >>>>>> 4.2
> >>>>>>
> >>>>>> Fan,
> >>>>>>
> >>>>>> I did a little more digging, and I don't think the file name extension 
> >>>>>> will matter - that was my misinterpretation of how the table was 
> >>>>>> getting picked up.  However, you will probably have the best result 
> >>>>>> directly embedding the table information in your featureCollection 
> >>>>>> element using the xml notation.  It looks like you tried this already, 
> >>>>>> so I am not sure why it did not work for you.  It is possible that it 
> >>>>>> broke because you did not follow udunits standard -- for instance, 
> >>>>>> square meters is notated as m^2, rather than m(superscript)2.  The 
> >>>>>> code you are asking about is just the number that identifies the grib 
> >>>>>> parameter.
> >>>>>>
> >>>>>> Inside your <featureCollection> element:
> >>>>>>
> >>>>>> <gribConfig>
> >>>>>> <parameterMap>
> >>>>>> <parameter code="2">  <<-- This will override the standard table 
> >>>>>> values for parameter 2 (code < 128 are standard, > 128 are local)
> >>>>>> <description>Pressure reduced to MSL</description>
> >>>>>> <units>Pa</units>
> >>>>>> <name>PRMSL</name>
> >>>>>> </parameter>
> >>>>>> <parameter code="131">
> >>>>>> <description>Snowfall rate</description>
> >>>>>> <units>kg/m^2.s</units> <<--Compound units are notated with a dot, so 
> >>>>>> /m^2/s is just /m^2.s
> >>>>>> <name>Snof</name>
> >>>>>> </parameter>
> >>>>>> ...
> >>>>>>
> >>>>>> </parameterMap>
> >>>>>>
> >>>>>> </gribConfig>
> >>>>>> -Lansing Madry
> >>>>>> Unidata
> >>>>>> Boulder, Colorado
> >>>>>>
> >>>>>> Ticket Details
> >>>>>> ===================
> >>>>>> Ticket ID: GJF-200148
> >>>>>> Department: Support THREDDS
> >>>>>> Priority: Low
> >>>>>> Status: Closed
> >>>>>>
> >>>>>>
> >>>>> -Lansing Madry
> >>>>>    Unidata
> >>>>>    Boulder, Colorado
> >>>>>
> >>>>> Ticket Details
> >>>>> ===================
> >>>>> Ticket ID: GJF-200148
> >>>>> Department: Support THREDDS
> >>>>> Priority: Low
> >>>>> Status: Open
> >>>>>
> >>> -Lansing Madry
> >>> Unidata
> >>> Boulder, Colorado
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: GJF-200148
> >>> Department: Support THREDDS
> >>> Priority: Low
> >>> Status: Open
> >>>
> >>>
> >> -Lansing Madry
> >>    Unidata
> >>    Boulder, Colorado
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: GJF-200148
> >> Department: Support THREDDS
> >> Priority: Low
> >> Status: Open
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: GJF-200148
> > Department: Support THREDDS
> > Priority: Low
> > Status: Open
> > Link:  
> > https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=22083
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: GJF-200148
> Department: Support THREDDS
> Priority: Low
> Status: Open
> 
> 
> 

Ticket Details
===================
Ticket ID: GJF-200148
Department: Support THREDDS
Priority: Low
Status: Closed