[THREDDS #AOP-269354]: Issue with loading GFS data set

Greetings Andy,

Sorry for the delay. This should now be fixed in the latest TDS and TDM 5.0
snapshots, which are both available via docker, and direct download at:

https://artifacts.unidata.ucar.edu/repository/unidata-snapshots/edu/ucar/tds/5.0.0-SNAPSHOT/tds-5.0.0-20201121.064829-36.war
https://artifacts.unidata.ucar.edu/repository/unidata-snapshots/edu/ucar/tdmFat/5.0.0-SNAPSHOT/tdmFat-5.0.0-20201121.064829-33.jar

An example can be seen at:

https://thredds-test.unidata.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_0p25deg/catalog.html?dataset=grib/NCEP/GFS/Global_0p25deg/TwoD

Cheers,

Sean

address@hidden> wrote:

> New Client Reply: Issue with loading GFS data set
>
> Hi Sean,
>
> I can see that the issue
> https://github.com/Unidata/tds/issues/54
> is still open in GitHub and was wondering if you had a chance to look and
> potentially fix the issue.
> I'm still interested in using THREDDS together with ERDDAP.
>
> Kind regards,
> Andy
>
>
>
> [cid:a54dc638-432e-4e47-9c0f-a4d102bdd1e9]
> Andy Ziegler
> Data Strategy and Governance Manager
> Information Services
> MetService – Te Ratonga Tirorangi
>
> d +64 4 4700 765    t +64 4 4700 700    m +64 21 025 37630
> 30 Salamanca Road, Kelburn, Wellington 6012
> PO Box 722, Wellington 6140, New Zealand
>
> [cid:083982a5-67f3-48d6-8b23-5302954a9621]
>
> metservice.com<http://www.metservice.com/about/about>  .  metraweather.com
> <http://www.metraweather.com/>  .  metocean.co.nz<
> http://www.metocean.co.nz/>
>
> MetraWeather (Australia) Pty Ltd [ACN 126 850 904], MetraWeather (UK) Ltd
> [No. 04833498] and MetraWeather (Thailand) Ltd [No. 0105558115059] are
> wholly-owned subsidiaries of Meteorological Service of New Zealand Ltd
> (MetService)
>
>
> ________________________________
> From: Andy Ziegler <address@hidden>
> Sent: Friday, May 1, 2020 9:01 AM
> To: address@hidden <address@hidden>
> Subject: Re: [THREDDS #AOP-269354]: Issue with loading GFS data set
>
> Thanks Sean, that is fantastic.
> Kind regards,
> Andy
>
>
>
>
>
>
> [cid:532304c0-9189-4200-98b2-d3baf71e51f1]
>
> Andy Ziegler
>
> Strategic Data Manager – Information Services
>
> MetService – Te Ratonga Tirorangi
>
>
>
> d +64 4 4700 765    t +64 4 4700 700    m +64 21 025 37630
>
> 30 Salamanca Road, Kelburn, Wellington 6012
>
> PO Box 722, Wellington 6140, New Zealand
>
>
>
> [cid:6dcebcd6-d859-4d64-8e9e-5a9dd3f47d57]
>
>
>
> metservice.com<http://www.metservice.com/about/about>  .  metraweather.com
> <http://www.metraweather.com/>  .  metocean.co.nz<
> http://www.metocean.co.nz/>
>
>
>
> MetraWeather (Australia) Pty Ltd [ACN 126 850 904], MetraWeather (UK) Ltd
> [No. 04833498] and MetraWeather (Thailand) Ltd [No. 0105558115059] are
> wholly-owned subsidiaries of Meteorological Service of New Zealand Ltd
> (MetService)
>
>
> ________________________________
> From: Unidata THREDDS Support <address@hidden>
> Sent: Friday, May 1, 2020 4:33 AM
> To: Andy Ziegler <address@hidden>
> Cc: address@hidden <address@hidden>;
> Andy Ziegler <address@hidden>
> Subject: [THREDDS #AOP-269354]: Issue with loading GFS data set
>
> Greetings Andy,
>
> Sorry for the delay. I will be taking a look at this issue today and
> tomorrow.
>
> Sean
>
> > Hi Sean,
> >
> > Hope you are well and safe in these times.
> >
> > I was wondering what a timeline for fixing bug
> https://github.com/Unidata/tds/issues/54 might look like. We are getting
> closer to establishing an ERDDAP prototype and whether we can use THREDDS
> to serve the NWP data to it or not will need a lasting architectural
> decision soon. Is there a plan to look into this?
> >
> > Kind regards,
> > Andy
> >
> >
> >
> > [cid:cdf50c11-0623-4ef0-b122-8c9867d9face]
> > Andy Ziegler
> > Strategic Data Manager – Information Services
> > MetService – Te Ratonga Tirorangi
> >
> > d +64 4 4700 765    t +64 4 4700 700    m +64 21 025 37630
> > 30 Salamanca Road, Kelburn, Wellington 6012
> > PO Box 722, Wellington 6140, New Zealand
> >
> > [cid:18b9f3a8-9b67-4c79-88b8-f5d02ca7f4a3]
> >
> > metservice.com<http://www.metservice.com/about/about>  .
> metraweather.com<http://www.metraweather.com/>  .  metocean.co.nz<
> http://www.metocean.co.nz/>
> >
> > MetraWeather (Australia) Pty Ltd [ACN 126 850 904], MetraWeather (UK)
> Ltd [No. 04833498] and MetraWeather (Thailand) Ltd [No. 0105558115059] are
> wholly-owned subsidiaries of Meteorological Service of New Zealand Ltd
> (MetService)
> >
> >
> > ________________________________
> > From: Andy Ziegler
> > Sent: Tuesday, November 26, 2019 3:19 PM
> > To: address@hidden <address@hidden>
> > Subject: RE: [THREDDS #AOP-269354]: Issue with loading GFS data set
> >
> > Hi Sean,
> >
> > Thank you very much for this update. Very helpful background information
> and thank you for putting the effort in clarifying where the "issues" lie .
> Much appreciated.
> >
> > I'll keep an eye on the GitHub issue but it would be great if you ping
> me once its fixed and a new release is out for me to test.
> > I'm keen on setting up a prototype of ERDDAP and THREDDS for New Zealand
> MetService and demonstrate how this couple could delivery on our needs for
> a catalogue of the variety of data sources we use as well as work with
> model forecast files as well which ERDDAP cannot do natively in an easy
> way. This issue currently stops me from connecting them together.
> >
> > Thank you for your help.
> >
> > Kind regards,
> > Andy
> >
> >
> > -----Original Message-----
> > From: Unidata THREDDS Support <address@hidden>
> > Sent: Tuesday, 26 November 2019 8:59 AM
> > To: Andy Ziegler <address@hidden>
> > Cc: address@hidden
> > Subject: [THREDDS #AOP-269354]: Issue with loading GFS data set
> >
> > Greetings Andy,
> >
> > There are a few issues raised in Bob's email, but they are unrelated.
> The first issue raised is a question about the compliance of the TwoD
> dataset to the CF Metadata Conventions, and the second is related to the
> use of a multidimensional variable in a DAP2 Map for a Grid. These two are
> totally unrelated, as DAP2 does not depend on the CF Metadata Conventions
> at all.
> >
> > I've opened a github issue with regards to the CF question:
> >
> > https://github.com/Unidata/netcdf-java/issues/152
> >
> > Strictly speaking, there isn't a CF compliance issue, as the dataset
> technically conforms to the CF Conventions. As described in the Overview of
> the the CF documentation (
> http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#_overview
> ),
> >
> > "The use of coordinate variables is required for all dimensions that
> correspond to one dimensional space or time coordinates. In cases where
> coordinate variables are not applicable, the variables containing
> coordinate data are identified by the coordinates attribute."
> >
> > Since the "time" coordinate in these variables cannot be one dimensional
> (that is, the valid time of a unique forecast is dependent the the unique
> runtime of a given model), the second method for identifying the coordinate
> variable is needed (i.e. inspecting the coordinates attribute).  However,
> it is "recommended" that a multidimensional coordinate variable not share
> the same name as a dimension, but not strictly "required" (
> http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#coordinate-system).
> As the github issue above describes, we'll change our GRIB collection code
> such that we follow the recommended practice in order to support clients
> that do have all of the nuances of the specification covered.
> >
> > I think the real issue here is that the dap2 server in the TDS is
> pointing to a multidimensional variable in the Maps list of all of the
> Grids in the TwoD dataset, and that is not supported by the dap2 spec (dap4
> spec, yes, but not dap2), which was pointed out in Bob's email (although
> not related to CF in any way). It's likely that we'll need a code change in
> the dap2 server used by the TDS to handle the case where individual Maps
> for a Grids cannot be created based on name matching alone due to the
> violation of the 1D constraint placed on a map (basically, it's a data
> model translation issue - netCDF-Java's Common Data Model to the DAP2 Data
> Model). This issue is outlined on github here:
> >
> > https://github.com/Unidata/tds/issues/54
> >
> > That's where things stand at this point. I can ping you once a fix is in
> place, or you can follow along via github (the one to watch would be TDS
> issue 54).
> >
> > Cheers,
> >
> > Sean
> >
> > > Hi Ryan,
> > >
> > >
> > >
> > > Bob provided the response below. The main issue sees to be the
> declaration of the second dimension as a 2D field although being one of two
> dimensions, so should be only 1D.
> > >
> > > Hope this helps.
> > >
> > >
> > >
> > > Kind regards,
> > >
> > > Andy
> > >
> > >
> > >
> > > The response from THREDDS that I think is incorrect can be seen in the
> URL that you gave me:
> > >
> > >
> https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD.dds
> > >
> > > The part that I think is wrong is wrong for every multidimensional
> variable shown there, e.g.,
> > >
> > > Grid {
> > >
> > > ARRAY:
> > >
> > > Float32 Total_ozone_entire_atmosphere_single_layer[reftime = 14][time
> = 93][lat = 721][lon = 1440];
> > >
> > > MAPS:
> > >
> > > Float64 reftime[reftime = 14];
> > >
> > > Float64 time[reftime = 14][time = 93];
> > >
> > > Float32 lat[lat = 721];
> > >
> > > Float32 lon[lon = 1440];
> > >
> > > } Total_ozone_entire_atmosphere_single_layer;
> > >
> > >
> > >
> > > Specifically, I think
> > >
> > > Float64 time[reftime = 14][time = 93];
> > >
> > > should be
> > >
> > > Float64 time[time = 93];
> > >
> > >
> > >
> > > Multidimensional variables in CF-compliant datasets are defined by a
> set of 1D dimensions. Those dimensions often/usually have an associated 1D
> coordinate variable with the same name (e.g., a 1D time variable that just
> uses the 1D time dimension) -- or they can simply be dimensions without any
> associated coordinate variable. Section 5 of the CF standard (
> http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#coordinate-system)
> says
> > >
> > >
> > >
> > > Any of a variable’s dimensions that is an independently varying
> latitude, longitude, vertical, or time dimension (see Section 1.2,
> "Terminology") and that has a size greater than one must have a
> corresponding coordinate variable, i.e., a one-dimensional variable with
> the same name as the dimension (see examples in Chapter 4, Coordinate
> Types). This is the only method of associating dimensions with coordinates
> that is supported by [COARDS].
> > >
> > >
> > >
> > > This use of the same name for the 1D dimension and for the 1D variable
> that uses only that 1D dimension is COARDS and CF's way of associating the
> dimension (which says "there are n values") and the actual values in the
> coordinate variable (which says "here are the n values").
> > >
> > >
> > >
> > > I admit that the CF specification could be clearer about this. Perhaps
> this use of "time" as the name of a 1D dimension and a 2D coordinate
> variable is legal. (It's never good when the exact meaning of a
> specification is open to interpretation.) But even if that is the case, it
> is unnecessary and confusing. It would be better if this dataset had 2
> dimensions (reftime1 and time), one 1D coordinate variable
> (reftime1[reftime1]) and a separate 2D variable with a different name which
> has the time value associated with each combination of [reftime1] and
> [time] (e.g., comboTime[reftime1][time]).
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Unidata THREDDS Support <address@hidden>
> > > Sent: Wednesday, 13 November 2019 8:55 AM
> > > To: Andy Ziegler <address@hidden>
> > > Cc: address@hidden
> > > Subject: [THREDDS #AOP-269354]: Issue with loading GFS data set
> > >
> > >
> > >
> > > Hi,
> > >
> > >
> > >
> > > How are you pointing ERDDAP at thredds.ucar.edu? As far as I
> understand ERDDAP is server software, so that means you've configured a
> server somewhere to point to that opendap endpoint on thredds.ucar.edu?
> > >
> > >
> > >
> > > We know Bob, and can reach out as necessary, but I'm still not quite
> sure what the issue is or where it's coming from. Even doing:
> > >
> > >
> > >
> > > wget
> https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD.dds
> > >
> > >
> > >
> > > works fine for me.
> > >
> > >
> > >
> > > Ryan
> > >
> > >
> > >
> > > > Hi Ryan,
> > >
> > > > Thanks for looking into this.
> > >
> > > > Yes, I just tried it again to discover this data set with ERDDAP and
> get
> > >
> > > > *** EDDGridFromDap.generateDatasetsXml
> > >
> > > > tLocalSourceUrl=
> https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD
> > >
> > > > dimNames=null reloadEveryNMinutes=-1
> > >
> > > > externalAddGlobalAttributes=null
> > >
> > > > getDAS try#0 (timeout is 5 minutes)
> > >
> > > > getDDS try#0 (timeout is 5 minutes)
> > >
> > > > Error while getting DDS from
> https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD.dds
> .
> > >
> > > > In the dataset descriptor object:
> > >
> > > > `Total_ozone_entire_atmosphere_single_layer' is not a valid
> declaration. (dods.dap.BadSemanticsException: In grid
> 'Total_ozone_entire_atmosphere_single_layer The size of dimension 1 in the
> array component 'Total_ozone_entire_atmosphere_single_layeris not equal to
> the size of the coresponding map vector 'time2.)
> > >
> > > >
> > >
> > > > Not sure what ERDDAP uses to request the structure information of
> the dataset but the error response seems to come from the TDS since Bob
> Simons found the error message in the TDS code I believe (including the
> typo).
> > >
> > > >
> > >
> > > > Do you know Bob and can get in touch with him if required or would
> you like me to make the introduction?
> > >
> > >
> > >
> > >
> > >
> > > Ticket Details
> > >
> > > ===================
> > >
> > > Ticket ID: AOP-269354
> > >
> > > Department: Support THREDDS
> > >
> > > Priority: Normal
> > >
> > > Status: Closed
> > >
> > > ===================
> > >
> > > NOTE: All email exchanges with Unidata User Support are recorded in
> the Unidata inquiry tracking system and then made publicly available
> through the web.  If you do not want to have your interactions made
> available in this way, you must let us know in each email you send to us.
> > >
> > >
> > >
> > >
> > >
> > >
> > > This email and any attachments may contain confidential information.
> If you are not the intended recipient, your use or communication of the
> information is strictly prohibited. If you have received this message in
> error please notify MetService immediately.
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: AOP-269354
> > Department: Support THREDDS
> > Priority: High
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publicly available through
> the web.  If you do not want to have your interactions made available in
> this way, you must let us know in each email you send to us.
> >
> >
> >
> >
>
>
> Ticket Details
> ===================
> Ticket ID: AOP-269354
> Department: Support THREDDS
> Priority: Critical
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publicly available through
> the web.  If you do not want to have your interactions made available in
> this way, you must let us know in each email you send to us.
>
>
>
>
>
> Ticket Details
> ===================
> Ticket ID: AOP-269354
> Department: Support THREDDS
> Priority: Critical
> Status: Open
> Link:
> https://andy.unidata.ucar.edu/staff/index.php?_m=tickets&_a=viewticket&ticketid=30851



Ticket Details
===================
Ticket ID: AOP-269354
Department: Support THREDDS
Priority: Critical
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.