[netCDFJava #MRA-346602]: NetCDF file that used to work now failing

Great - thank you! I will be sure this fix gets included in our next release.

Cheers,

Sean

> Hi Sean,
> Glad you and Tom were able to work out a good plan for the main issues
> (e.g., geostationary projection and default for semi_minor_axis) raised
> in your emails below.
> 
> For the record, the answer is YES to the "Is it ok if I used the example
> you sent for that purpose" question you asked in the 25 July email
> below. :-)
> 
> Thanks,
> Barry
> 
> 
> 
> On 8/18/2016 4:16 PM, Unidata netCDF Java Support wrote:
> > Thanks for you messages Tom! They are very helpful. I'll add some checks 
> > and code to use the
> > inverse flattening if it is specified and the semi-minor-axes isn't. I'll 
> > also double check to make sure
> > that the Geostationary class can deal with radians, not just meters, and 
> > can deal with a scale/offset.
> >
> > Thanks again!
> >
> > Sean
> >
> > > Hi Sean,
> > >
> > >
> > > On 7/26/16 8:11 AM, McIDAS Help Desk wrote:
> > > > Hope I'm sending this to the right people... (Not sure if there's an
> > > > email list for the McV team that would have been better.)
> > > >
> > > > In any case, there are some issues raised and questions asked (which I
> > > > *bolded*) in the email below from Sean Arms that are not just for Bob,
> > > > but rather for the McV developers/team. We will need to gather our
> > > > thoughts and send him a reply.
> > > >
> > > > Barry
> > > >
> > > >
> > > > -------- Forwarded Message --------
> > > > Subject:        [netCDFJava #MRA-346602]: NetCDF file that used to work 
> > > > now
> > > > failing
> > > > Date:   Mon, 25 Jul 2016 17:37:19 -0600
> > > > From:   Unidata netCDF Java Support <address@hidden>
> > > > Reply-To:       address@hidden
> > > > To:     address@hidden
> > > > CC:     address@hidden
> > > >
> > > >
> > > > Greetings Bob,
> > > >
> > > > (comments in-line)
> > > >
> > > > > Hello -
> > > > >
> > > > > We have a NetCDF file that worked in McIDAS-V 1.5 which uses NetCDF
> > > > > version 4.3.22 and now fails in McIDAS-V 1.6 which uses NetCDF version
> > > > > 4.6.5.  The error when trying to load the file is:
> > > > > > There was an error loading the data:
> > > > > > Error creating data source:file.grid with:
> > > > > > C:\Users\rcarp\Data\UserFunctions\H8\HS_H08_20150415_0420_B14_FLDK.subset.nc
> > > > > > Grid data source failed making data set:
> > > > > > C:\Users\rcarp\Data\UserFunctions\H8\HS_H08_20150415_0420_B14_FLDK.subset.nc
> > > > > > Must specify semi_minor_axis
> > > > >
> > > > > We created a NcML file that fixes this by adding a semi_minor_axis
> > > > > attribute to the projection variable, but we wanted to get your 
> > > > > thoughts
> > > > > on this.
> > > > >
> > > > > * Do you think this problem came about because of more strict rules
> > > > > with the new NetCDF version?
> > > >
> > > > The problem came about when we merged a pull request from Tom Rink in 
> > > > 2014,
> > > > which made netCDF-Java's handle of Geostationary projections more 
> > > > general:
> > > >
> > > > https://github.com/Unidata/thredds/commit/0b0b49ef0527e2a1b4d9aef7e169bfdaa9db1980
> > > >
> > > > The Geostationary projection isn't defined in a standard that I know of
> > > > (for sure not CF). The original code in netCDF-Java was implemented by
> > > > John Caron following this document:
> > > >
> > > > http://www.cgms-info.org/documents/pdf_cgms_03.pdf
> > > >
> > > > and was originally obtained via eumetsat, but the link in the code
> > > > is now dead. Rather than a standard, this looks to be one way of doing
> > > > a geostationary projection, and was used for a particular mission.
> > >
> > > This was supposed to be a standard by now for this type of projection.
> > > It had been accepted
> > > for evaluation a few years ago, but I'm not sure what happened - I need
> > > to look through old
> > > emails. All of the GOES-R data stored in NetCDF will use this
> > > projection. MTG may also
> > > be stored in a similar manner, and the code is flexible enough to handle
> > > both.
> > >
> > > Two departures from CF, which my understanding were part of the
> > > specification now, are
> > > necessary for this projection type:
> > >
> > > projection coordinate variables can have units of 'radian', not just 
> > > 'meter'
> > > projection coordinate variables can have a scaleFactor and addOffset.
> > > I'm pretty sure
> > > John Caron took care of this in the CDM.
> > >
> > >
> > > > It looks like Tom's code generalized the code a bit, and there wasn't
> > > > a default set if semi_minor_axis was undefined (previously these were
> > > > hardcoded values).*Would it be reasonable to have a default value 
> > > > in****the case that it isn't defined? Maybe the same as 
> > > > semi_major_axis?*
> > > >
> > > > > * Assuming this is a NetCDF issue, is this something that you are
> > > > > concerned with (backward compatibility with files that worked with
> > > > > previous NetCDF versions)?  Or do you feel it is more on the data
> > > > > producer to keep their files up with NetCDF standards?
> > > >
> > > > As long as standards are being followed and versioned appropriately,
> > > > we do our best to make sure things keep working. If there is a
> > > > fundamental flaw in the way the something was being defined (for
> > > > example, is semi_minor_axis absolutely required to get the projection
> > > > correct?), then in an ideal world, data providers should update the
> > > > files to have the correct metadata to ensure all libraries have
> > > > access to the required parameters. In this case, maybe some default
> > > > values should have been set? Not sure, but we should have had a test
> > > > file in our test suite to detect this situation.*Is it ok if I used the 
> > > > example you sent for that purpose****(again, assuming the correct thing 
> > > > to do is set a default value)?*
> > > >
> > > > > * Just as a reference, the original NetCDF file works in IDV 5.0
> > > > > (NetCDF 4.3.22) and fails the same way McIDAS-V currently does in
> > > > > IDV 5.1 (NetCDF 4.5.5).
> > > >
> > > > Yes, 4.3.22 used the old code, which was before Tom's additions, and so
> > > > set values of major and minor axes were used. Were they correct? Maybe 
> > > > not.
> > > >
> > > > *Please let us know what you guys think in terms of whether or not 
> > > > we****should have a default value for the projection parameters in 
> > > > the****Geostationary projection.*
> > > >
> > > > Cheers,
> > > > Sean
> > > >
> > > >
> > > > >
> > > > > I posted the following files to our anonymous ftp location:
> > > > >
> > > > > * Geographically subsetted NetCDF file that is currently failing:
> > > > >ftp://ftp.ssec.wisc.edu/pub/incoming/HS_H08_20150415_0420_B14_FLDK.subset.nc
> > > > > * NcML file that adds a semi_minor_axis attribute to the projection
> > > > > variable, which works:
> > > > >ftp://ftp.ssec.wisc.edu/pub/incoming/HS_H08_20150415_0420_B14_FLDK.ncml
> > > > >
> > > > > These files should remain in there for a week before they are removed,
> > > > > but if you need access to them after they are removed let me know and 
> > > > > I
> > > > > can re-post them.
> > > > >
> > > > > Thanks -
> > > > > Bob Carp
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: MRA-346602
> > > > Department: Support netCDF Java
> > > > Priority: Normal
> > > > Status: Open
> > > > ===================
> > > > NOTE: All email exchanges with Unidata User Support are recorded in the
> > > > Unidata inquiry tracking system and then made publicly available through
> > > > the web.  If you do not want to have your interactions made available in
> > > > this way, you must let us know in each email you send to us.
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: MRA-346602
> > Department: Support netCDF Java
> > Priority: Critical
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> 
> 


Ticket Details
===================
Ticket ID: MRA-346602
Department: Support netCDF Java
Priority: Critical
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.