[netCDF #LGJ-997921]: Large Classic File Questions

Thank you very much for the follow-up.  I'm a little curious what version of 
netcdf the Mathworks folk are using in the version of Matlab you are working 
from.  It was certainly the case in the past that the netCDF library did not 
strictly enforce limits that it perhaps should have.  This lead to the case 
where, if limits were now strictly enforced, files which had been written could 
no longer be read without complaint.  But that doesn't offer up much of a 
solution so much as explain a little bit how we have to consider how 'fixing' 
behavior can (and does) have unintended consequences for our community.  
Hindsight, and all that.  

Thanks again for following up, we appreciate the information!

Have a great day,

-Ward

> 
> I'm updating this inquiry to let you know that I think we understand
> what's happening.
> 
> After some additional testing, we now realize that the last dimension of
> the last variable in a file is treated implicitly as "unlimited" which
> allows that variable to have sizes that fundamentally exceed the format
> limits.  This is quite useful in some ways.  But it's awkward in others.
> 
> In particular, we tried to add a moderate sized variable to our 10.8 Gb
> netcdf file.  If we add the variable before FWF, the classic file is
> 
> still valid.  If we add the new variable after FWF, it is not. To be
> 
> honest, it's a little surprising that this behavior would be allowed. 
> I
> now understand why we got so confused and can imagine there are lots of
> cases that lead to confusion within the community.  We did not trust
> our
> large classic file, but also couldn't understand why tools were not
> complaining.  Another case that could cause confusion is if you have
> two
> files with a large variable in each, each file is valid but the merged
> file isn't. There is also the issue of nco trying to extract a single
> large variable from a cdf5 to a classic file.  This doesn't seem to work
> even though the input and output files are both valid.  I understand
> 
> this is probably an nco implementation issue and we may raise an issue
> with nco, but that only adds to a sense of mistrust and confusion in the
> large classic files.
> 
> Anyway, I think you can close this inquiry.  Feel free to let us know if
> there are any additional details that would be useful for us to understand.
> 
> thanks,
> 
> tony......
> 
> 
> On 5/7/21 12:39 PM, Unidata netCDF Support wrote:
> > Anthony.P.Craig,
> >
> > Thank you for contacting us. This is an automated response confirming the 
> > receipt of your ticket. One of our agents will get back to you as soon
> as possible. For your records, the details of the ticket are listed below. 
> When replying, please make sure that the ticket ID is kept in the subject 
> line to ensure that your replies are tracked appropriately.
> >
> >      Ticket ID: LGJ-997921
> >      Subject: Large Classic File Questions
> >      Department: Support netCDF
> >      Priority: Normal
> >      Status: Open
> >
> >
> >
> > The NetCDF libraries are developed at the Unidata Program Center,
> > in Boulder, Colorado, funded primarily by the National Science Foundation.
> >
> > All support requests are handled by the development team. No dedicated
> > support staff are funded at this time. For this reason we cannot guarantee
> > response times, nor that we can resolve every support issue, although we
> > do our best to respond within 72 hours.
> >
> > It is in the nature of support requests that the same question is asked
> many
> > times. We urge you to peruse the support archives for material relating
> to
> > your support request:
> >
> > https://www.unidata.ucar.edu/support/help/MailArchives/netcdf/
> >
> > If you are having trouble building netCDF, please take a look at the
> > "Building NetCDF" page:
> >
> > https://www.unidata.ucar.edu/software/netcdf/docs/getting_and_building_netcdf.html
> >
> > Windows users should see this page:
> >
> > https://www.unidata.ucar.edu/software/netcdf/docs/winbin.html
> >
> > Complete documentation (including a tutorial, and sample programs in C,
> Fortran,
> > Java, and other programming languages) can be found on the netCDF 
> > Documentation page:
> >
> > http://www.unidata.ucar.edu/software/netcdf/docs/
> > http://www.unidata.ucar.edu/software/netcdf/examples/programs/
> >
> > If you resolve your issue through one of these methods, please send a reply 
> > to
> > this email, letting us know that you no longer need support. This will help
> > us spend more time on netCDF development.
> >
> > Best regards,
> >
> > Unidata User Support
> >
> 
> 
> 


Ticket Details
===================
Ticket ID: LGJ-997921
Department: Support netCDF
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.