[netCDF #UPO-861727]: getVar crashed with compressed file

Baffling. The only thing I can think of is that for some reason
the dataset you have does not compress well with libz.
Perhaps you might try different compression levels to see if that affects
things either slower or faster. 

> 
> I would like to add that the getVar operation is just very slow. It will 
> eventually return after 10-15 mins on a compressed file using nccopy. If 
> using non compressed, or other already compressed files, the function returns 
> well within a second.
> 
> Thanks,
> 
> Norman
> 
> -----Original Message-----
> From: Unidata netCDF Support <address@hidden>
> Sent: Monday, July 30, 2018 12:51 PM
> To: Lo, Norman <address@hidden>
> Cc: address@hidden; Lo, Norman <address@hidden>
> Subject: [netCDF #UPO-861727]: getVar crashed with compressed file
> 
> Ok, it sounds like the netcdf-c library per-se is ok (else nccopy would 
> fail). So, the problem would seem to be in either your program or in getVar().
> 
> What I would like to see is the arguments to getVar that are failing plus the 
> ncdump output of that specific variable. I would be looking to make sure that 
> the arguments are consistent with the actual variable definition. You could 
> try doing some print statements before the getVar, or if you are on Linux, 
> you could use gdb to put a breakpoint inside getVar.
> 
> > Correct. I can read step (3), but not step (2).
> >
> > The program is proprietary. Is there something you can suggest me to try ?
> >
> > Thanks,
> >
> > Norman
> >
> > -----Original Message-----
> > From: Unidata netCDF Support <address@hidden>
> > Sent: Sunday, July 29, 2018 5:48 PM
> > To: Lo, Norman <address@hidden>
> > Cc: address@hidden; Lo, Norman <address@hidden>
> > Subject: [netCDF #UPO-861727]: getVar crashed with compressed file
> >
> > So I read your response to mean that nccopy can read b.nc (step 3) but your 
> > program cannot read b.nc using getVar().
> > Is the program small enough (and non-proprietary) so that you can send it 
> > to me to try out?
> >
> > > The steps are correct. When I try to read the c.nc file again, the 
> > > program does not actually crash (Sorry I described the situation 
> > > incorrectly), instead, it's just freeze at the call for a long time (CPU 
> > > is running 100%). So I have to break it eventually.
> > >
> > > I tried to do the following:
> > >
> > > (1) original file: a.nc
> > > (2) compress a.nc to become b.nc  (using nccopy -d 1)
> > > (3) decompress b.nc to become c.nc. (using nccopy -d 0)
> > >
> > > I can read the c.nc file fine.
> > >
> > >
> > > Thank you very much for your time,
> > >
> > > Norman
> > >
> > >
> > > -----Original Message-----
> > > From: Unidata netCDF Support <address@hidden>
> > > Sent: Thursday, July 26, 2018 10:30 PM
> > > To: Lo, Norman <address@hidden>
> > > Cc: address@hidden
> > > Subject: [netCDF #UPO-861727]: getVar crashed with compressed file
> > >
> > > Let me make sure I understand.
> > > 1. You have some uncompressed file, call it u.nc.
> > > 2. You compress it using nccopy -d 1 u.nc c.nc to produce the file
> > > c.nc 3. You then try to read the c.nc file using a program that
> > > calls getVar and when you do so, your program crashes.
> > > Do I have that right?
> > >
> > > >
> > > > We have  program that uses the netcdf-cxx-4.2 interface to manipulate 
> > > > data stored in netcdf files. As far as I can tell, everything works 
> > > > fine if the netcdf files are not compressed.
> > > >
> > > > However, if I use "nccopy -d 1" to compress the files. The function 
> > > > getVar crashed.
> > > >
> > > >
> > > > The function we are using is:
> > > >
> > > >
> > > > void netCDF::NcVar::getVar        (              const std::vector< 
> > > > size_t > &        start,
> > > > const std::vector< size_t > &        count,
> > > > const std::vector< ptrdiff_t > & stride, const std::vector<
> > > > ptrdiff_t
> > > > > & imap, short * dataValues
> > > > )                              const
> > > >
> > > >
> > > >
> > > >
> > > > Could you let me know if I did something wrong? Thank you very
> > > > much,
> > > >
> > > > Norman
> > > >
> > > >
> > >
> > > =Dennis Heimbigner
> > > Unidata
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: UPO-861727
> > > Department: Support netCDF
> > > Priority: High
> > > Status: Open
> > > ===================
> > > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > > Unidata inquiry tracking system and then made publicly available through 
> > > the web.  If you do not want to have your interactions made available in 
> > > this way, you must let us know in each email you send to us.
> > >
> > >
> > >
> >
> > =Dennis Heimbigner
> > Unidata
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: UPO-861727
> > Department: Support netCDF
> > Priority: High
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> >
> 
> =Dennis Heimbigner
> Unidata
> 
> 
> Ticket Details
> ===================
> Ticket ID: UPO-861727
> Department: Support netCDF
> Priority: High
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: UPO-861727
Department: Support netCDF
Priority: High
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.