[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #YRD-168110]: Multiple Section reads appear to be skipping data? (java)

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #YRD-168110]: Multiple Section reads appear to be skipping data? (java)
  • Date: Wed, 05 Nov 2014 08:12:44 -0700
Hi Clifford,

> So lets say I have NetCDF file that I am trying to read with a Section that
> looks like this:
> 
> 0:0, 0:0, 0:719, 0:1439
> 
> When I read this entire section at once, I get back 1036800 elements
> clocking in at 4147200 bytes. So I says to myself, I says 'what if I split
> the read up into multiple sections, and process the partial Array results
> one at a time, so that I'm not holding a reference to ~4mb of memory".
> 
> So I split the read up such that it looks like the following (this is the
> actual toString() result of each Section immediately before the read takes
> place)
> 
> 0:0, 0:0, 0:359, 0:719
> 
> 0:0, 0:0, 360:719, 720:1439
> 
> BUT, the results from each of these arrays combined totals only
> 518400 elements weighing in at 2073600 bytes - I'm basically missing half
> the data here!

It appears that you have only requested half the data.  To get all the data,
you either need the four sections specified by

 0:0, 0:0, 0:359, 0:719
 0:0, 0:0, 0:359, 720:1439
 0:0, 0:0, 360:719, 0:719
 0:0, 0:0, 360:719, 720:1439

or you need to make each of the two sections request half the data:

 0:0, 0:0, 0:359, 0:1439
 0:0, 0:0, 360:719, 0:1439

--Russ

> Whats more, If I make the Sections smaller and increase the number of
> variable reads in an attempt to reduce memory usage even further, I lose
> even more data!
> 
> What am I doing wrong here? If Sections are not appropriate for 'chunking'
> data reads, is there a mechanism in the library that is more suited to this
> task?
> --
> Clifford M. Harms
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: YRD-168110
Department: Support netCDF
Priority: Normal
Status: Closed