Re: Memory problems with UnidataStationObsDataset

Hi Eric: my profiler claims the following code consumes about 5 MB on the heap. 
Not sure if you are doing something a lot more complicated. Are you keeping 
pointers to the data? Anything non-standard with garbage collection?


public void testMetarDataset() throws IOException {
   long start = System.currentTimeMillis();

   StationObsDataset sod = (StationObsDataset) PointObsDatasetFactory.open( 
topDir+"ldm/Surface_METAR_20060701_0000.nc", null, null);
   DataIterator iter = sod.getDataIterator(0);
   double sum = 0.0;
   int count = 0;
   while (iter.hasNext()) {
     PointObsDatatype obs = (PointObsDatatype) iter.nextData();
     StructureData sdata = obs.getData();
     sum += sdata.getScalarDouble("wind_speed");
     count++;
   }
   long took = System.currentTimeMillis() - start;
   System.out.println(" that took = "+took+" msec");
   System.out.println("sum= "+sum+" count = "+count);
 }

that took = 6405 msec
sum= -1.4362039E8 count = 112763

If i open through catalog and read from dods, it takes ~ 20 Mb:

 public void testMetarDataset() throws IOException {
   long start = System.currentTimeMillis();

   ThreddsDataFactory fac = new ThreddsDataFactory();
   ThreddsDataFactory.Result result = fac.openDatatype( 
"http://motherlode.ucar.edu:8080/thredds/catalog/station/metar/catalog.xml#NWS/METAR/Surface_METAR_20060701_0000.nc";,
 null);
   StationObsDataset sod = (StationObsDataset) result.pobsDataset;
   long took = System.currentTimeMillis() - start;
   System.out.println(" open took = "+took+" msec");
   start = System.currentTimeMillis();

   //StationObsDataset sod = (StationObsDataset) PointObsDatasetFactory.open( 
topDir+"ldm/Surface_METAR_20060701_0000.nc", null, null);
   DataIterator iter = sod.getDataIterator(0);
   double sum = 0.0;
   int count = 0;
   while (iter.hasNext()) {
     PointObsDatatype obs = (PointObsDatatype) iter.nextData();
     StructureData sdata = obs.getData();
     sum += sdata.getScalarDouble("wind_speed");
     count++;
   }
   took = System.currentTimeMillis() - start;
   System.out.println(" read took = "+took+" msec");
   System.out.println("sum= "+sum+" count = "+count);
 } 

open took = 2498 msec
read took = 22249 msec
sum= -1.4362039E8 count = 112763


Eric Russell wrote:
> At 1:26 PM -0600 7/4/06, John Caron wrote:
>
> > Hi Eric:
> >
> > Can you make the file you are working on available through ftp or http? thanks
>
>
> I was using the link from the catalog you sent me:
> http://motherlode.ucar.edu:8080/thredds/catalog/station/metar/catalog.xml
> (the third on the list, because the "latest" one is usually in the process 
> of growing and is not yet large enough to cause problems), and opening using
> ThreddsDataFactory.openDatatype(InvDataset, CancelTask). Thanks,
>
> Eric
>
>
> > Eric Russell wrote:
> >
> > > I've been having OutOfMemoryError issues with UnidataStationObsDataset.
> > > Even when I iterate throught the file without retaining a reference to
> > > anything, like this:
> > >
> > > DataIterator iter = _dataset.getDataIterator(0);
> > > while (iter.hasNext()) {
> > >    PointObsDatatype obs = (PointObsDatatype)iter.nextData();
> > > }
> > >
> > > I get an OutOfMemoryError, even with a 512MB max heap size. I did some 
> > > investigation with a profiler, and the culprit seems to be the ArrayList 
> > > of Station objects defined on line 45 of StationObsDatasetImpl. Is it 
> > > necessary to keep the Station objects around? If so, could you provide
> > > an alternate means of iterating throught the PointObsDatatype's that 
> > > doesn't maintain those references? Thanks,
> > >
> > > Eric