[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #XXB-855611]: consider this bundle for your memory profiling

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [IDV #XXB-855611]: consider this bundle for your memory profiling
  • Date: Fri, 08 Feb 2013 12:20:06 -0700
Brian,

Thanks for submitting this bundle for analysis. The IDV team has scrutinized 
this bundle and we would like to tell you about our conclusions.

This bundle is comprised 21 remote data sources from a NASA GrADS-DODS Server. 
As you know, these data are comprised of monthly, 3 hourly, and hourly data 
spanning several decades. We count 2,202,131 time steps. While the IDV does not 
load the entire data source, it does load the times for use in the Field 
Selector.  It is the loading of the times that accounts for the bulk of the 
activity observed in the IDV both in terms of memory usage and time to load. On 
my Macbook Pro, it takes about 2 minutes to load these times.

In addition, we have examined the memory usage numbers you reported from the 
memory monitor. Without getting into too much detail, unfortunately, the 3 
gigabyte number is deceptive because is comprises both "live" and "unreachable" 
memory. The real memory usage to load these times is more around ~350 
megabytes. (Indeed, you can load this bundle while only allocating 500 
megabytes to the IDV). The rest is memory that was once used by the IDV, but is 
now waiting to be reclaimed (garbage collected in computer science parlance). 
In sum, we did not uncover any anomalous memory consumption problems here.

Over the course of this analysis, we did not uncover problems with the IDV. 
This bundle is simply loading a lot of times from a remote source.

Please let us know if you are seeing performance issues with other bundles.

Best Wishes,

Unidata IDV Team 

> Here's a little bundle that merely loads some URL data sources.
> No displays or views are created.
> 
> They are big datasets - hourly for decades - so just indexing them seems
> to use a lot of memory (like 3G according to the memory counter).
> I'm not sure if that is really a problem, or just a virtual one.
> Don suggested I submit this for your profiling activity.
> 
> 
> 
> Brian Mapes
> address@hidden or address@hidden


Ticket Details
===================
Ticket ID: XXB-855611
Department: Support IDV
Priority: Critical
Status: Closed