[IDV #AWX-208976]: Dataset subsetting gets lost, perhaps because for reasons related to this projection related NullPointer error?

Hi Brian,

> memory, I can load the template and change the time driver to get 4 time 
> steps. If I try 5 time steps, I got error message. If you kill my InteliJ 
> (the Java IDE), I can do 5.
> 
> And does the memory counter top out?
> Is it actually memory, or is it a bug that only shows itself when you try 
> something ambitious?
> 

When you get near the upper limit of memory allocated to the Java Virtual 
Machine, interesting, 
sometimes unpredictable, and unintended consequences can happen. This can be 
due, in part, 
to the fact that java needs contiguous pieces of memory to put objects into. 
It's not a bug in the 
IDV; it due to the design of the Java Virtual Machine with regards to memory 
management. Java
7 has several improvements memory management, and hopefully this improvements 
will help
projects like the IDV, where users are allowed to do 'ambitious' things 
(assuming their hardware
can keep up).
 
> want, you can run the activity monitor in your Mac to check the memory usage.
> Yes I am constantly doing that.
> 
> 
> In the IDV, we use 80% of the total memory of your system instead of the free 
> memory. This may cause some confusion. Sometime, you already max out the 
> memory before you reach the third number.
> 
> Yes I hit the memory limit frequently.

This is why sometimes we cannot reproduce your bugs - you are hitting the 
limitations
of Java on your current hardware.

> I am accustomed to IDV getting very slow when it hits a memory limit. 

Sometimes the slowness is due to garbage collection - GC is expensive for large 
objects like
we deal with in the IDV.

>Sometimes it freezes.
> But not having projection-related NullPointer errors immediately upon hitting 
> "OK" in the animation control, or white messages in the viewport.
> This is different.

As I said, when you get near the upper limit of memory allocated to the Java 
Virtual Machine, interesting, 
sometimes unpredictable, and unintended consequences can happen. There is a 
reason Tom harps on the
memory usage, and here is an example of why it's not wise to ignore it. The 
errors generated
under these circumstances can be quite misleading and send developers down many 
dead-end
rabbit holes.

> 
> 
> The solution is either to add more memory to your Mac, or to remove a few 3D 
> displays.
> 
> I run this on a desktop machine not my laptop.

How much memory does your desktop have, and are you unable to increase it
due to available RAM slots?

> I simply don't believe this is just a size issue.

Unfortunately it is a size/memory issue. Yuan has shown it by artificially 
limiting the
amount of memory in his IDV session. He can control wether or not the
NullPointerException appears or disappears without fail.

> Yes I could declare IDV unfit for big problems (for reasons that remain 
> spooky and undiagnosed), and 
> just never dare to try anything that might get near some unknowable limits, 
> superstitiously. 
> That would be one way to solve this problem. I am still hoping for a better 
> solution.

Or, you could declare the the IDV on insufficient hardware unfit for big 
problems, which is the
case here. More memory and hope for improvements in Java 7 is the only thing 
that will fix / 
mitigate these issues. Sometimes, you just have to throw more hardware at the 
problem when
you hit software limitations. Don't get me wrong, there are bugs in the IDV, 
but that's not what
you are seeing here.

Sean

PS: By the way, here is a nine panel .xidv I created on my desktop, starting 
from your
bundle, without any issues (I like brews from Boulder Beer, thank you):

http://motherlode.ucar.edu/repository/entry/show/RAMADDA/IDV+Community+Resources/Bundles/Sean/Bob+monsoon+depression+dissected+9time+%28xidv%29?entryid=e0ee895e-03f8-49c6-ac36-a3827a44dcde

We can settle the eating hat score next time your in town :-)

> 
> 

Ticket Details
===================
Ticket ID: AWX-208976
Department: Support IDV
Priority: Normal
Status: Open