[IDV #JHS-477170]: 20130607: WRF like model aggregation in the IDV

Greetings  Chia-Ying!

> Hi Sean,
> 
> Thanks for the suggestion, looks like I can only do the analysis on a
> single time step and animate outside of the IDV. Could you refer me the
> docs how to create animation outside of IDV?  Thanks.

We do not maintain any docs on how to create animations outside of the
IDV, but there are several tools that exist which will create animations
from single images. For a few examples, check out:

http://www.pymolwiki.org/index.php/Making_Movies

I generally use Mencoder/Mplayer.

> 
> two more question about WRF analysis (about jython).
> 1) I am using formula function to create a derived field, following the
> method from:
> http://www.unidata.ucar.edu/software/idv/docs/workshop/formulas/JythonMethods.html
> 
> In the user's library, I define trmass as:
> 
> 1 def trmass(ph,phb,p,pb,var):
> 2       dx = 1333.33
> 3       dy = 1333.33
> 4       z1 = (ph + phb)
> 5       z = 0.5*(z1[1:,:,:]+z1[0:-1,:,:])
> 6       pp = p + pb
> 7       rho = pp/z
> 8       dz = z1[1:,:,:]-z1[0:-1,:,:]
> 9       mass = dx*dy*dz*rho*var
> 10     return mass
> 
> While  loading it, I received an error message as:
> ceback (most recent call last):
> File "<string>", line 1, in <module>
> File "<string>", line 6, in trmass
> TypeError: __getitem__(): 1st arg can't be coerced to visad.Real,
> visad.RealTuple, int
> 
> I have tried function trmass with python, and it works. Not sure where
> goes wrong?

It's not easy to debug the Jython errors without playing around with it
locally. The thing is that the fields (variable) in the IDV are VisAD objects,
which means you have to be careful how you mathematically manipulate
them. It's not as simple as it is for a numpy array, for example.

I recommend taking a look at the following page:

https://www.unidata.ucar.edu/software/idv/docs/userguide/tools/JythonLib.html

Note that to get a slice of a field in the IDV, you cannot simply index into it 
as you
would a numpy array, but you need to use a Jython method, something like

getNthTimeGrid

or 

make2D

> 
> 2) how to load external python module in IDV, such as numpy?
> On My Mac machine, I have installed numpy, and I tried to import it with
> the absolute path:
> import
> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy
> as np
> This cause the syntax Error.

So this is perhaps the most critical comment to understand - Jython is not 
Python,
at least as Python is understood by people who actually use Python to get 
scientific
work done :-) Python, like you and I understand, ties into C quite nicely, and 
in fact 
is built on C. Python, from python.org, is sometimes called CPython. 

Jython is basically the core CPython v2.6 api implemented in Java. What that 
means 
is that you can write a script using pure python, and there is underlying Java 
code 
that understands what that syntax means and how to do that in Java. So, for 
example, 
if you type out

print "hello, world!"

the underlying Java code will see the print statement and execute:

System.out.println("hello, world!");

Note that Jython isn't something that Unidata came up with, but we did
choose to use it because it's easier for non-Java coders to work with.

When it comes to all of the awesome modules available outside of the core
CPython distribution, such as numpy and scipy, they cannot be used in
Jython because they are not pure python libraries (they end up wrapping
C and/or FORTRAN code at some point, which Jython does not know what
to do with).

I really wish Jython could be more, but it's just not there, and given the
availability and popularity of other JVM based languages, such as Groovy, 
I don't ever see it getting close to CPython's capabilities.

I hope this helps explain some things; sorry I don't have better news.

Cheers,

Sean

> 
> I will continue working on solving these two questions, but at the mean
> time, if you have any advice, please let me know.  Thanks again.
> 
> Chia-Ying
> 
> > Greetings Chia-Ying!
> >
> > Yes, I think we've found the problem. I would suggest trying to spatially
> > subset
> > in the horizontal directions to get a quick look, and only use the full
> > resolution
> > datasets when you have to do so. For more information on subsetting,
> > checkout
> > the Dealing with Large Datasets topic in our docs:
> >
> > http://www.unidata.ucar.edu/software/idv/docs/workshop/datadisplays/wrf/InteractiveWRFGridDisplays.html
> >
> > It's also possible to do the analysis on a single time step and animate
> > outside of the IDV software.
> >
> > Cheers,
> >
> > Sean
> >
> >> Hi Sean,
> >>
> >> Thanks, Here are the information.
> >>
> >> 1) loading multiple wrfout files (24) with grid numbers as 601, 400, 36
> >> in
> >> x, y, and z directions, the three numbers are:
> >> 182/201/7331
> >> selecting to display an iso-surface of one 3D variable:
> >> 6621/7331/7331
> >>
> >> 2) loading one wrfout file and selecting display an iso-surface of one
> >> 3D
> >> variable:
> >> 1085/1874/7331
> >> displaying the derived horizontal wind vectors
> >> 1997/3917/7331
> >>
> >> It seems display 3D variables with multiple wrfout files is the main
> >> issue. However, even I load only one file, after display two to three 3D
> >> variables, the the maximum amount of memory used becomes very close to
> >> total amount of memory allocated to IDV.
> >>
> >>
> >> Chia-Ying
> >>
> >> > Greetings Chiaying!
> >> >
> >> > In the lower left-hand corner of the IDV is a display of the current
> >> time,
> >> > in
> >> > UTC. If you click that clock, the display will change and show you
> >> > information
> >> > about your current memory usage. There are three numbers, seperated by
> >> > slashes, and they describe the follow: the first is the amount of
> >> memory
> >> > currently used by the IDV, the second is the maximum amount of memory
> >> used
> >> > at any point during your IDV session, and the third is the total
> >> amount of
> >> > memory allocated to the IDV. Would you mind loading the display you
> >> > described
> >> > and sending me those three numbers? If the second number equals the
> >> third,
> >> > the IDV can become quite unresponsive.
> >> >
> >> > Also, what is the grid spacing of the WRF output you are viewing?
> >> >
> >> > Thanks!
> >> >
> >> > Sean
> >> >
> >> >> Hi Sean,
> >> >>
> >> >> I have another question about loading multiple wrfout files and the
> >> IDV
> >> >> performance.  While loading 3D data, it takes IDV a noticeable time
> >> to
> >> >> respond to every commend. Each wrfout I am currently using is about
> >> 500
> >> >> M,
> >> >> and I load 36 of them.  I display only the first 15 layers (in total,
> >> >> there are 36 layers for 3D fields).  IDV became very slow when I
> >> tried
> >> >> to
> >> >> display point volume... which took 5 minutes or longer every time
> >> when I
> >> >> try  to adjust color bar, change view angle, or change vertical
> >> range,
> >> >> etc.
> >> >> we have choice to put data on local machine (MAC) or on the cluster
> >> >> (load
> >> >> it trough internet).  I load it through the local machine, which
> >> should
> >> >> be
> >> >> a more efficient way.
> >> >>
> >> >> The system setting I set for IDV is 7000M memory for IDV (I have 8G
> >> >> memory), permGen size is 128 M, write grib index in disk Cache, data
> >> >> cache
> >> >> Memory Percent 50, unlimited image size, enable geometry by
> >> reference,
> >> >> and
> >> >> enable access to image data by reference.  I wondering if there is
> >> >> anyway
> >> >> I can increase the performance of IDV.
> >> >>
> >> >> Also, I sent you and Tom another email about displaying observed
> >> >> trajectories from drifter data yesterday, but I forgot to keep the
> >> title
> >> >> "
> >> >> [IDV #JHS-477170]", and not sure if the email will go to you (and
> >> Tom)
> >> >> or
> >> >> not.  The automatic response assigned me a number "[IDV #TRI-715395]:
> >> >> display drifter trajectory files in IDV".
> >> >>
> >> >> Thanks.
> >> >>
> >> >> -Chiaying
> >> >>
> >> >>
> >> >> > Hi Sean,
> >> >> >
> >> >> > Thanks for the reply.  I can display/animate all the three model
> >> >> > components without problem. Thanks again.
> >> >> >
> >> >> >
> >> >> > -Chiaying
> >> >> >> Hi Chiaying!
> >> >> >>
> >> >> >>> Hi Sean,
> >> >> >>>
> >> >> >>> Thanks for the answer, I just tried and umwmout.ncml works well
> >> now
> >> >> >>> after
> >> >> >>> the changes!
> >> >> >>
> >> >> >> Great, I'm glad it worked! See below for the solution to the HYCOM
> >> >> >> output.
> >> >> >>
> >> >> >>>
> >> >> >>> I have another question about loading ocean model (HYCOM) outputs
> >> >> (also
> >> >> >>> NetCDF file) into IDV and loop them.
> >> >> >>>
> >> >> >>> In the HYCOM output, there is time-dimension for all the
> >> variables.
> >> >> So,
> >> >> >>> I
> >> >> >>> first tried to load them into IDV with "Aggregate Grids by Time"
> >> >> >>> option,
> >> >> >>> and then I got an error message saying  "no coordinate variable
> >> for
> >> >> agg
> >> >> >>> dimension = time."  My first guess was that the dimension name
> >> for
> >> >> time
> >> >> >>> in
> >> >> >>> HYCOM output is "MT" , and IDV might not able to recognize it.
> >> >> another
> >> >> >>> possible issue is that MT is written as day since  1900-12-31
> >> (see
> >> >> >>> below)
> >> >> >>> dimensions:
> >> >> >>> MT = UNLIMITED ; // (1 currently)
> >> >> >>> Latitude = 1000 ;
> >> >> >>> Longitude = 1000 ;
> >> >> >>> Depth = 40 ;o
> >> >> >>> variables:
> >> >> >>> double MT(MT) ;
> >> >> >>> MT:long_name = "time" ;
> >> >> >>> MT:units = "days since 1900-12-31 00:00:00" ;
> >> >> >>> MT:calendar = "gregorian" ;
> >> >> >>> MT:axis = "T" ;
> >> >> >>> float salinity(MT, Depth, Latitude, Longitude) ;
> >> >> >>> salinity:coordinates = "Date" ;
> >> >> >>> salinity:standard_name = "sea_water_salinity" ;
> >> >> >>> salinity:units = "psu" ;
> >> >> >>> salinity:_FillValue = 1.267651e+30f ;
> >> >> >>> salinity:valid_range = 9.70682f, 38.86856f ;
> >> >> >>> salinity:long_name = " salinity [20.1H]" ;
> >> >> >>>
> >> >> >>>
> >> >> >>> Then, I tried to write a .ncml file to see if I can aggregate
> >> HYCOM
> >> >> >>> output
> >> >> >>> as what you did for umwm. No luck in this either (with an error
> >> >> message
> >> >> >>> saying no griddata found for archv.ncml).  Not sure what causes
> >> the
> >> >> >>> problem.
> >> >> >>>
> >> >> >>> Can you give me some hint how I should solve this issue?  I put
> >> the
> >> >> >>> hycom
> >> >> >>> data and my archv.ncml online at:
> >> >> >>> http://orca.rsmas.miami.edu/~chiaying/IDV/
> >> >> >>
> >> >> >> Your ncml is very close! The issue is that a time dimension and
> >> >> variable
> >> >> >> already
> >> >> >> exists, but it's not called "time", which is what the IDV expects
> >> >> when
> >> >> >> aggregating
> >> >> >> (this is a deficiency in the IDV that needs to be addressed).
> >> Since
> >> >> you
> >> >> >> already have
> >> >> >> time in your files, you can simply do a joinExisting aggregation
> >> in
> >> >> >> ncml,
> >> >> >> rather
> >> >> >> than a joinNew, and there is no need to create a new time variable
> >> >> from
> >> >> >> the file
> >> >> >> name. In order to do this, make the following changes to your ncml
> >> >> file.
> >> >> >>
> >> >> >> Change the contents of the aggregation tag to:
> >> >> >>
> >> >> >> aggregation dimName="MT" type="joinExisting"
> >> >> >>
> >> >> >> Change the contents of the scan tag to:
> >> >> >>
> >> >> >> scan location="/home/disk/manta9/chiaying/2012_isaac/archvnc/"
> >> >> >> suffix=".nc" subdirs="false"
> >> >> >>
> >> >> >> By the way, the units on the MT variable are perfectly acceptable,
> >> so
> >> >> no
> >> >> >> issues
> >> >> >> there :-)
> >> >> >>
> >> >> >> If you are interested in learning more about ncml, check out the
> >> ncml
> >> >> >> tutorial
> >> >> >> and cookbook:
> >> >> >>
> >> >> >> http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html
> >> >> >>
> >> >> >> http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Cookbook.html
> >> >> >>
> >> >> >> Let me know if you have any questions!
> >> >> >>
> >> >> >> Cheers!
> >> >> >>
> >> >> >> Sean
> >> >> >>
> >> >> >>>
> >> >> >>> Thanks for your help.
> >> >> >>>
> >> >> >>> Chiaying
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> > Greetings Chiaying!
> >> >> >>> >
> >> >> >>> > My name is Sean, and I put together the README file and the
> >> ncml
> >> >> >>> file.
> >> >> >>> >
> >> >> >>> > First, I would rename the ncml file from umwmout.nml.xml to
> >> >> >>> > umwmout.ncml.
> >> >> >>> >
> >> >> >>> > Then, I would check the following:
> >> >> >>> >
> >> >> >>> > In the scan tag, check to make sure the dateFormatMark matches
> >> >> >>> > how the date is expressed in your file name. For example, the
> >> >> error
> >> >> >>> > message you passed along indicates that your files are named
> >> like:
> >> >> >>> >
> >> >> >>> > umwmout_2012-08-26_12:00:00.nc
> >> >> >>> >
> >> >> >>> > which means the dateFormatMark should be
> >> >> >>> >
> >> >> >>> > umwmout_#yyyy-MM-dd_HH:mm:ss"
> >> >> >>> >
> >> >> >>> > The ncml file I sent used a slightly different dateFormatMark.
> >> >> >>> >
> >> >> >>> > If everything works, then loading the ncml file using "I'm
> >> feeling
> >> >> >>> > lucky" is all you will need to do (that is, you won't need to
> >> do
> >> >> >>> > step three as you described in your previous email).
> >> >> >>> >
> >> >> >>> > Please let me know if it works with these changes.
> >> >> >>> >
> >> >> >>> > Cheers!
> >> >> >>> >
> >> >> >>> > Sean
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >> Hi Tom,
> >> >> >>> >>
> >> >> >>> >> Thanks for the files.  I have tried to use umwmout.nml.xml
> >> file.
> >> >> >>> >> However,
> >> >> >>> >> I got an error message wile trying loading umwmout*.nc data.
> >> Here
> >> >> is
> >> >> >>> >> what
> >> >> >>> >> i did:
> >> >> >>> >>
> >> >> >>> >> 1) I put umwmout.nml.xml in the same folder as where the
> >> >> unwmout*.nc
> >> >> >>> >> are.
> >> >> >>> >> 2)  load umwmout.nml.xml with "I'm feeling lucky" data choice
> >> (I
> >> >> >>> have
> >> >> >>> >> changed the path)
> >> >> >>> >> 3)  load umwmout*.nc file with "Aggregate Grids by Time"
> >> option,
> >> >> and
> >> >> >>> I
> >> >> >>> >> got
> >> >> >>> >> an error message:
> >> >> >>> >>
> >> >> >>> >> There was an error loading the data:
> >> >> >>> >> Error creating data source:aggregated.netcdf.grid with:
> >> >> >>> >> [/home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_12:00:00.nc,
> >> >> >>> >> /home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_13:00:00.nc,
> >> >> >>> >> /home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_14:00:00.nc]
> >> >> >>> >> Grid data source failed making data set:
> >> >> >>> >> /Users/chiayinglee/.unidata/idv/DefaultIdv/tmp/multigrid_b3ad0ca8-652c-4a8e-bd66-093cd3468f3b0.ncml
> >> >> >>> >> AggregationExisting: no coordinate variable for agg
> >> >> >>> >> dimension= time
> >> >> >>> >>
> >> >> >>> >> It seems that  IDV still can not recognize the time.  Am I
> >> >> missing
> >> >> >>> >> something? Thanks.
> >> >> >>> >>
> >> >> >>> >> Chiaying
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> > Hi Shuyi,
> >> >> >>> >> >
> >> >> >>> >> > re:
> >> >> >>> >> >> Thank you so much for your help on IDV!  My group is
> >> learning
> >> >> to
> >> >> >>> use
> >> >> >>> >> >> IDV for our coupled model output. I really like the feature
> >> >> that
> >> >> >>> we
> >> >> >>> >> can
> >> >> >>> >> >> display/analyze the atmos/ocean/wave fields together in
> >> IDV.
> >> >> >>> >> >> Milan and Chiaying (CC'd) may email you for some questions
> >> as
> >> >> we
> >> >> >>> >> >> are learning/experimenting with various functions.
> >> >> >>> >> >
> >> >> >>> >> > This sounds good.  Did you pass along the README file with
> >> >> >>> comments
> >> >> >>> on
> >> >> >>> >> > how your coupled model output could be better structured for
> >> >> use
> >> >> >>> by
> >> >> >>> >> > the IDV and other applications that provide support for data
> >> in
> >> >> >>> netCDF
> >> >> >>> >> > format?
> >> >> >>> >> >
> >> >> >>> >> > Tom
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >
> >> >> >>> > Ticket Details
> >> >> >>> > ===================
> >> >> >>> > Ticket ID: JHS-477170
> >> >> >>> > Department: Support IDV
> >> >> >>> > Priority: Normal
> >> >> >>> > Status: Open
> >> >> >>> >
> >> >> >>>
> >> >> >>>
> >> >> >>
> >> >> >> Ticket Details
> >> >> >> ===================
> >> >> >> Ticket ID: JHS-477170
> >> >> >> Department: Support IDV
> >> >> >> Priority: Normal
> >> >> >> Status: Open
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >
> >> > Ticket Details
> >> > ===================
> >> > Ticket ID: JHS-477170
> >> > Department: Support IDV
> >> > Priority: Normal
> >> > Status: Open
> >> >
> >>
> >>
> >
> > Ticket Details
> > ===================
> > Ticket ID: JHS-477170
> > Department: Support IDV
> > Priority: Normal
> > Status: Open
> >
> 
> 

Ticket Details
===================
Ticket ID: JHS-477170
Department: Support IDV
Priority: Normal
Status: Open