[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #JHS-477170]: 20130607: WRF like model aggregation in the IDV



Greetings Chia-Ying,

Apologies for the delay. Very cool animation - I look forward to seeing
more of what you and your group can do with the IDV! Before answering
your specific questions, were you able to solve any of them on your own?
If you let me know which remain, I'll be happy to respond to you ASAP.

Thanks!

Sean

> Thanks, Sean,
> 
> In the end, I went back to use python to do the scientific calculations
> and put the results into wrfout, which gives me more flexibility on
> manipulating model results.  And for the animation, I output .isl and
> .xidv, then script them to generate the images for each time step.  Then,
> as you suggested, I created animation outside of IDV.
> 
> Here is our first IDV animation:
> http://orca.rsmas.miami.edu/~chiaying/2013_CARTHE/hercules/2tke_trmass_sst_w.html
> 
> In this image, we have air-mass (tracer) emitted from  a single point
> source, and advected by winds. The background is SST from ocean model, and
> the blue marble map. There is still some fine tuning I need to do improve
> it, but we really appreciate your help.
> 
> Some questions for the fine tuning:
> 1) the legends and IDV logo  are not in the images. I think this is
> because the last data source I load covers smaller area (as the magenta
> axes indicate) and the view angle I chose.  I can see the legends  if I
> zoom in to the magenta box or I change the angle.  Is there a way I can
> fixe the legends to make sure it is alway there.
> 
> 2) the tick-label for longitude and latitude are below the blue marble and
> SST images.  Is there any way I can bring them up?
> 
> 3) saving the auto-rotation animation.  On the left side of the IDV view
> window, there is an auto-rotation button.   Does IDV have a function to
> save a movie while image is rotating?
> 
> Thanks.
> 
> Chia-Ying
> 
> 
> > Greetings  Chia-Ying!
> >
> >> Hi Sean,
> >>
> >> Thanks for the suggestion, looks like I can only do the analysis on a
> >> single time step and animate outside of the IDV. Could you refer me the
> >> docs how to create animation outside of IDV?  Thanks.
> >
> > We do not maintain any docs on how to create animations outside of the
> > IDV, but there are several tools that exist which will create animations
> > from single images. For a few examples, check out:
> >
> > http://www.pymolwiki.org/index.php/Making_Movies
> >
> > I generally use Mencoder/Mplayer.
> >
> >>
> >> two more question about WRF analysis (about jython).
> >> 1) I am using formula function to create a derived field, following the
> >> method from:
> >> http://www.unidata.ucar.edu/software/idv/docs/workshop/formulas/JythonMethods.html
> >>
> >> In the user's library, I define trmass as:
> >>
> >> 1 def trmass(ph,phb,p,pb,var):
> >> 2       dx = 1333.33
> >> 3       dy = 1333.33
> >> 4       z1 = (ph + phb)
> >> 5       z = 0.5*(z1[1:,:,:]+z1[0:-1,:,:])
> >> 6       pp = p + pb
> >> 7       rho = pp/z
> >> 8       dz = z1[1:,:,:]-z1[0:-1,:,:]
> >> 9       mass = dx*dy*dz*rho*var
> >> 10     return mass
> >>
> >> While  loading it, I received an error message as:
> >> ceback (most recent call last):
> >> File "<string>", line 1, in <module>
> >> File "<string>", line 6, in trmass
> >> TypeError: __getitem__(): 1st arg can't be coerced to visad.Real,
> >> visad.RealTuple, int
> >>
> >> I have tried function trmass with python, and it works. Not sure where
> >> goes wrong?
> >
> > It's not easy to debug the Jython errors without playing around with it
> > locally. The thing is that the fields (variable) in the IDV are VisAD
> > objects,
> > which means you have to be careful how you mathematically manipulate
> > them. It's not as simple as it is for a numpy array, for example.
> >
> > I recommend taking a look at the following page:
> >
> > https://www.unidata.ucar.edu/software/idv/docs/userguide/tools/JythonLib.html
> >
> > Note that to get a slice of a field in the IDV, you cannot simply index
> > into it as you
> > would a numpy array, but you need to use a Jython method, something like
> >
> > getNthTimeGrid
> >
> > or
> >
> > make2D
> >
> >>
> >> 2) how to load external python module in IDV, such as numpy?
> >> On My Mac machine, I have installed numpy, and I tried to import it with
> >> the absolute path:
> >> import
> >> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy
> >> as np
> >> This cause the syntax Error.
> >
> > So this is perhaps the most critical comment to understand - Jython is not
> > Python,
> > at least as Python is understood by people who actually use Python to get
> > scientific
> > work done :-) Python, like you and I understand, ties into C quite nicely,
> > and in fact
> > is built on C. Python, from python.org, is sometimes called CPython.
> >
> > Jython is basically the core CPython v2.6 api implemented in Java. What
> > that means
> > is that you can write a script using pure python, and there is underlying
> > Java code
> > that understands what that syntax means and how to do that in Java. So,
> > for example,
> > if you type out
> >
> > print "hello, world!"
> >
> > the underlying Java code will see the print statement and execute:
> >
> > System.out.println("hello, world!");
> >
> > Note that Jython isn't something that Unidata came up with, but we did
> > choose to use it because it's easier for non-Java coders to work with.
> >
> > When it comes to all of the awesome modules available outside of the core
> > CPython distribution, such as numpy and scipy, they cannot be used in
> > Jython because they are not pure python libraries (they end up wrapping
> > C and/or FORTRAN code at some point, which Jython does not know what
> > to do with).
> >
> > I really wish Jython could be more, but it's just not there, and given the
> > availability and popularity of other JVM based languages, such as Groovy,
> > I don't ever see it getting close to CPython's capabilities.
> >
> > I hope this helps explain some things; sorry I don't have better news.
> >
> > Cheers,
> >
> > Sean
> >
> >>
> >> I will continue working on solving these two questions, but at the mean
> >> time, if you have any advice, please let me know.  Thanks again.
> >>
> >> Chia-Ying
> >>
> >> > Greetings Chia-Ying!
> >> >
> >> > Yes, I think we've found the problem. I would suggest trying to
> >> spatially
> >> > subset
> >> > in the horizontal directions to get a quick look, and only use the
> >> full
> >> > resolution
> >> > datasets when you have to do so. For more information on subsetting,
> >> > checkout
> >> > the Dealing with Large Datasets topic in our docs:
> >> >
> >> > http://www.unidata.ucar.edu/software/idv/docs/workshop/datadisplays/wrf/InteractiveWRFGridDisplays.html
> >> >
> >> > It's also possible to do the analysis on a single time step and
> >> animate
> >> > outside of the IDV software.
> >> >
> >> > Cheers,
> >> >
> >> > Sean
> >> >
> >> >> Hi Sean,
> >> >>
> >> >> Thanks, Here are the information.
> >> >>
> >> >> 1) loading multiple wrfout files (24) with grid numbers as 601, 400,
> >> 36
> >> >> in
> >> >> x, y, and z directions, the three numbers are:
> >> >> 182/201/7331
> >> >> selecting to display an iso-surface of one 3D variable:
> >> >> 6621/7331/7331
> >> >>
> >> >> 2) loading one wrfout file and selecting display an iso-surface of
> >> one
> >> >> 3D
> >> >> variable:
> >> >> 1085/1874/7331
> >> >> displaying the derived horizontal wind vectors
> >> >> 1997/3917/7331
> >> >>
> >> >> It seems display 3D variables with multiple wrfout files is the main
> >> >> issue. However, even I load only one file, after display two to three
> >> 3D
> >> >> variables, the the maximum amount of memory used becomes very close
> >> to
> >> >> total amount of memory allocated to IDV.
> >> >>
> >> >>
> >> >> Chia-Ying
> >> >>
> >> >> > Greetings Chiaying!
> >> >> >
> >> >> > In the lower left-hand corner of the IDV is a display of the
> >> current
> >> >> time,
> >> >> > in
> >> >> > UTC. If you click that clock, the display will change and show you
> >> >> > information
> >> >> > about your current memory usage. There are three numbers, seperated
> >> by
> >> >> > slashes, and they describe the follow: the first is the amount of
> >> >> memory
> >> >> > currently used by the IDV, the second is the maximum amount of
> >> memory
> >> >> used
> >> >> > at any point during your IDV session, and the third is the total
> >> >> amount of
> >> >> > memory allocated to the IDV. Would you mind loading the display you
> >> >> > described
> >> >> > and sending me those three numbers? If the second number equals the
> >> >> third,
> >> >> > the IDV can become quite unresponsive.
> >> >> >
> >> >> > Also, what is the grid spacing of the WRF output you are viewing?
> >> >> >
> >> >> > Thanks!
> >> >> >
> >> >> > Sean
> >> >> >
> >> >> >> Hi Sean,
> >> >> >>
> >> >> >> I have another question about loading multiple wrfout files and
> >> the
> >> >> IDV
> >> >> >> performance.  While loading 3D data, it takes IDV a noticeable
> >> time
> >> >> to
> >> >> >> respond to every commend. Each wrfout I am currently using is
> >> about
> >> >> 500
> >> >> >> M,
> >> >> >> and I load 36 of them.  I display only the first 15 layers (in
> >> total,
> >> >> >> there are 36 layers for 3D fields).  IDV became very slow when I
> >> >> tried
> >> >> >> to
> >> >> >> display point volume... which took 5 minutes or longer every time
> >> >> when I
> >> >> >> try  to adjust color bar, change view angle, or change vertical
> >> >> range,
> >> >> >> etc.
> >> >> >> we have choice to put data on local machine (MAC) or on the
> >> cluster
> >> >> >> (load
> >> >> >> it trough internet).  I load it through the local machine, which
> >> >> should
> >> >> >> be
> >> >> >> a more efficient way.
> >> >> >>
> >> >> >> The system setting I set for IDV is 7000M memory for IDV (I have
> >> 8G
> >> >> >> memory), permGen size is 128 M, write grib index in disk Cache,
> >> data
> >> >> >> cache
> >> >> >> Memory Percent 50, unlimited image size, enable geometry by
> >> >> reference,
> >> >> >> and
> >> >> >> enable access to image data by reference.  I wondering if there is
> >> >> >> anyway
> >> >> >> I can increase the performance of IDV.
> >> >> >>
> >> >> >> Also, I sent you and Tom another email about displaying observed
> >> >> >> trajectories from drifter data yesterday, but I forgot to keep the
> >> >> title
> >> >> >> "
> >> >> >> [IDV #JHS-477170]", and not sure if the email will go to you (and
> >> >> Tom)
> >> >> >> or
> >> >> >> not.  The automatic response assigned me a number "[IDV
> >> #TRI-715395]:
> >> >> >> display drifter trajectory files in IDV".
> >> >> >>
> >> >> >> Thanks.
> >> >> >>
> >> >> >> -Chiaying
> >> >> >>
> >> >> >>
> >> >> >> > Hi Sean,
> >> >> >> >
> >> >> >> > Thanks for the reply.  I can display/animate all the three model
> >> >> >> > components without problem. Thanks again.
> >> >> >> >
> >> >> >> >
> >> >> >> > -Chiaying
> >> >> >> >> Hi Chiaying!
> >> >> >> >>
> >> >> >> >>> Hi Sean,
> >> >> >> >>>
> >> >> >> >>> Thanks for the answer, I just tried and umwmout.ncml works
> >> well
> >> >> now
> >> >> >> >>> after
> >> >> >> >>> the changes!
> >> >> >> >>
> >> >> >> >> Great, I'm glad it worked! See below for the solution to the
> >> HYCOM
> >> >> >> >> output.
> >> >> >> >>
> >> >> >> >>>
> >> >> >> >>> I have another question about loading ocean model (HYCOM)
> >> outputs
> >> >> >> (also
> >> >> >> >>> NetCDF file) into IDV and loop them.
> >> >> >> >>>
> >> >> >> >>> In the HYCOM output, there is time-dimension for all the
> >> >> variables.
> >> >> >> So,
> >> >> >> >>> I
> >> >> >> >>> first tried to load them into IDV with "Aggregate Grids by
> >> Time"
> >> >> >> >>> option,
> >> >> >> >>> and then I got an error message saying  "no coordinate
> >> variable
> >> >> for
> >> >> >> agg
> >> >> >> >>> dimension = time."  My first guess was that the dimension name
> >> >> for
> >> >> >> time
> >> >> >> >>> in
> >> >> >> >>> HYCOM output is "MT" , and IDV might not able to recognize it.
> >> >> >> another
> >> >> >> >>> possible issue is that MT is written as day since  1900-12-31
> >> >> (see
> >> >> >> >>> below)
> >> >> >> >>> dimensions:
> >> >> >> >>> MT = UNLIMITED ; // (1 currently)
> >> >> >> >>> Latitude = 1000 ;
> >> >> >> >>> Longitude = 1000 ;
> >> >> >> >>> Depth = 40 ;o
> >> >> >> >>> variables:
> >> >> >> >>> double MT(MT) ;
> >> >> >> >>> MT:long_name = "time" ;
> >> >> >> >>> MT:units = "days since 1900-12-31 00:00:00" ;
> >> >> >> >>> MT:calendar = "gregorian" ;
> >> >> >> >>> MT:axis = "T" ;
> >> >> >> >>> float salinity(MT, Depth, Latitude, Longitude) ;
> >> >> >> >>> salinity:coordinates = "Date" ;
> >> >> >> >>> salinity:standard_name = "sea_water_salinity" ;
> >> >> >> >>> salinity:units = "psu" ;
> >> >> >> >>> salinity:_FillValue = 1.267651e+30f ;
> >> >> >> >>> salinity:valid_range = 9.70682f, 38.86856f ;
> >> >> >> >>> salinity:long_name = " salinity [20.1H]" ;
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> Then, I tried to write a .ncml file to see if I can aggregate
> >> >> HYCOM
> >> >> >> >>> output
> >> >> >> >>> as what you did for umwm. No luck in this either (with an
> >> error
> >> >> >> message
> >> >> >> >>> saying no griddata found for archv.ncml).  Not sure what
> >> causes
> >> >> the
> >> >> >> >>> problem.
> >> >> >> >>>
> >> >> >> >>> Can you give me some hint how I should solve this issue?  I
> >> put
> >> >> the
> >> >> >> >>> hycom
> >> >> >> >>> data and my archv.ncml online at:
> >> >> >> >>> http://orca.rsmas.miami.edu/~chiaying/IDV/
> >> >> >> >>
> >> >> >> >> Your ncml is very close! The issue is that a time dimension and
> >> >> >> variable
> >> >> >> >> already
> >> >> >> >> exists, but it's not called "time", which is what the IDV
> >> expects
> >> >> >> when
> >> >> >> >> aggregating
> >> >> >> >> (this is a deficiency in the IDV that needs to be addressed).
> >> >> Since
> >> >> >> you
> >> >> >> >> already have
> >> >> >> >> time in your files, you can simply do a joinExisting
> >> aggregation
> >> >> in
> >> >> >> >> ncml,
> >> >> >> >> rather
> >> >> >> >> than a joinNew, and there is no need to create a new time
> >> variable
> >> >> >> from
> >> >> >> >> the file
> >> >> >> >> name. In order to do this, make the following changes to your
> >> ncml
> >> >> >> file.
> >> >> >> >>
> >> >> >> >> Change the contents of the aggregation tag to:
> >> >> >> >>
> >> >> >> >> aggregation dimName="MT" type="joinExisting"
> >> >> >> >>
> >> >> >> >> Change the contents of the scan tag to:
> >> >> >> >>
> >> >> >> >> scan location="/home/disk/manta9/chiaying/2012_isaac/archvnc/"
> >> >> >> >> suffix=".nc" subdirs="false"
> >> >> >> >>
> >> >> >> >> By the way, the units on the MT variable are perfectly
> >> acceptable,
> >> >> so
> >> >> >> no
> >> >> >> >> issues
> >> >> >> >> there :-)
> >> >> >> >>
> >> >> >> >> If you are interested in learning more about ncml, check out
> >> the
> >> >> ncml
> >> >> >> >> tutorial
> >> >> >> >> and cookbook:
> >> >> >> >>
> >> >> >> >> http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html
> >> >> >> >>
> >> >> >> >> http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Cookbook.html
> >> >> >> >>
> >> >> >> >> Let me know if you have any questions!
> >> >> >> >>
> >> >> >> >> Cheers!
> >> >> >> >>
> >> >> >> >> Sean
> >> >> >> >>
> >> >> >> >>>
> >> >> >> >>> Thanks for your help.
> >> >> >> >>>
> >> >> >> >>> Chiaying
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> > Greetings Chiaying!
> >> >> >> >>> >
> >> >> >> >>> > My name is Sean, and I put together the README file and the
> >> >> ncml
> >> >> >> >>> file.
> >> >> >> >>> >
> >> >> >> >>> > First, I would rename the ncml file from umwmout.nml.xml to
> >> >> >> >>> > umwmout.ncml.
> >> >> >> >>> >
> >> >> >> >>> > Then, I would check the following:
> >> >> >> >>> >
> >> >> >> >>> > In the scan tag, check to make sure the dateFormatMark
> >> matches
> >> >> >> >>> > how the date is expressed in your file name. For example,
> >> the
> >> >> >> error
> >> >> >> >>> > message you passed along indicates that your files are named
> >> >> like:
> >> >> >> >>> >
> >> >> >> >>> > umwmout_2012-08-26_12:00:00.nc
> >> >> >> >>> >
> >> >> >> >>> > which means the dateFormatMark should be
> >> >> >> >>> >
> >> >> >> >>> > umwmout_#yyyy-MM-dd_HH:mm:ss"
> >> >> >> >>> >
> >> >> >> >>> > The ncml file I sent used a slightly different
> >> dateFormatMark.
> >> >> >> >>> >
> >> >> >> >>> > If everything works, then loading the ncml file using "I'm
> >> >> feeling
> >> >> >> >>> > lucky" is all you will need to do (that is, you won't need
> >> to
> >> >> do
> >> >> >> >>> > step three as you described in your previous email).
> >> >> >> >>> >
> >> >> >> >>> > Please let me know if it works with these changes.
> >> >> >> >>> >
> >> >> >> >>> > Cheers!
> >> >> >> >>> >
> >> >> >> >>> > Sean
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> >> Hi Tom,
> >> >> >> >>> >>
> >> >> >> >>> >> Thanks for the files.  I have tried to use umwmout.nml.xml
> >> >> file.
> >> >> >> >>> >> However,
> >> >> >> >>> >> I got an error message wile trying loading umwmout*.nc
> >> data.
> >> >> Here
> >> >> >> is
> >> >> >> >>> >> what
> >> >> >> >>> >> i did:
> >> >> >> >>> >>
> >> >> >> >>> >> 1) I put umwmout.nml.xml in the same folder as where the
> >> >> >> unwmout*.nc
> >> >> >> >>> >> are.
> >> >> >> >>> >> 2)  load umwmout.nml.xml with "I'm feeling lucky" data
> >> choice
> >> >> (I
> >> >> >> >>> have
> >> >> >> >>> >> changed the path)
> >> >> >> >>> >> 3)  load umwmout*.nc file with "Aggregate Grids by Time"
> >> >> option,
> >> >> >> and
> >> >> >> >>> I
> >> >> >> >>> >> got
> >> >> >> >>> >> an error message:
> >> >> >> >>> >>
> >> >> >> >>> >> There was an error loading the data:
> >> >> >> >>> >> Error creating data source:aggregated.netcdf.grid with:
> >> >> >> >>> >> [/home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_12:00:00.nc,
> >> >> >> >>> >> /home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_13:00:00.nc,
> >> >> >> >>> >> /home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_14:00:00.nc]
> >> >> >> >>> >> Grid data source failed making data set:
> >> >> >> >>> >> /Users/chiayinglee/.unidata/idv/DefaultIdv/tmp/multigrid_b3ad0ca8-652c-4a8e-bd66-093cd3468f3b0.ncml
> >> >> >> >>> >> AggregationExisting: no coordinate variable for agg
> >> >> >> >>> >> dimension= time
> >> >> >> >>> >>
> >> >> >> >>> >> It seems that  IDV still can not recognize the time.  Am I
> >> >> >> missing
> >> >> >> >>> >> something? Thanks.
> >> >> >> >>> >>
> >> >> >> >>> >> Chiaying
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >> > Hi Shuyi,
> >> >> >> >>> >> >
> >> >> >> >>> >> > re:
> >> >> >> >>> >> >> Thank you so much for your help on IDV!  My group is
> >> >> learning
> >> >> >> to
> >> >> >> >>> use
> >> >> >> >>> >> >> IDV for our coupled model output. I really like the
> >> feature
> >> >> >> that
> >> >> >> >>> we
> >> >> >> >>> >> can
> >> >> >> >>> >> >> display/analyze the atmos/ocean/wave fields together in
> >> >> IDV.
> >> >> >> >>> >> >> Milan and Chiaying (CC'd) may email you for some
> >> questions
> >> >> as
> >> >> >> we
> >> >> >> >>> >> >> are learning/experimenting with various functions.
> >> >> >> >>> >> >
> >> >> >> >>> >> > This sounds good.  Did you pass along the README file
> >> with
> >> >> >> >>> comments
> >> >> >> >>> on
> >> >> >> >>> >> > how your coupled model output could be better structured
> >> for
> >> >> >> use
> >> >> >> >>> by
> >> >> >> >>> >> > the IDV and other applications that provide support for
> >> data
> >> >> in
> >> >> >> >>> netCDF
> >> >> >> >>> >> > format?
> >> >> >> >>> >> >
> >> >> >> >>> >> > Tom
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >
> >> >> >> >>> > Ticket Details
> >> >> >> >>> > ===================
> >> >> >> >>> > Ticket ID: JHS-477170
> >> >> >> >>> > Department: Support IDV
> >> >> >> >>> > Priority: Normal
> >> >> >> >>> > Status: Open
> >> >> >> >>> >
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>
> >> >> >> >> Ticket Details
> >> >> >> >> ===================
> >> >> >> >> Ticket ID: JHS-477170
> >> >> >> >> Department: Support IDV
> >> >> >> >> Priority: Normal
> >> >> >> >> Status: Open
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >
> >> >> > Ticket Details
> >> >> > ===================
> >> >> > Ticket ID: JHS-477170
> >> >> > Department: Support IDV
> >> >> > Priority: Normal
> >> >> > Status: Open
> >> >> >
> >> >>
> >> >>
> >> >
> >> > Ticket Details
> >> > ===================
> >> > Ticket ID: JHS-477170
> >> > Department: Support IDV
> >> > Priority: Normal
> >> > Status: Open
> >> >
> >>
> >>
> >
> > Ticket Details
> > ===================
> > Ticket ID: JHS-477170
> > Department: Support IDV
> > Priority: Normal
> > Status: Open
> >
> 
> 

Ticket Details
===================
Ticket ID: JHS-477170
Department: Support IDV
Priority: Critical
Status: Open