[IDV #JHS-477170]: 20130607: WRF like model aggregation in the IDV

Greetings Chia-Ying!

Yes, I think we've found the problem. I would suggest trying to spatially subset
in the horizontal directions to get a quick look, and only use the full 
resolution
datasets when you have to do so. For more information on subsetting, checkout
the Dealing with Large Datasets topic in our docs:

http://www.unidata.ucar.edu/software/idv/docs/workshop/datadisplays/wrf/InteractiveWRFGridDisplays.html

It's also possible to do the analysis on a single time step and animate 
outside of the IDV software.

Cheers,

Sean

> Hi Sean,
> 
> Thanks, Here are the information.
> 
> 1) loading multiple wrfout files (24) with grid numbers as 601, 400, 36 in
> x, y, and z directions, the three numbers are:
> 182/201/7331
> selecting to display an iso-surface of one 3D variable:
> 6621/7331/7331
> 
> 2) loading one wrfout file and selecting display an iso-surface of one 3D
> variable:
> 1085/1874/7331
> displaying the derived horizontal wind vectors
> 1997/3917/7331
> 
> It seems display 3D variables with multiple wrfout files is the main
> issue. However, even I load only one file, after display two to three 3D
> variables, the the maximum amount of memory used becomes very close to
> total amount of memory allocated to IDV.
> 
> 
> Chia-Ying
> 
> > Greetings Chiaying!
> >
> > In the lower left-hand corner of the IDV is a display of the current time,
> > in
> > UTC. If you click that clock, the display will change and show you
> > information
> > about your current memory usage. There are three numbers, seperated by
> > slashes, and they describe the follow: the first is the amount of memory
> > currently used by the IDV, the second is the maximum amount of memory used
> > at any point during your IDV session, and the third is the total amount of
> > memory allocated to the IDV. Would you mind loading the display you
> > described
> > and sending me those three numbers? If the second number equals the third,
> > the IDV can become quite unresponsive.
> >
> > Also, what is the grid spacing of the WRF output you are viewing?
> >
> > Thanks!
> >
> > Sean
> >
> >> Hi Sean,
> >>
> >> I have another question about loading multiple wrfout files and the IDV
> >> performance.  While loading 3D data, it takes IDV a noticeable time to
> >> respond to every commend. Each wrfout I am currently using is about 500
> >> M,
> >> and I load 36 of them.  I display only the first 15 layers (in total,
> >> there are 36 layers for 3D fields).  IDV became very slow when I tried
> >> to
> >> display point volume... which took 5 minutes or longer every time when I
> >> try  to adjust color bar, change view angle, or change vertical range,
> >> etc.
> >> we have choice to put data on local machine (MAC) or on the cluster
> >> (load
> >> it trough internet).  I load it through the local machine, which should
> >> be
> >> a more efficient way.
> >>
> >> The system setting I set for IDV is 7000M memory for IDV (I have 8G
> >> memory), permGen size is 128 M, write grib index in disk Cache, data
> >> cache
> >> Memory Percent 50, unlimited image size, enable geometry by reference,
> >> and
> >> enable access to image data by reference.  I wondering if there is
> >> anyway
> >> I can increase the performance of IDV.
> >>
> >> Also, I sent you and Tom another email about displaying observed
> >> trajectories from drifter data yesterday, but I forgot to keep the title
> >> "
> >> [IDV #JHS-477170]", and not sure if the email will go to you (and Tom)
> >> or
> >> not.  The automatic response assigned me a number "[IDV #TRI-715395]:
> >> display drifter trajectory files in IDV".
> >>
> >> Thanks.
> >>
> >> -Chiaying
> >>
> >>
> >> > Hi Sean,
> >> >
> >> > Thanks for the reply.  I can display/animate all the three model
> >> > components without problem. Thanks again.
> >> >
> >> >
> >> > -Chiaying
> >> >> Hi Chiaying!
> >> >>
> >> >>> Hi Sean,
> >> >>>
> >> >>> Thanks for the answer, I just tried and umwmout.ncml works well now
> >> >>> after
> >> >>> the changes!
> >> >>
> >> >> Great, I'm glad it worked! See below for the solution to the HYCOM
> >> >> output.
> >> >>
> >> >>>
> >> >>> I have another question about loading ocean model (HYCOM) outputs
> >> (also
> >> >>> NetCDF file) into IDV and loop them.
> >> >>>
> >> >>> In the HYCOM output, there is time-dimension for all the variables.
> >> So,
> >> >>> I
> >> >>> first tried to load them into IDV with "Aggregate Grids by Time"
> >> >>> option,
> >> >>> and then I got an error message saying  "no coordinate variable for
> >> agg
> >> >>> dimension = time."  My first guess was that the dimension name for
> >> time
> >> >>> in
> >> >>> HYCOM output is "MT" , and IDV might not able to recognize it.
> >> another
> >> >>> possible issue is that MT is written as day since  1900-12-31 (see
> >> >>> below)
> >> >>> dimensions:
> >> >>> MT = UNLIMITED ; // (1 currently)
> >> >>> Latitude = 1000 ;
> >> >>> Longitude = 1000 ;
> >> >>> Depth = 40 ;o
> >> >>> variables:
> >> >>> double MT(MT) ;
> >> >>> MT:long_name = "time" ;
> >> >>> MT:units = "days since 1900-12-31 00:00:00" ;
> >> >>> MT:calendar = "gregorian" ;
> >> >>> MT:axis = "T" ;
> >> >>> float salinity(MT, Depth, Latitude, Longitude) ;
> >> >>> salinity:coordinates = "Date" ;
> >> >>> salinity:standard_name = "sea_water_salinity" ;
> >> >>> salinity:units = "psu" ;
> >> >>> salinity:_FillValue = 1.267651e+30f ;
> >> >>> salinity:valid_range = 9.70682f, 38.86856f ;
> >> >>> salinity:long_name = " salinity [20.1H]" ;
> >> >>>
> >> >>>
> >> >>> Then, I tried to write a .ncml file to see if I can aggregate HYCOM
> >> >>> output
> >> >>> as what you did for umwm. No luck in this either (with an error
> >> message
> >> >>> saying no griddata found for archv.ncml).  Not sure what causes the
> >> >>> problem.
> >> >>>
> >> >>> Can you give me some hint how I should solve this issue?  I put the
> >> >>> hycom
> >> >>> data and my archv.ncml online at:
> >> >>> http://orca.rsmas.miami.edu/~chiaying/IDV/
> >> >>
> >> >> Your ncml is very close! The issue is that a time dimension and
> >> variable
> >> >> already
> >> >> exists, but it's not called "time", which is what the IDV expects
> >> when
> >> >> aggregating
> >> >> (this is a deficiency in the IDV that needs to be addressed). Since
> >> you
> >> >> already have
> >> >> time in your files, you can simply do a joinExisting aggregation in
> >> >> ncml,
> >> >> rather
> >> >> than a joinNew, and there is no need to create a new time variable
> >> from
> >> >> the file
> >> >> name. In order to do this, make the following changes to your ncml
> >> file.
> >> >>
> >> >> Change the contents of the aggregation tag to:
> >> >>
> >> >> aggregation dimName="MT" type="joinExisting"
> >> >>
> >> >> Change the contents of the scan tag to:
> >> >>
> >> >> scan location="/home/disk/manta9/chiaying/2012_isaac/archvnc/"
> >> >> suffix=".nc" subdirs="false"
> >> >>
> >> >> By the way, the units on the MT variable are perfectly acceptable, so
> >> no
> >> >> issues
> >> >> there :-)
> >> >>
> >> >> If you are interested in learning more about ncml, check out the ncml
> >> >> tutorial
> >> >> and cookbook:
> >> >>
> >> >> http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html
> >> >>
> >> >> http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Cookbook.html
> >> >>
> >> >> Let me know if you have any questions!
> >> >>
> >> >> Cheers!
> >> >>
> >> >> Sean
> >> >>
> >> >>>
> >> >>> Thanks for your help.
> >> >>>
> >> >>> Chiaying
> >> >>>
> >> >>>
> >> >>>
> >> >>> > Greetings Chiaying!
> >> >>> >
> >> >>> > My name is Sean, and I put together the README file and the ncml
> >> >>> file.
> >> >>> >
> >> >>> > First, I would rename the ncml file from umwmout.nml.xml to
> >> >>> > umwmout.ncml.
> >> >>> >
> >> >>> > Then, I would check the following:
> >> >>> >
> >> >>> > In the scan tag, check to make sure the dateFormatMark matches
> >> >>> > how the date is expressed in your file name. For example, the
> >> error
> >> >>> > message you passed along indicates that your files are named like:
> >> >>> >
> >> >>> > umwmout_2012-08-26_12:00:00.nc
> >> >>> >
> >> >>> > which means the dateFormatMark should be
> >> >>> >
> >> >>> > umwmout_#yyyy-MM-dd_HH:mm:ss"
> >> >>> >
> >> >>> > The ncml file I sent used a slightly different dateFormatMark.
> >> >>> >
> >> >>> > If everything works, then loading the ncml file using "I'm feeling
> >> >>> > lucky" is all you will need to do (that is, you won't need to do
> >> >>> > step three as you described in your previous email).
> >> >>> >
> >> >>> > Please let me know if it works with these changes.
> >> >>> >
> >> >>> > Cheers!
> >> >>> >
> >> >>> > Sean
> >> >>> >
> >> >>> >
> >> >>> >> Hi Tom,
> >> >>> >>
> >> >>> >> Thanks for the files.  I have tried to use umwmout.nml.xml file.
> >> >>> >> However,
> >> >>> >> I got an error message wile trying loading umwmout*.nc data. Here
> >> is
> >> >>> >> what
> >> >>> >> i did:
> >> >>> >>
> >> >>> >> 1) I put umwmout.nml.xml in the same folder as where the
> >> unwmout*.nc
> >> >>> >> are.
> >> >>> >> 2)  load umwmout.nml.xml with "I'm feeling lucky" data choice (I
> >> >>> have
> >> >>> >> changed the path)
> >> >>> >> 3)  load umwmout*.nc file with "Aggregate Grids by Time" option,
> >> and
> >> >>> I
> >> >>> >> got
> >> >>> >> an error message:
> >> >>> >>
> >> >>> >> There was an error loading the data:
> >> >>> >> Error creating data source:aggregated.netcdf.grid with:
> >> >>> >> [/home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_12:00:00.nc,
> >> >>> >> /home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_13:00:00.nc,
> >> >>> >> /home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_14:00:00.nc]
> >> >>> >> Grid data source failed making data set:
> >> >>> >> /Users/chiayinglee/.unidata/idv/DefaultIdv/tmp/multigrid_b3ad0ca8-652c-4a8e-bd66-093cd3468f3b0.ncml
> >> >>> >> AggregationExisting: no coordinate variable for agg
> >> >>> >> dimension= time
> >> >>> >>
> >> >>> >> It seems that  IDV still can not recognize the time.  Am I
> >> missing
> >> >>> >> something? Thanks.
> >> >>> >>
> >> >>> >> Chiaying
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> > Hi Shuyi,
> >> >>> >> >
> >> >>> >> > re:
> >> >>> >> >> Thank you so much for your help on IDV!  My group is learning
> >> to
> >> >>> use
> >> >>> >> >> IDV for our coupled model output. I really like the feature
> >> that
> >> >>> we
> >> >>> >> can
> >> >>> >> >> display/analyze the atmos/ocean/wave fields together in IDV.
> >> >>> >> >> Milan and Chiaying (CC'd) may email you for some questions as
> >> we
> >> >>> >> >> are learning/experimenting with various functions.
> >> >>> >> >
> >> >>> >> > This sounds good.  Did you pass along the README file with
> >> >>> comments
> >> >>> on
> >> >>> >> > how your coupled model output could be better structured for
> >> use
> >> >>> by
> >> >>> >> > the IDV and other applications that provide support for data in
> >> >>> netCDF
> >> >>> >> > format?
> >> >>> >> >
> >> >>> >> > Tom
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>> > Ticket Details
> >> >>> > ===================
> >> >>> > Ticket ID: JHS-477170
> >> >>> > Department: Support IDV
> >> >>> > Priority: Normal
> >> >>> > Status: Open
> >> >>> >
> >> >>>
> >> >>>
> >> >>
> >> >> Ticket Details
> >> >> ===================
> >> >> Ticket ID: JHS-477170
> >> >> Department: Support IDV
> >> >> Priority: Normal
> >> >> Status: Open
> >> >>
> >> >
> >> >
> >>
> >>
> >
> > Ticket Details
> > ===================
> > Ticket ID: JHS-477170
> > Department: Support IDV
> > Priority: Normal
> > Status: Open
> >
> 
> 

Ticket Details
===================
Ticket ID: JHS-477170
Department: Support IDV
Priority: Normal
Status: Open