[IDV #BDO-300797]: catalog choking on folder of OK .ncml files

Brian,

Are you on the latest RAMADDA now? (I was not sure since it seems like there are
some offline discussions going.) As I understand it, there was some code checked
in last summer that inadvertently cracked open those NCML files. That should be
fixed now with the latest RAMADDA.

Best,

Unidata IDV Support

> I tried updating RAMADDA to 1.7b, tomcat failed to come back up with sudo 
> service repos restart,
> and after a half hour I reverted to 1.6. Jeff is engaged by email in helping 
> me figure out why.
> e offli
> 
> Trying now, it works fine. Both the Albany and the ESRL union aggregation 
> .ncmls are quickly exposed in the IDV catalog.
> But that may be because I am all the way back at 1.6, not because you updated 
> your 1.7b.
> 
> 
> Confusing at what level the problem is/was.
> 
> 
> Brian
> 
> 
> 
> 
> 
> On Dec 8, 2014, at 10:25 PM, Tyle, Kevin R wrote:
> 
> Folks,
> 
> I've updated the version of RAMADDA on 
> ramadda.atmos.albany.edu<http://ramadda.atmos.albany.edu> to include today's 
> fix.  Brian, check the NcML files and see if things look better.
> 
> Cheers,
> 
> Kevin
> 
> _____________________________________________
> Kevin Tyle, Systems Administrator
> Dept. of Atmospheric & Environmental Sciences
> University at Albany
> Earth Science 235, 1400 Washington Avenue
> Albany, NY 12222
> Email: address@hidden<mailto:address@hidden>
> Phone: 518-442-4578
> _____________________________________________
> 
> ________________________________________
> From: Don Murray (NOAA Affiliate) <address@hidden<mailto:address@hidden>>
> Sent: Monday, December 8, 2014 11:19 AM
> To: Brian Mapes; <address@hidden<mailto:address@hidden>>
> Cc: <address@hidden<mailto:address@hidden>>; 
> address@hidden<mailto:address@hidden>; Tyle, Kevin R
> Subject: Re: [IDV #BDO-300797]: catalog choking on folder of OK .ncml files
> 
> All-
> 
> Julien found that the problem was with RAMADDA due to a change I made
> back in May.  I've checked in a fix and there will be a new release on
> SourceForge soon that will include this.
> 
> Don
> 
> On 12/3/14 5:40 PM, Brian Mapes wrote:
> Thanks!
> 
> 
> Any plans for time aggregations would obviate all this, and be hugely 
> preferable.
> Is that going to sweep the field anytime soon?
> 
> Realistically, I suppose MERRA on its GDS will always be my main go-to 
> reanalysis,
> everything else is much harder and worse to access.
> 
> In fact I’m not sure how much to further/ continue to invest in “making good” 
> the
> PSD datasets “as they stand.” (year by year, variable by variable).
> I do it mostly out of minor obsession.
> Actually, with my synoptic class ending next week, it may all go on a back 
> burner
> for another year (and the last time I know this .ncml all worked was last 
> year about now).
> 
> 
> But I hope these pow wos are useful conversations at many levels, and for your
> plans on many timescales. Thanks for keeping me apprised.
> 
> 
> Brian
> 
> 
> 
> 
> On Dec 3, 2014, at 6:31 PM, Unidata IDV Support 
> <address@hidden<mailto:address@hidden>> wrote:
> 
> Brian,
> 
> Tom, Yuan and I had a long pow wow about this topic today. We brought your 
> NCML
> data over to Unidata machines. Your analysis coupled with our experiments 
> ruled
> out a number of different possibilities (e.g., web servers limiting traffic,
> networking issues).  BTW, we see the same behavior with older versions of the
> IDV.
> 
> We are now working with a few different theories.
> 
> 1/ The IDV needs to make smarter requests to RAMADDA in order to not crack 
> open
> each individual NCML aggregation in a directory.
> 
> 2/ Something changed within RAMADDA and it now cracks open more of the dataset
> than we would like.
> 
> We will keep you posted.
> 
> Best,
> 
> Unidata IDV Support
> 
> It is unworkable that IDV has to parse every .ncml file in a directory
> before exposing them in Catalog view.
> 
> I have the same problem with CFSR at Albany (see screen shot),
> so the issue is not specific the PSD datasets.
> It is a change in IDV or its libraries, since this used to work.
> 
> 
> 
> I did a lot of work to build these .ncml aggregations.
> It's a shame they are now useless, or downright dangerous to the session
> (the IDV progress indicator bar remains busy, although other tasks can be 
> done).
> 
> What's the solution?
> Will this always be the way Catalog view operates?
> If so I need to refolderize my aggregations, one or two .ncml per folder, I 
> guess.
> 
> [cid:CBE3C095-1917-4644-9718-7C4A267CE31A@wireless.miami.edu]
> 
> 
> 
> On Nov 24, 2014, at 10:51 AM, Unidata IDV Support wrote:
> 
> Brian,
> 
> The change may not be related to Unidata.  As far as we could tell, the
> "slowness" is related to the URLs aggregated in the NCML.  Bear in that these
> NCML files are hitting the (NCEP?) web server quite a bit. It could that NCEP
> set up some mechanisms to limit the rate of URL access on their web 
> servers(???)
> (Especially after the cyber attacks that hit the NWS recently.)
> 
> Best,
> 
> Unidata IDV Support
> 
> Let me clarify that it used to work.
> What has changed?
> 
> Last year about this time, for example, my class used these same server side 
> ncml aggregations with no trouble. This is not experimental new 
> functionality, it is a workhorse that fell down. Always disappointing when 
> the March of progress or at least of workable fixes goes backward.
> 
> What has changed?
> 
> Thanks for your help.
> Brian
> 
> Brian Mapes
> This message was typed on a cell phone, please forgive its terseness and any 
> errors.
> 
> On Nov 21, 2014, at 5:38 PM, Unidata IDV Support 
> <address@hidden<mailto:address@hidden><mailto:address@hidden>> wrote:
> 
> Hi Brian,
> 
> re: IDV Catalog listing of directory with lots of NCML files that reference 
> data
> through URLs
> 
> Yes I did this and it never came back.
> Help ??
> 
> Julien and I proved that the catalog view in the IDV will _eventually_
> show all of the NCML files.  We did this by:
> 
> - create a new server side view of a single NCML file
> 
> This listing is reasonably fast.
> 
> - create a server side view of 10 NCML files
> 
> This listing is slow, but not slow enough to look like the IDV is
> hung.
> 
> - full set
> 
> We didn't bother to wait for this to complete, but we are convinced
> that it eventually will.
> 
> What is going on?
> 
> Evidently, netCDF Java is actually opening the dataset referenced in each
> URL named in the NCML files.  It seems to take between 4 and 5 seconds to
> get the information from the 7 URLs referenced in each NCML file.  If the
> wait time scales linearly, then it should take 335 (5 x 67) seconds to 
> generate
> the list for all of the NCML files you have in the NCEP1 6 hourly pressure 
> level
> data server side set of files.
> 
> Can this be sped up?
> 
> We don't know.  We'll have to run more tests to determine if the slowness
> is really in the netCDF Java code or simply a slowness in the ESRL web
> server's response to the query being made.
> 
> Sorry we didn't have better news!
> 
> Cheers,
> 
> Tom
> --
> ****************************************************************************
> Unidata User Support                                    UCAR Unidata Program
> (303) 497-8642                                                 P.O. Box 3000
> address@hidden<mailto:address@hidden><mailto:address@hidden>                  
>                  Boulder, CO 80307
> ----------------------------------------------------------------------------
> Unidata HomePage                       http://www.unidata.ucar.edu
> ****************************************************************************
> 
> 
> Ticket Details
> ===================
> Ticket ID: BDO-300797
> Department: Support IDV
> Priority: Normal
> Status: Closed
> 
> 
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: BDO-300797
> Department: Support IDV
> Priority: Normal
> Status: Closed
> 
> 
> Brian Mapes
> address@hidden<mailto:address@hidden><mailto:address@hidden>
> 
> 
> 
> 
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: BDO-300797
> Department: Support IDV
> Priority: Normal
> Status: Open
> 
> 
> *********************************************
> Brian Mapes, Professor
> Department of Atmospheric Sciences
> Meteorology and Physical Oceanography Program
> RSMAS, University of Miami
> 4600 Rickenbacker Causeway
> Miami, FL 33149-1098
> 
> phone: (305) 421-4275
> fax: (305) 421-4696
> email: address@hidden<mailto:address@hidden>
> Web: http://www.rsmas.miami.edu/users/bmapes/
> **********************************************
> 
> 
> 
> 
> 
> --
> Don Murray
> NOAA/ESRL/PSD and CU-CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/people/don.murray/
> 
> Brian Mapes
> address@hidden<mailto:address@hidden>
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: BDO-300797
Department: Support IDV
Priority: Urgent
Status: Closed