[IDV #BDO-300797]: catalog choking on folder of OK .ncml files

Brian,

Tom, Yuan and I had a long pow wow about this topic today. We brought your NCML
data over to Unidata machines. Your analysis coupled with our experiments ruled
out a number of different possibilities (e.g., web servers limiting traffic,
networking issues).  BTW, we see the same behavior with older versions of the
IDV.

We are now working with a few different theories.

1/ The IDV needs to make smarter requests to RAMADDA in order to not crack open
each individual NCML aggregation in a directory.

2/ Something changed within RAMADDA and it now cracks open more of the dataset
than we would like.

We will keep you posted.

Best,

Unidata IDV Support

> It is unworkable that IDV has to parse every .ncml file in a directory
> before exposing them in Catalog view.
> 
> I have the same problem with CFSR at Albany (see screen shot),
> so the issue is not specific the PSD datasets.
> It is a change in IDV or its libraries, since this used to work.
> 
> 
> 
> I did a lot of work to build these .ncml aggregations.
> It's a shame they are now useless, or downright dangerous to the session
> (the IDV progress indicator bar remains busy, although other tasks can be 
> done).
> 
> What's the solution?
> Will this always be the way Catalog view operates?
> If so I need to refolderize my aggregations, one or two .ncml per folder, I 
> guess.
> 
> [cid:CBE3C095-1917-4644-9718-7C4A267CE31A@wireless.miami.edu]
> 
> 
> 
> On Nov 24, 2014, at 10:51 AM, Unidata IDV Support wrote:
> 
> Brian,
> 
> The change may not be related to Unidata.  As far as we could tell, the
> "slowness" is related to the URLs aggregated in the NCML.  Bear in that these
> NCML files are hitting the (NCEP?) web server quite a bit. It could that NCEP
> set up some mechanisms to limit the rate of URL access on their web 
> servers(???)
> (Especially after the cyber attacks that hit the NWS recently.)
> 
> Best,
> 
> Unidata IDV Support
> 
> Let me clarify that it used to work.
> What has changed?
> 
> Last year about this time, for example, my class used these same server side 
> ncml aggregations with no trouble. This is not experimental new 
> functionality, it is a workhorse that fell down. Always disappointing when 
> the March of progress or at least of workable fixes goes backward.
> 
> What has changed?
> 
> Thanks for your help.
> Brian
> 
> Brian Mapes
> This message was typed on a cell phone, please forgive its terseness and any 
> errors.
> 
> On Nov 21, 2014, at 5:38 PM, Unidata IDV Support 
> <address@hidden<mailto:address@hidden>> wrote:
> 
> Hi Brian,
> 
> re: IDV Catalog listing of directory with lots of NCML files that reference 
> data
> through URLs
> 
> Yes I did this and it never came back.
> Help ??
> 
> Julien and I proved that the catalog view in the IDV will _eventually_
> show all of the NCML files.  We did this by:
> 
> - create a new server side view of a single NCML file
> 
> This listing is reasonably fast.
> 
> - create a server side view of 10 NCML files
> 
> This listing is slow, but not slow enough to look like the IDV is
> hung.
> 
> - full set
> 
> We didn't bother to wait for this to complete, but we are convinced
> that it eventually will.
> 
> What is going on?
> 
> Evidently, netCDF Java is actually opening the dataset referenced in each
> URL named in the NCML files.  It seems to take between 4 and 5 seconds to
> get the information from the 7 URLs referenced in each NCML file.  If the
> wait time scales linearly, then it should take 335 (5 x 67) seconds to 
> generate
> the list for all of the NCML files you have in the NCEP1 6 hourly pressure 
> level
> data server side set of files.
> 
> Can this be sped up?
> 
> We don't know.  We'll have to run more tests to determine if the slowness
> is really in the netCDF Java code or simply a slowness in the ESRL web
> server's response to the query being made.
> 
> Sorry we didn't have better news!
> 
> Cheers,
> 
> Tom
> --
> ****************************************************************************
> Unidata User Support                                    UCAR Unidata Program
> (303) 497-8642                                                 P.O. Box 3000
> address@hidden<mailto:address@hidden>                                   
> Boulder, CO 80307
> ----------------------------------------------------------------------------
> Unidata HomePage                       http://www.unidata.ucar.edu
> ****************************************************************************
> 
> 
> Ticket Details
> ===================
> Ticket ID: BDO-300797
> Department: Support IDV
> Priority: Normal
> Status: Closed
> 
> 
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: BDO-300797
> Department: Support IDV
> Priority: Normal
> Status: Closed
> 
> 
> Brian Mapes
> address@hidden<mailto:address@hidden>
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: BDO-300797
Department: Support IDV
Priority: Normal
Status: Open