[IDV #BDO-300797]: catalog choking on folder of OK .ncml files

Brian, Don, et al.,

Speaking entirely speculatively, does this have anything to do with the NWS hack
from a few weeks ago?  Could the NOAA web servers be limiting successive URL
requests (which the NCML aggregation seems to do) to prevent DOS attacks?

(No, this problem is not b/c of a change in Java.)

Best,

Unidata IDV Support

> Tom and Julien, (cc hoop)
> 
> 
> Might this (cc from Hoop) be an issue ?
> It used to be faster so we are narrowing to what might have changed?
> 
> 
> Has there been a change in java or something? About 2-3 other things went a 
> bit
> haywire this week, including on weather.rsmas. I haven’t updated weather 
> recently,
> but some ISL scripts stopped working. (Updating to IDV5.0u2 fixed that).
> 
> 
> 
> Brian
> 
> 
> 
> On Nov 24, 2014, at 11:00 AM, NOAA ESRL/PSD Web & Data Team <address@hidden> 
> wrote:
> 
> > Something has changed recently, though I would be surprised if it
> > was the culprit.  Last month, we changed the format of the R1 files
> > from NetCDF3 to NetCDF4-Classic.  In general, this means the files
> > read more quickly, in our experience at least.  Since OPeNDAP renders
> > all files the same to the reading application regardless of source
> > format (I.E., essentially as NetCDF3 files), this would imply that
> > the Java NetCDF API used by the TDS server software is less efficient
> > reading NetCDF4-Classic files.  This seems unreasonable as the Java
> > API is generally the most forward-thinking and capable of the NetCDF
> > APIs.  As I recall, it spoke NetCDF4 before the others.  So, I'm
> > puzzled. (from Hoop)
> 
> 
> On Nov 21, 2014, at 5:38 PM, Unidata IDV Support <address@hidden> wrote:
> 
> > Hi Brian,
> >
> > re: IDV Catalog listing of directory with lots of NCML files that reference 
> > data
> > through URLs
> 
> > Julien and I proved that the catalog view in the IDV will _eventually_
> > show all of the NCML files.  We did this by:
> >
> > - create a new server side view of a single NCML file
> >
> >  This listing is reasonably fast.
> >
> > - create a server side view of 10 NCML files
> >
> >  This listing is slow, but not slow enough to look like the IDV is
> >  hung.
> >
> > - full set
> >
> >  We didn't bother to wait for this to complete, but we are convinced
> >  that it eventually will.
> >
> > What is going on?
> >
> > Evidently, netCDF Java is actually opening the dataset referenced in each
> > URL named in the NCML files.  It seems to take between 4 and 5 seconds to
> > get the information from the 7 URLs referenced in each NCML file.  If the
> > wait time scales linearly, then it should take 335 (5 x 67) seconds to 
> > generate
> > the list for all of the NCML files you have in the NCEP1 6 hourly pressure 
> > level
> > data server side set of files.
> >
> > Can this be sped up?
> >
> > We don't know.  We'll have to run more tests to determine if the slowness
> > is really in the netCDF Java code or simply a slowness in the ESRL web
> > server's response to the query being made.
> >
> > Sorry we didn't have better news!
> >
> > Cheers,
> >
> > Tom
> > --
> > ****************************************************************************
> > Unidata User Support                                    UCAR Unidata Program
> > (303) 497-8642                                                 P.O. Box 3000
> > address@hidden                                   Boulder, CO 80307
> > ----------------------------------------------------------------------------
> > Unidata HomePage                       http://www.unidata.ucar.edu
> > ****************************************************************************
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: BDO-300797
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed
> >
> 
> *********************************************
> Brian Mapes, Professor
> Department of Atmospheric Sciences
> Meteorology and Physical Oceanography Program
> RSMAS, University of Miami
> 4600 Rickenbacker Causeway
> Miami, FL 33149-1098
> 
> phone: (305) 421-4275
> fax: (305) 421-4696
> email: address@hidden
> Web: http://www.rsmas.miami.edu/users/bmapes/
> **********************************************
> 
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: BDO-300797
Department: Support IDV
Priority: Normal
Status: Closed