[netCDFJava #OSY-357654]: delays in opening remote NCML file

As near as I can tell. The open can be simulated
by timing the following 3 request using e.g. curl or browser.
1. <url>.das
2. <url>.dds
3. preload of coordinate variables: <url>.dods?lat,lon



> Just added the dds call and I get:
> URL:
> https://dataserver.nccs.nasa.gov/thredds/dodsC/bypass/CREATE-IP/reanalysis/CFSR/6hr/atmos/psl.ncml
> das:9619 ms
> dds:8195 ms
> ucar.nc2.dt.grid.GridDataset.open:66611 ms
> 
> -Jeff
> 
> 
> address@hidden> wrote:
> 
> > To be precise, when the request is made to the server, client will
> > get the dds and the das, not the das alone.
> >
> > I did a second experiment to see how long the
> > NetcdfDataset.acquireFile(location, task) operation
> > took in ToolUI.
> >
> > I used this url
> >     https://dataserver.nccs.nasa.gov/thredds/dodsC/bypass/
> > CREATE-IP/reanalysis/CFSR/6hr/atmos/psl.ncml
> >
> > The result was that the time for just the acquireFile
> > took 31 seconds which is way longer than 8 seconds.
> > So there is something else going on that I need to track down.
> >
> > >
> > > On Dec 4, 2017, at 12:27 PM, Jeff McWhirter <address@hidden<
> > mailto:address@hidden>> wrote:
> > >
> > > Through the browser the http request is sending a if-modified-since
> > request header. The server responds in sub-second time for the .das request
> > as the request is cached. If the if-modified-since header is removed then
> > it takes about 8 seconds to serve the request, same as with CURL.
> > >
> > > The question is why does ucar.nc2.dt.grid.GridDataset.open take a full
> > minute or more to make the request? I assume its hitting the .das to read
> > the metadata and just opening the dataset does not read any actual data.
> > Are there multiple requests going on? Is a different dods url being hit?
> > >
> > > -Jeff
> > >
> > >
> > address@hidden<mailto:support-
> > address@hidden>> wrote:
> > > So I did some simple experiments.
> > > When that URL is opened, some coordinate variables are preloaded.
> > > It appears that the time to do this preload read is about 8.5 seconds.
> > > This seems rather long to me. My speculation is that the server
> > > is responsible for most of this time. I suspect that code is somehow
> > > associated with ncml processing on the server.
> > >
> > >
> > > > I guess the question is is what is the griddataset.open call doing?
> > There
> > > > shouldn’t be any data reading going on and I assume it’s just hitting
> > the
> > > > .das url?
> > > >
> > > > Jeff
> > > >
> > > >
> > > > address@hidden<mailto:support-
> > address@hidden>> wrote:
> > > >
> > > > > My speculation is that this just a very big file and
> > > > > the cost is transmission + read time + time to convert to
> > > > > CDM on the client. Does that seem reasonable,
> > > > > or am I missing something.
> > > > >
> > > > > > I get ~7 seconds in the browser. Same with CURL
> > > > > >
> > > > > > -Jeff
> > > > > >
> > > > > >
> > > > > address@hidden<mailto:address@hidden>>
> > > > > > wrote:
> > > > > >
> > > > > > > A few seconds to return this, for me
> > > > > > >
> > > > > > > Error {
> > > > > > >     code = 403;
> > > > > > >     message = "Request too big=58988.094912 Mbytes, max=500.0";
> > > > > > > };
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > address@hidden<mailto:support-
> > address@hidden>> wrote:
> > > > > > >
> > > > > > > How long does the browser take if you access the .dods e.g.
> > > > > > > https://na01.safelinks.protection.outlook.com/?url=
> > > > > > > https%3A%2F%2Fdataserver.nccs.nasa.gov<https://na01.
> > safelinks.protection.outlook.com/?url=http%3A%2F%
> > 2F2Fdataserver.nccs.nasa.gov&data=02%7C01%7Cbmapes%40rsmas.miami.edu%
> > 7Ccd1bbeda9121439b40cc08d53b3c53ed%7C2a144b72f23942d48c0e6f0f17c4
> > 8e33%7C0%7C0%7C636480052656297117&sdata=LTwHF%2FAK%2F%
> > 2FqlaThVMsMvRj3nPXu4fXi1OT7lTvX78bc%3D&reserved=0>%2Fthredds%2FdodsC%
> > > > > > > 2Fbypass%2FCREATE-IP%2Freanalysis%2FCFSR%2F6hr%
> > > > > > > 2Fatmos%2Fpsl.ncml.dods&data=02%7C01%7Cbmapes%40rsmas.miami.edu<
> > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40rsmas .
> > miami.edu&data=02%7C01%7Cbmapes%40rsmas.miami.edu%
> > 7Ccd1bbeda9121439b40cc08d53b3c53ed%7C2a144b72f23942d48c0e6f0f17c4
> > 8e33%7C0%7C0%7C636480052656297117&sdata=EzfRhivi6RZMGXT5Bwv28giT1pbWnx
> > nGdCP0sV7HkdU%3D&reserved=0>%
> > > > > > > 7Cb669f01390d34fe8a83108d53a871659%
> > 7C2a144b72f23942d48c0e6f0f17c4
> > > > > > > 8e33%7C0%7C0%7C636479274232120145&sdata=TJcc4icYvzL6%
> > > > > > > 2Fb59dRDbNTtWQdx14zuCUJBoQciV42Q%3D&reserved=0
> > > > > > >
> > > > > > >
> > > > > > > I'm tracking a performance problem down for Brian Mapes (cc'ed) .
> > He
> > > > > has an
> > > > > > > ncml file that aggregates a number of ncml files and it takes a
> > very
> > > > > very
> > > > > > > long time to open the file.
> > > > > > >
> > > > > > > Just looking at one of the NCML links that is being accessed:
> > > > > > >
> > > > > > > https://na01.safelinks.protection.outlook.com/?url=
> > > > > > > https%3A%2F%2Fdataserver.nccs.nasa.gov<https://na01.
> > safelinks.protection.outlook.com/?url=http%3A%2F%
> > 2F2Fdataserver.nccs.nasa.gov&data=02%7C01%7Cbmapes%40rsmas.miami.edu%
> > 7Ccd1bbeda9121439b40cc08d53b3c53ed%7C2a144b72f23942d48c0e6f0f17c4
> > 8e33%7C0%7C0%7C636480052656297117&sdata=LTwHF%2FAK%2F%
> > 2FqlaThVMsMvRj3nPXu4fXi1OT7lTvX78bc%3D&reserved=0>%2Fthredds%2FdodsC%
> > > > > > >
> > > > > 2Fbypass%2FCREATE-IP%2Freanalysis%2FCFSR%2F6hr%
> > 2Fatmos%2Fpsl.ncml&data=02%
> > > > > > > 7C01%7Cbmapes%40rsmas.miami.edu<https://na01.safelinks.
> > protection.outlook.com/?url=http%3A%2F%2F40rsmas.miami.
> > edu&data=02%7C01%7Cbmapes%40rsmas.miami.edu%7Ccd1bbeda9121439b40cc08d53b3c
> > 53ed%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%
> > 7C636480052656297117&sdata=EzfRhivi6RZMGXT5Bwv28giT1pbWnx
> > nGdCP0sV7HkdU%3D&reserved=0>%7Cb669f01390d34fe8a83108d53a871659%
> > > > > > > 7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%
> > 7C636479274232120145&sdata=
> > > > > > > 5EBQlAWULuqQldFMKTm%2FJGYOTIKIvSAdVfL%2Fip6pDRA%3D&reserved=0
> > > > > > >
> > > > > > > In the browser it is sub-second to access the .das of this file .
> > > > > However,
> > > > > > > running CURL or the IDV's IOUtil.readcontents it takes ~10
> > seconds to
> > > > > > > fetch.  Running ucar.nc2.dt.grid.GridDataset.open it takes ~ 80
> > > > > seconds to
> > > > > > > open the link
> > > > > > >
> > > > > > > Any ideas?
> > > > > > >
> > > > > > > -Jeff
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > =Dennis Heimbigner
> > > > > > >  Unidata
> > > > > > >
> > > > > > >
> > > > > > > Ticket Details
> > > > > > > ===================
> > > > > > > Ticket ID: OSY-357654
> > > > > > > Department: Support netCDF Java
> > > > > > > Priority: Normal
> > > > > > > Status: Open
> > > > > > > ===================
> > > > > > > NOTE: All email exchanges with Unidata User Support are recorded
> > in the
> > > > > > > Unidata inquiry tracking system and then made publicly available
> > > > > through
> > > > > > > the web.  If you do not want to have your interactions made
> > available
> > > > > in
> > > > > > > this way, you must let us know in each email you send to us.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *********************************************
> > > > > > > Brian Mapes, Professor
> > > > > > > Department of Atmospheric Sciences
> > > > > > > Meteorology and Physical Oceanography Program
> > > > > > > RSMAS, University of Miami
> > > > > > > 4600 Rickenbacker Causeway
> > > > > https://maps.google.com/?q=4600+Rickenbacker+Causeway+%
> > 0D+Miami,+FL+33149&entry=gmail&source=g<https://na01.
> > safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaps.
> > google.com%2F%3Fq%3D4600%2BRickenbacker%2BCauseway%2B%
> > 250D%2BMiami%2C%2BFL%2B33149%26entry%3Dgmail%26source%3Dg&
> > data=02%7C01%7Cbmapes%40rsmas.miami.edu%7Ccd1bbeda9121439b40cc08d53b3c
> > 53ed%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%
> > 7C636480052656297117&sdata=jaKqb9E3fpBuDRpTCSD5W3xOyxSBwz
> > %2BkiyF1j02BF3E%3D&reserved=0>
> > > > > >
> > > > > > > Miami, FL 33149-1098
> > > > > > >
> > > > > > > phone: (305) 421-4275<tel:%28305%29%20421-4275>
> > > > > > > fax: (305) 421-4696<tel:%28305%29%20421-4696>
> > > > > > > email: address@hidden<mailto:address@hidden>
> > > > > > > Web: http://www.rsmas.miami.edu/users/bmapes/<https://na01.
> > safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.
> > rsmas.miami.edu%2Fusers%2Fbmapes%2F&data=02%7C01%
> > 7Cbmapes%40rsmas.miami.edu%7Ccd1bbeda9121439b40cc08d53b3c53ed%
> > 7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636480052656297117&sdata=
> > JNaF6vjTUXB64s1rw1ifLe%2B71OqYmk7qrBuk4vJsd2I%3D&reserved=0>
> > > > > > > **********************************************
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > =Dennis Heimbigner
> > > > >   Unidata
> > > > >
> > > > >
> > > > > Ticket Details
> > > > > ===================
> > > > > Ticket ID: OSY-357654
> > > > > Department: Support netCDF Java
> > > > > Priority: Normal
> > > > > Status: Open
> > > > > ===================
> > > > > NOTE: All email exchanges with Unidata User Support are recorded in
> > the
> > > > > Unidata inquiry tracking system and then made publicly available
> > through
> > > > > the web.  If you do not want to have your interactions made
> > available in
> > > > > this way, you must let us know in each email you send to us.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > =Dennis Heimbigner
> > > Unidata
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: OSY-357654
> > > Department: Support netCDF Java
> > > Priority: Normal
> > > Status: Open
> > > ===================
> > > NOTE: All email exchanges with Unidata User Support are recorded in the
> > Unidata inquiry tracking system and then made publicly available through
> > the web.  If you do not want to have your interactions made available in
> > this way, you must let us know in each email you send to us.
> > >
> > >
> > >
> > >
> > > *********************************************
> > > Brian Mapes, Professor
> > > Department of Atmospheric Sciences
> > > Meteorology and Physical Oceanography Program
> > > RSMAS, University of Miami
> > > 4600 Rickenbacker Causeway
> > > Miami, FL 33149-1098
> > >
> > > phone: (305) 421-4275
> > > fax: (305) 421-4696
> > > email: address@hidden<mailto:address@hidden>
> > > Web: http://www.rsmas.miami.edu/users/bmapes/
> > > **********************************************
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> > =Dennis Heimbigner
> >   Unidata
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: OSY-357654
> > Department: Support netCDF Java
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the
> > Unidata inquiry tracking system and then made publicly available through
> > the web.  If you do not want to have your interactions made available in
> > this way, you must let us know in each email you send to us.
> >
> >
> >
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: OSY-357654
Department: Support netCDF Java
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.