[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Fwd: Re: Simple demonstration]

Hi Jerome, Bob:

Im back from vacation, so im looking at this issue again.

What I think is happening is that the first time that a dataset is accessed, 
all the files have to be cracked and the coordinate value extracted. So that 
time will be proportional to the # of files. Subsequently, the info is read 
from the cached XML files (like the one you sent me), and it should be fast. 
Bob, is that what you are seeing? I dont think there is a differrence between 
using the opendap library or the nj22 library.

Im unclear what the systen is doing if/when files are added or deleted. Jerome, 
are these datasets stable, or do they change?

There are a few other details I need to investigate, so consider this a best 
guess for the moment, and let me know if you see any contrary evidence.

Thanks for checking it out...

Jerome King wrote:

Hi John,

I have attached the files requested.
Thanks for looking into this.
Jerome.

----- Original Message -----
From: John Caron <address@hidden>
Date: Friday, July 14, 2006 4:08 pm
Subject: Re: [Fwd: Re: Simple demonstration]



hi jerome:
can you send me 3 or file of the files in the /u00/satellite/CG/usfc/hday/ directory? 

and also send me the satellite-CG-usfc-hday file in cacheAged?

thanks!

Jerome King wrote:


hi John,

Bob asked me to respond to your questions:
I am guessing you're asking for the NcML for 

"satellite/CM/usfc/hday"> dataset.
<netcdf xmlns="" 

target="l">http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>


 <aggregation dimName="time" type="joinExisting">
   <variableAgg name="CGusfc" />
   <scan dateFormatMark="CG#yyyyDDD_HHmmss"
location="/u00/satellite/CG/usfc/hday/" suffix=".nc" />
</netcdf>
2) There are plenty of files in 

$tomcat_home/content/thredds/cacheAged/>  And there is a satellite-
CG-usfc-hday file of 37K.


Let me know if I can check something else,
Thanks,
Jerome.



On Fri, 2006-07-14 at 15:43, Bob Simons wrote:



Can you please answer this and reply to John and me?

-------- Original Message --------
Subject: Re: Simple demonstration
Date: Fri, 14 Jul 2006 16:40:53 -0600
From: John Caron <address@hidden>
Organization: UCAR/Unidata
To: Bob Simons <address@hidden>
References: <address@hidden>
Im guessing that theres something wrong with the TDS caching, so
that it 
has to recreate the dataset each time by reading all of the files.
What does the aggregation look like, for say the "satellite/CM/usfc/hday" dataset?
Can you look and see if there is a directory ${tomcat_home}/content/thredds/cacheAged, and if anything is in it? 

thanks

Bob Simons wrote:



I have reduced the tests to their core:

  /**
* This connects to the opendapUrl and gets the dataDds from 

the query.


   *
* @param opendapUrl, e.g., "http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellit * @param query e.g.,
"?CMusfc.CMusfc[0:1:0][0:1:0][0:1:20][0:1:20]">>> * @throws Exception if trouble 


   */
public static void simpleSpeedTest(String url, String query)
throws 
Exception {
      boolean acceptDeflate = true;
dods.dap.DConnect dConnect = new dods.dap.DConnect(url, acceptDeflate); 
      long time = System.currentTimeMillis();
      dods.dap.DataDDS dataDds = dConnect.getData(query, null);
      System.out.println("Opendap.simpleSpeedTest(\n" +
          "url=" + url + "\n" +
          "query=" + query + "\n" +
          "time=" + (System.currentTimeMillis() - time));
  }

  /**
   * This performs a series of simple speed tests.
   *
   * @throws Exception if trouble
   */
  public static void doSimpleSpeedTests() throws Exception {
      simpleSpeedTest(
"http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellit, 
          "?AGssta.AGssta[0:1:0][0:1:0][0:1:20][0:1:20]");
      simpleSpeedTest(




"http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellite/CM/usfc/hday";,




          "?CMusfc.CMusfc[0:1:0][0:1:0][0:1:20][0:1:20]");
      simpleSpeedTest(




"http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellite/GA/ssta/hday";,




          "?GAssta.GAssta[0:1:0][0:1:0][0:1:20][0:1:20]");
      simpleSpeedTest(




"http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellite/MB/chla/1day";,




          "?MBchla.MBchla[0:1:0][0:1:0][0:1:20][0:1:20]");
      simpleSpeedTest(




"http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellite/QN/curl/8day";,




          "?QNcurl.QNcurl[0:1:0][0:1:0][0:1:20][0:1:20]");
  }

The results from running simpleSpeedTests is

Opendap.simpleSpeedTest

url=http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellit

query=?AGssta.AGssta[0:1:0][0:1:0][0:1:20][0:1:20]
time=672
Opendap.simpleSpeedTest




url=http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellite/CM/usfc/hday


query=?CMusfc.CMusfc[0:1:0][0:1:0][0:1:20][0:1:20]
time=6922
Opendap.simpleSpeedTest




url=http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellite/GA/ssta/hday


query=?GAssta.GAssta[0:1:0][0:1:0][0:1:20][0:1:20]
time=3141
Opendap.simpleSpeedTest




url=http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellite/MB/chla/1day


query=?MBchla.MBchla[0:1:0][0:1:0][0:1:20][0:1:20]
time=515
Opendap.simpleSpeedTest




url=http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellite/QN/curl/8day


query=?QNcurl.QNcurl[0:1:0][0:1:0][0:1:20][0:1:20]
time=1610
These times are proportional to the times I mentioned earlier
and are 
strongly correlated to the number of files in each directory.
On the other hand, I see that if I go to the CMusfc opendap web
page and 
generate an ascii request for the same data



(http://oceanwatch.pfeg.noaa.gov:8081/thredds/dodsC/satellite/CM/usfc/hday.ascii?CMusfc[0:1:0][0:1:0][0:1:20][0:1:20])




I get the response very quickly.
That seems to point to the problem being a quirk of
dConnect.getData. Or 
am I misusing it?

Any suggestions?

Thank you.

Sincerely,

Bob Simons
Satellite Data Product Manager
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
(831)658-3205
address@hidden
<>< <>< <>< <>< <>< <>< <>< <>< <><