[netCDFDecoders #BRC-314440]: Motherlode Data Reterival

Hi Liam,

What is the IfCanOpen Java function you mention? Are you (is IfCanOpen) 
checking in the THREDDS XML catalog for the NCEP/WW3/Global data:

http://motherlode.ucar.edu/thredds/catalog/fmrc/NCEP/WW3/Global/files/catalog.xml

It lists all available NCEP/WW3/Global datasets. It also has a "Latest" URL

http://motherlode.ucar.edu/thredds/catalog/fmrc/NCEP/WW3/Global/files/latest.xml

which returns a THREDDS XML catalog which lists only the latest (and complete) 
NCEP/WW3/Global dataset.

[NOTE: You can look at both URLs above as HTML by changing the ".xml" suffix to 
".html".]

That latest.xml catalog is probably the best way to go about this at the 
moment. You could ping that every half hour or something until it shows that 
the latest dataset is the one you expect. Unfortunately, we don't have 
"LastModified" HTTP headers and the like working on that catalog. If that were 
in place, you could use a conditional HTTP GET request to determine when the 
"Latest" had changed.

If you aren't already, you should make sure you are on the address@hidden email 
list. We are working on the next version of the THREDDS Data Server (version 
4.3) which will introduce some changes to the URLs above and to some of the 
variable names in these datasets. Once the software is released, we'll have 
some time with that version available on our test server on motherlode. After 
some time for users to test things out, we'll upgrade our main motherlode 
server to that version. All of this activity will be communicated on the 
address@hidden email list.

Hope that helps,

Ethan

Liam Eastop wrote:
> 
> Hi Ethan,
> 
> Thanks so much for getting back to us and apologies again for the 95% failure 
> rate.
> 
> I'll write down our process of how we have been obtaining the data and 
> perhaps you
> might be able to suggest a preferred method.
> 
> The file which we are wanting to grab is the:
> http://motherlode.ucar.edu/thredds/dodsC/fmrc/NCEP/WW3/Global/files/WW3_Global_20120619_1800.grib2
> 
> Current Method:
> JAVA Web App
> 
> Steps:
> 1. Recreate a URL based on the current GMT date, plus forecastTime
> (e.g: /WW3_Global_currentDate_forecastTime.grib2)
> for example: /WW3_Global_20120619_1800.grib2
> 
> 2. We then check to see if the file exists on the catalogue using the 
> (IfCanOpen Java
> function). If True we download the file, Else we check again (previously 
> before last
> Friday we were checking every hour, however on friday we changed stepping 
> time to
> every 30min).
> 
> Second Error:
> We also had a second error, which occurred after we reset the server on 
> Friday.
> This initiated two checkTimers. This meant we were checking for a new file 4 
> times an
> hour and possibly 20 times per six hour block. Which would have occurred in 
> the 95%
> failure rate.
> 
> Our Reason for manipulating our timers:
> We changed our checkTimer because the catalogue doesn't seem to update at any 
> exact
> time based off the 'last modified' column 
>(http://motherlode.ucar.edu/thredds/catalog/fmrc/NCEP/WW3/Global/files/catalog.html),
> although the files are strictly 6hour forecasts.
> 
> Could you make any recommendations of a better technique or how we could 
> obtain the
> grib files closer to there creation time?
> 
> Currently we have stopped the server data retrieval until we know the best 
> technique.
> 
> 
> Thanks again Ethan, great emails and a big help.
> 
> 
> Liam


Ticket Details
===================
Ticket ID: BRC-314440
Department: Support THREDDS
Priority: Normal
Status: Closed