Re: Paper on THREDDS Data Server (TDS)

Hi Leilei :

I have made a new release that hopefully will work, find it at the same place. 
It will have version 3.3.05.
Im not sure how you are installing, but its a good idea to delete the expanded 
webapp/thredds directory to be sure you get the new stuff.

Leilei Wang wrote:
> I'd already tried the latest v3.3 downloaded from your tech status page:
> http://www.unidata.ucar.edu/projects/THREDDS/tech/index.html#TDS*
> *TDS server just can't up and run at all:*
>
> **type* Exception report
>
> *message*
>
> *description* _The server encountered an internal error () that 
> prevented it from fulfilling this request._
>
> *exception*
>
> javax.servlet.ServletException: Servlet.init() for servlet ThreddsDefault threw 
> exception
>         
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
>         
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
>         org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
>         
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
>         
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
>         
> org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
>         
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
>         java.lang.Thread.run(Thread.java:534)
>
> *root cause*
>
> java.lang.NoSuchMethodError: 
> ucar.nc2.iosp.grib.GribServiceProvider.extendIndex(Z)V
>         thredds.servlet.ThreddsDefault.init(ThreddsDefault.java:55)
>         javax.servlet.GenericServlet.init(GenericServlet.java:211)
>         
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
>         
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
>         org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
>         
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
>         
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
>         
> org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
>         
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
>         java.lang.Thread.run(Thread.java:534)
>
> Leilei
>
> Ethan Davis wrote:
>
> > Hi Leilei,
> >
> > I grabbed the file mentioned below from your ftp site. Robb tried it 
> > out and could read it fine (and a few others I got from there).
> >
> > We are not using TDS 3.2 on motherlode. We are using the code from our 
> > development (currently 3.3) branch. We're guessing there is a bug fix 
> > on that branch that didn't get into 3.2. If you are interested in 
> > trying that version, I can send you the thredds.war file. Since it 
> > isn't a stable release yet, we don't advise relying on it but it 
> > should be fine for testing and such.
> >
> > Let me know if you want the 3.3 thredds.war file.
> >
> > Ethan
> >
> > Leilei Wang wrote:
> >
> > > Seems it is an unfortunate day for me.
> > > I've reloaded correct GRIB-2 NDFD data files onto our TDS. Metadata 
> > > seems fine with Netcdf Tool UI. But on "DODS Dataset Access Form",
> > > when I try to get ascii for a subset of some variables, I've got an 
> > > error.  I'm using latest official thredds.war 3.2 and even 
> > > reinstalled tomcat with the version you suggested, still got no luck. 
> > > Part of related error msg is followed. Seems array index is out of 
> > > bound. I remember we got similar problem back to Aug and you've fixed 
> > > it for other data. Weirdest thing is same data file(downloaded one 
> > > NDFD file from your site)works on your motherlode while not working 
> > > on our TDS. Are you using v3.2 with your motherlode thredds?
> > >
> > > http://eclipse.ncdc.noaa.gov:9090/thredds/dodsC/ndfd/20040901/YGUZ99_KWBN_200409012154.html 
> > >
> > > Error {
> > >
> > >    code = 0;
> > >    message = "Connection cannot be read 
> > > http://eclipse.ncdc.noaa.gov:9090/thredds/dodsC/ndfd/20040901/YGUZ99_KWBN_200409012154.dods?Maximum_temperature[0:1:0][0:1:10][0:1:10]";; 
> > >
> > > };
> > >
> > > 2005-11-14T16:46:41.302 -0500 [   2395772][     348] INFO  - 
> > > thredds.servlet.ServletUtil - Remote host: 205.167.25.154 - Request: 
> > > "GET 
> > > /thredds/dodsC/ndfd/20040901/YGUZ99_KWBN_200409012154.dods?Maximum_temperature[0:1:0][0:1:10][0:1:10] 
> > > HTTP/1.1"
> > > 2005-11-14T16:46:41.384 -0500 [   2395854][     348] ERROR - 
> > > dods.servers.netcdf.NcSDArray -   ERROR NcSDArray Read 
> > > Maximum_temperature
> > > java.lang.ArrayIndexOutOfBoundsException: 121
> > >        at ucar.ma2.ArrayFloat.setFloat(ArrayFloat.java:193)
> > >        at ucar.ma2.IteratorFast.setFloatNext(IteratorFast.java:64)
> > >        at 
> > > ucar.nc2.iosp.grib.GribServiceProvider.readXY(GribServiceProvider.java:218) 
> > >
> > >        at 
> > > ucar.nc2.iosp.grib.GribServiceProvider.readData(GribServiceProvider.java:177) 
> > >
> > >        at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:967)
> > >        at ucar.nc2.Variable._read(Variable.java:786)
> > >        at ucar.nc2.Variable.read(Variable.java:500)
> > >        at dods.servers.netcdf.NcSDArray.read(NcSDArray.java:110)
> > >        at dods.servers.netcdf.NcSDGrid.read(NcSDGrid.java:62)
> > >        at dods.dap.Server.SDGrid.serialize(SDGrid.java:490)
> > >        at dods.dap.Server.CEEvaluator.send(CEEvaluator.java:254)
> > >        at dods.servlet.DODSServlet.doGetDODS(DODSServlet.java:651)
> > >        at dods.servlet.DODSServlet.doGet(DODSServlet.java:1454)
> > >        at 
> > > dods.servers.netcdf.NcDODSServlet.doGet(NcDODSServlet.java:291)
> > > Thanks,
> > > Leilei
> > >
> > >
> > > Ethan Davis wrote:
> > >
> > > > Interesting. I wonder what is different about them. Can you send us 
> > > > (or make available for us to grab) an example file?
> > > >
> > > > Thanks,
> > > >
> > > > Ethan
> > > >
> > > > Leilei Wang wrote:
> > > >
> > > > > Thanks, Ethan! I'm attaching the error messages. And also you can 
> > > > > check our server for these data: 
> > > > > http://eclipse.ncdc.noaa.gov:9090/thredds/dodsC/ndfd/20050928/catalog.html 
> > > > >
> > > > >
> > > > > This is the error message I've got when the opendap link is clicked.
> > > > > ***************************************************************
> > > > > Error { code = 5; message = "NcDODSServlet ERROR opening NcDataset 
> > > > > dodsC/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb access = 
> > > > > /raid/ftp/pub/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb 
> > > > > null"; };
> > > > > *****************************************************************
> > > > > And this is message in log file:
> > > > > ******************************************************************
> > > > > 2005-11-14T11:06:21.588 -0500 [ 331405962][     268] INFO  - 
> > > > > thredds.servlet.ServletUtil - Remote host: 192.153.129.249 - 
> > > > > Request: "GET 
> > > > > /thredds/dodsC/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb.html 
> > > > > HTTP/1.1"
> > > > > 2005-11-14T11:06:21.653 -0500 [ 331406027][     268] ERROR - 
> > > > > dods.servers.netcdf.NcDODSServlet - NcDODSServlet ERROR opening 
> > > > > NcDataset dodsC/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb 
> > > > > access = 
> > > > > /raid/ftp/pub/ndfd/20050928/ndfd-all_020238_20050929_0000_fff.grb
> > > > > null
> > > > > java.lang.NullPointerException
> > > > >        at ucar.grib.grib1.Grib1Lookup.<init>(Grib1Lookup.java:36)
> > > > >        at 
> > > > > ucar.nc2.iosp.grib.Grib1ServiceProvider.open(Grib1ServiceProvider.java:72) 
> > > > >
> > > > >        at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:754)
> > > > >        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:334)
> > > > >        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:219)
> > > > >        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:165)
> > > > >        at 
> > > > > ucar.nc2.util.NetcdfFileCache.acquire(NetcdfFileCache.java:161)
> > > > >        at 
> > > > > dods.servers.netcdf.GuardedDatasetImpl.<init>(GuardedDatasetImpl.java:53) 
> > > > >
> > > > >        at 
> > > > > dods.servers.netcdf.NcDODSServlet.getDataset(NcDODSServlet.java:338)
> > > > >        at dods.servlet.DODSServlet.doGetHTML(DODSServlet.java:951)
> > > > >        at dods.servlet.DODSServlet.doGet(DODSServlet.java:1462)
> > > > >        at 
> > > > > dods.servers.netcdf.NcDODSServlet.doGet(NcDODSServlet.java:291)
> > > > >        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> > > > >        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> > > > >        at 
> > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) 
> > > > >
> > > > >        at 
> > > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) 
> > > > >
> > > > >        at 
> > > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) 
> > > > >
> > > > >        at 
> > > > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) 
> > > > >
> > > > >        at 
> > > > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) 
> > > > >
> > > > >        at 
> > > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) 
> > > > >
> > > > >        at 
> > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) 
> > > > >
> > > > >        at 
> > > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) 
> > > > >
> > > > >        at 
> > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) 
> > > > >
> > > > >        at 
> > > > > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:868) 
> > > > >
> > > > >        at 
> > > > > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744) 
> > > > >
> > > > >        at 
> > > > > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) 
> > > > >
> > > > >        at 
> > > > > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) 
> > > > >
> > > > >        at 
> > > > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) 
> > > > >
> > > > >        at java.lang.Thread.run(Unknown Source)
> > > > > 2005-11-14T11:06:21.680 -0500 [ 331406054][     268] INFO  - 
> > > > > thredds.servlet.ServletUtil - Request Completed - 200 - -1 - 92
> > > > > ************************************************************************************* 
> > > > >
> > > > > Seems it can't be opened at all and  Netcdf Tool verified this too.
> > > > >
> > > > > Yes, GRIB-2 data files directly downloaded from NDFD work fine in 
> > > > > TDS. But the files I've got problem are from NCDC archive system.
> > > > > I'm puzzled because these files seem in GRIB-1 format when I use 
> > > > > 'degrib' decoder to decode them. And current 'degrib' can't handle 
> > > > > them too because center & subcenter are not included in local use 
> > > > > tables.
> > > > > Thanks,
> > > > > Leilei
> > > > >
> > > > > Ethan Davis wrote:
> > > > >
> > > > > > Hi Leilei,
> > > > > >
> > > > > > The GRIB data is being read by the netCDF-java 2.2 library which 
> > > > > > has a number of adapters for reading various file formats. I'm 
> > > > > > CCing Robb Kambic who developed and maintains our GRIB code. I 
> > > > > > just checked on our server and looks like we are serving NDFD 
> > > > > > data. Here's the catalog for the NDFD data we're serving in case 
> > > > > > you want to take a look at that
> > > > > >
> > > > > > http://motherlode.ucar.edu:8080/thredds/idd/model/NDFD/CONUS_5km/catalog.xml 
> > > > > >
> > > > > >
> > > > > > Are you getting any relavent messages in the TDS logs in the 
> > > > > > ${TOMCAT_HOME}/content/thredds/logs directory?
> > > > > >
> > > > > > Another thing you could do is use the netCDF Tools UI to look at 
> > > > > > one of your data files and see if it can read it in. In the 
> > > > > > "Viewer" tab you can click on the folder icon to find a local file 
> > > > > > and try to open onf the your NDFD files. That way we'll know if it 
> > > > > > is a TDS or nc/GRIB issue. If the ToolsUI can't open the file, can 
> > > > > > you make one available so Robb can take a closer look?
> > > > > >
> > > > > > Let us know what kind of error messages (if any) you get from the 
> > > > > > ToolsUI or in the TDS logs.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Ethan
> > > > > >
> > > > > > Leilei Wang wrote:
> > > > > >
> > > > > > > Hi Ethan:
> > > > > > >    Which decoder does TDS use to decode GRIB files? Is it degrib? 
> > > > > > > We're trying to serve NCDC archived NDFD data files through TDS 
> > > > > > > which in GRIB but failed.
> > > > > > > Thanks,
> > > > > > > Leilei
> > > > > > >
> > > > > > > Ethan Davis wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > We will be presenting a paper on the THREDDS Data Server (TDS) 
> > > > > > > > at the 2006 AMS meeting in Atlanta. It is available online now 
> > > > > > > > if you want to take a look before January.
> > > > > > > >
> > > > > > > > http://www.unidata.ucar.edu/projects/THREDDS/PublicationsAndPresentations/2006AMS-TDS/ 
> > > > > > > >
> > > > > > > >
> > > > > > > > Ethan