[THREDDS #GHR-456072]: nullPointerException in when accessing 4.3.15 GRIB featureCollection's

Hi Dan:

Can you send me one of the files getting this error:

> ./featureCollectionScan.log:[2013-02-22T12:02:25.034-0500] ERROR
> ucar.nc2.grib.GribCollection: Error reading index
> <tomcat_home>/content/thredds/cache/GribIndex/.../gfsanl/201001/20100104.ncx

John

> Ethan:
> More information...
> 
> There were two of these:
> ./catalogInit.log:*** XML parser error (59:8)= cvc-complex-type.3.2.2:
> Attribute 'suffix' is not allowed to appear in element 'collection'.
> Which I corrected.. but that appeared to be superficial / unrelated.
> 
> I want to make a dataformatmark
> dateFormatMark="#gfsanl_3_#yyyyMMdd_HHmm#_000.grb#"
> But I do not understand how to make this work.  So i reverted to:
> dateFormatMark="#gfsanl_3_#yyyyMMdd_HHmm"
> This resolved all XML parser problems.
> 
> 
> 
> The system does not have any trouble writing to
> 
> /home/tomcat/tomcat-main.4.3.15/content/thredds/cache/GribIndex
> 
> (I rm'ed the entire thing and it was recreating index files &
> directories upon application restart)
> 
> However, I am seeing alot of these that I cannot explain:
> 
> ./featureCollectionScan.log:[2013-02-22T12:02:25.034-0500] ERROR
> ucar.nc2.grib.GribCollection: Error reading index
> /home/tomcat/tomcat-main.4.3.15/content/thredds/cache/GribIndex/global/nomads/nexus/gfsanl/201001/20100104.ncx
> 
> ./featureCollectionScan.log:[2013-02-22T12:04:45.962-0500] ERROR
> ucar.nc2.grib.GribCollection: Error reading index
> /home/tomcat/tomcat-main.4.3.15/content/thredds/cache/GribIndex/global/nomads/nexus/gfsanl/201001/20100114.ncx
> 
> ./featureCollectionScan.log:[2013-02-22T12:04:45.964-0500] ERROR
> ucar.nc2.grib.GribCollection: Error reading index
> /home/tomcat/tomcat-main.4.3.15/content/thredds/cache/GribIndex/global/nomads/nexus/gfsanl/201001/20100114.ncx
> 
> -rw-rw-r-- 1 tomcat tomcat 34213 Feb 22 12:04
> /home/tomcat/tomcat-main.4.3.15/content/thredds/cache/GribIndex/global/nomads/nexus/gfsanl/201001/20100114.ncx
> 
> (The whole thing is running under tomcat user) so leads me to believe
> perms are not the problem here.
> 
> Seems the TDM is having trouble crunching certain files and this is
> ultimately causing
> trouble upon accessing the feature Collection.  This would be my guess.
> 
> 
> -Dan
> 
> 
> On Fri, Feb 22, 2013 at 10:40 AM, Dan Swank - NOAA Affiliate
> <address@hidden> wrote:
> > Ethan,
> >
> > I believe your first paragraph is getting toward the cause.
> >
> > I shall try PermGen options and get back to you.
> >
> > Here is the info from psiprobe:
> > At this point the TDS is not running.
> >
> > Memory utilization
> > Current memory usage is         ++++++++++++     2.4%
> > Free: 13.08 GB Total: 13.77 GB Max: 28.44 GB
> > OS information
> > JVM:  Java(TM) SE Runtime Environment  1.7.0_03-b04  Java HotSpot(TM)
> > 64-Bit Server VM)
> > OS: Linux (unknown) amd64 2.6.32-279.9.1.el6.x86_64
> > Processors: 24
> > Current time: Fri Feb 22 10:31:45 EST 2013
> >
> > Container information
> > Container: Apache Tomcat/7.0.27
> >
> >         -Dan
> >
> >
> > On Wed, Feb 20, 2013 at 10:45 PM, Unidata THREDDS Support
> > <address@hidden> wrote:
> >> Hi Dan,
> >>
> >> What version of Tomcat and Java are you running?
> >>
> >> I see an "OutOfMemoryError: PermGen space" in the serverStartup.log file. 
> >> There is some psiprobe code wound up in the stack. Is that the OOM error 
> >> you mention? Have you increased the maximum amount of PermGen space 
> >> allowed with a "-XX:MaxPermSize" java option? We have an FAQ with some 
> >> PermGen information [1]. Not sure if this one is related to the same 
> >> issues or not.
> >>
> >> In the threddsServlet.log file I'm seeing a NullPointerException. However, 
> >> given the NPE stacktrace and the preceeding WARNing message [3], I'm 
> >> wondering if there is a permission problem. Does the user that runs this 
> >> Tomcat instance have permission to write in the content/thredds directory 
> >> and/or the directory where the data files are located?
> >>
> >> If that is the issue, another option is to run the THREDDS Data Manager 
> >> (TDM) [2]. The TDM is a stand-alone process that can run in the background 
> >> to crawl data directories and keep indexes up to date. It can also notify 
> >> the TDS when indexes have been updated.
> >>
> >> Let us know if any of that helps.
> >>
> >> Ethan
> >>
> >> Dan Swank wrote:
> >>> Upon Ethan's request, here is the last two messages
> >>> I sent to Caron, whom is out right now:
> >>> I will be sending the full config & logs to Ethan
> >>> in a separate message:
> >>>
> >>> ==== Begin FWD ==============================
> >>>
> >>> John
> >>>
> >>> I need to raise this thread after about 40 days of staleness.
> >>> After wading through the docs regarding the feature collection
> >>> and trying a few things, I just cannot get this working for our
> >>> gfsanl dataset.  (Around L40 on catalog.xml).
> >>>
> >>> Attached tar.bz2 package contains configs & logs.
> >>> The exception in featureCollection.xml* is different than before.
> >>> However I had my psi probe come up with an OOM error.
> >>> so it very well may still be sucking it dry.
> >>>
> >>> I have my opts set as  -Xms8192m -Xmx16384m
> >>>
> >>> Any help is appreciated.
> >>>
> >>>
> >>> ==== next 
> >>> ==================================================================
> >>>
> >>> John:
> >>> I found that it didn't like my dateFormatMark
> >>> ~ corrected that, it actually builds the .gbx9 indecies now,
> >>> but the same NullPointerException  is occuring:
> >>>
> >>> java.lang.NullPointerException
> >>> at 
> >>> thredds.catalog.InvDatasetFcGrib.makeCatalogTop(InvDatasetFcGrib.java:449)
> >>> at thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:373)
> >>> at 
> >>> thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1469)
> >>> at thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1432)
> >>> at 
> >>> thredds.server.catalogservice.LocalCatalogServiceController.handleRequestInternal(LocalCatalogServiceController.java:191)
> >>> at 
> >>> org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
> >>> at 
> >>> org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
> >>> at 
> >>> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
> >>> at 
> >>> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
> >>> ~ more ~
> >>>
> >>> Also changed the path so it only scans 1 month.
> >>>
> >>>
> >>> =======================================
> >>> Dan Swank
> >>> NCDC-NOMADS Project
> >>> Team ERT/STG
> >>
> >> ============================
> >>
> >> [1] 
> >> http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.2/faq.html#permGenOutOfMemoryError
> >>
> >> [2] 
> >> http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.3/reference/collections/TDM.html
> >>
> >> [3]
> >> 2013-01-16T15:55:06.525 -0500 [     12250][       7] INFO  - 
> >> threddsServlet - Remote host: 127.0.0.1 - Request: "GET 
> >> /thredds/catalog/gribFC/gfsanlfc/catalog.html HTTP/1.1"
> >> 2013-01-16T15:55:06.533 -0500 [     12258][       7] WARN  - 
> >> ucar.nc2.util.DiskCache2 - Cant create directories for file 
> >> .../content/thredds/cache/GribIndex/global/nomads/nexus/gfsanl
> >> 2013-01-16T15:55:33.820 -0500 [     39545][       7] ERROR - 
> >> thredds.server.catalogservice.LocalCatalogServiceController - 
> >> handleRequestInternal(): Problem handling request.
> >> java.lang.NullPointerException
> >>   at 
> >> thredds.filesystem.ControllerOS$Traversal.<init>(ControllerOS.java:245)
> >>   at 
> >> thredds.filesystem.ControllerOS$MFileIteratorAll.getNextIterator(ControllerOS.java:218)
> >>   at 
> >> thredds.filesystem.ControllerOS$MFileIteratorAll.getNextIterator(ControllerOS.java:224)
> >>   at 
> >> thredds.filesystem.ControllerOS$MFileIteratorAll.getNextIterator(ControllerOS.java:224)
> >>   at 
> >> thredds.filesystem.ControllerOS$MFileIteratorAll.hasNext(ControllerOS.java:196)
> >>   at 
> >> thredds.filesystem.ControllerOS$FilteredIterator.nextFilteredFile(ControllerOS.java:137)
> >>   at 
> >> thredds.filesystem.ControllerOS$FilteredIterator.hasNext(ControllerOS.java:119)
> >>   at 
> >> thredds.inventory.MFileCollectionManager.reallyScan(MFileCollectionManager.java:517)
> >>   at 
> >> thredds.inventory.MFileCollectionManager.scanFirstTime(MFileCollectionManager.java:478)
> >>   at 
> >> thredds.inventory.MFileCollectionManager.getFiles(MFileCollectionManager.java:440)
> >>   at 
> >> thredds.inventory.TimePartitionCollection.makePartitionsByDays(TimePartitionCollection.java:254)
> >>   at 
> >> thredds.inventory.TimePartitionCollection.makePartitions(TimePartitionCollection.java:93)
> >>   at 
> >> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.needsUpdate(Grib1TimePartitionBuilder.java:137)
> >>   at 
> >> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.readOrCreateIndex(Grib1TimePartitionBuilder.java:123)
> >>   at 
> >> ucar.nc2.grib.grib1.Grib1TimePartitionBuilder.factory(Grib1TimePartitionBuilder.java:73)
> >>   at ucar.nc2.grib.TimePartition.factory(TimePartition.java:96)
> >>   at 
> >> thredds.catalog.InvDatasetFcGrib.updateCollection(InvDatasetFcGrib.java:220)
> >>   at thredds.catalog.InvDatasetFcGrib.checkState(InvDatasetFcGrib.java:205)
> >>   at 
> >> thredds.catalog.InvDatasetFcGrib.makeCatalog(InvDatasetFcGrib.java:364)
> >>   at 
> >> thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1469)
> >>   at thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1432)
> >>   ...
> >>
> >>
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: GHR-456072
> >> Department: Support THREDDS
> >> Priority: High
> >> Status: Open
> >>
> >
> >
> >
> > --
> > =======================================
> > Dan Swank
> > NCDC-NOMADS Project
> > Team ERT/STG
> > (NCDC Government Contractor)
> > Data Access & Applications Branch
> > Veach-Baley Federal Building
> > 151 Patton Avenue
> > Asheville, NC 28801-5001
> > Email: address@hidden
> > Phone: 828-271-4007
> > ============================================
> > Any opinions expressed in this message are mine personally and
> > do not represent any position from STG, ERT, NCDC, or NOAA
> > ============================================
> 
> 
> 
> --
> =======================================
> Dan Swank
> NCDC-NOMADS Project
> Team ERT/STG
> (NCDC Government Contractor)
> Data Access & Applications Branch
> Veach-Baley Federal Building
> 151 Patton Avenue
> Asheville, NC 28801-5001
> Email: address@hidden
> Phone: 828-271-4007
> ============================================
> Any opinions expressed in this message are mine personally and
> do not represent any position from STG, ERT, NCDC, or NOAA
> ============================================
> 
> 


Ticket Details
===================
Ticket ID: GHR-456072
Department: Support THREDDS
Priority: High
Status: Open