[THREDDS #GHR-456072]: nullPointerException in when accessing 4.3.15 GRIB featureCollection's

Hi Dan,

Marcos has been looking at some other GRIB indexing problems so I'm going to 
ask him to take a look at this one as well. 

As for the dateFormatMark, double checking with the "Data Extractor" section of 
the featureCollection reference document [1], I think you could either use a 
dateFormatMark like this

    <collection 
        spec="<baseDir>/gfsanl/**/gfsanl_3_.*\.grb$" 
        timePartition="1 day"
        dateFormatMark="#gfsanl_3_#yyyyMMdd_HHmm"
        />

Or you could include the date pattern in the collection@spec attribute like this

    <collection 
        spec="<baseDir>/gfsanl/**/gfsanl_3_#yyyyMMdd_HHmm#_000\.grb$" 
        timePartition="1 day"
        />

A side note on where you are writing index files ...

It looks like you are writing the index files in the content/thredds/cache 
directory. We always write ours in the same directory as the GRIB files. We do 
this for a few reasons: 1) we have multiple TDS (live, dev, test e.g.) serving 
the same GRIB files and we don't want duplicate index files; and 2) our data 
are on a disk with lots of space whereas content/thredds is on a much smaller 
disk that we wouldn't want to fill up.

The TDM allows us to separate the index writing process from the TDS process. 
This allows us to run the indexer as a user that can write to the data 
directory. Whereas, the tomcat user has very limited permissions and can't 
write to the data directory.

Hope that helps. 

Ethan

[1] 
http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.3/reference/collections/FeatureCollections.html

Dan Swank wrote: 
> Ethan:
> More information...
> 
> There were two of these:
> ./catalogInit.log:*** XML parser error (59:8)= cvc-complex-type.3.2.2:
> Attribute 'suffix' is not allowed to appear in element 'collection'.
> Which I corrected.. but that appeared to be superficial / unrelated.
> 
> I want to make a dataformatmark
>         dateFormatMark="#gfsanl_3_#yyyyMMdd_HHmm#_000.grb#"
> But I do not understand how to make this work.  So i reverted to:
>         dateFormatMark="#gfsanl_3_#yyyyMMdd_HHmm"
> This resolved all XML parser problems.
> 
> 
> 
> The system does not have any trouble writing to
> 
> <tomcat_home>/content/thredds/cache/GribIndex
> 
> (I rm'ed the entire thing and it was recreating index files &
>     directories upon application restart)
> 
> However, I am seeing alot of these that I cannot explain:
> 
> ./featureCollectionScan.log:[2013-02-22T12:02:25.034-0500] ERROR
> ucar.nc2.grib.GribCollection: Error reading index
> <tomcat_home>/content/thredds/cache/GribIndex/.../gfsanl/201001/20100104.ncx
> 
> ./featureCollectionScan.log:[2013-02-22T12:04:45.962-0500] ERROR
> ucar.nc2.grib.GribCollection: Error reading index
> <tomcat_home>/content/thredds/cache/GribIndex/.../gfsanl/201001/20100114.ncx
> 
> ./featureCollectionScan.log:[2013-02-22T12:04:45.964-0500] ERROR
> ucar.nc2.grib.GribCollection: Error reading index
> <tomcat_home>/content/thredds/cache/GribIndex/.../gfsanl/201001/20100114.ncx
> 
> -rw-rw-r-- 1 tomcat tomcat 34213 Feb 22 12:04
> <tomcat_home>/content/thredds/cache/GribIndex/.../gfsanl/201001/20100114.ncx
> 
> (The whole thing is running under tomcat user) so leads me to believe
>    perms are not the problem here.
> 
> Seems the TDM is having trouble crunching certain files and this is
> ultimately causing
> trouble upon accessing the feature Collection.  This would be my guess.
> 
> 
>         -Dan
> 
> 
> On Fri, Feb 22, 2013 at 10:40 AM, Dan Swank - NOAA Affiliate
> <address@hidden> wrote:
>> Ethan,
>>
>> I believe your first paragraph is getting toward the cause.
>>
>> I shall try PermGen options and get back to you.
>>
>> Here is the info from psiprobe:
>> At this point the TDS is not running.
>>
>> Memory utilization
>> Current memory usage is         ++++++++++++     2.4%
>> Free: 13.08 GB Total: 13.77 GB Max: 28.44 GB
>> OS information
>> JVM:  Java(TM) SE Runtime Environment  1.7.0_03-b04  Java HotSpot(TM)
>> 64-Bit Server VM)

>>
>> Container information
>> Container: Apache Tomcat/7.0.27
>>
>>         -Dan
>>
>>
>> On Wed, Feb 20, 2013 at 10:45 PM, Unidata THREDDS Support 
>> <address@hidden> wrote:
>>> Hi Dan,
>>> 
>>> What version of Tomcat and Java are you running?
>>> 
>>> I see an "OutOfMemoryError: PermGen space" in the
>>> serverStartup.log file. There is some psiprobe code wound up in
>>> the stack. Is that the OOM error you mention? Have you increased
>>> the maximum amount of PermGen space allowed with a
>>> "-XX:MaxPermSize" java option? We have an FAQ with some PermGen
>>> information [1]. Not sure if this one is related to the same
>>> issues or not.
>>> 
>>> In the threddsServlet.log file I'm seeing a NullPointerException.
>>> However, given the NPE stacktrace and the preceeding WARNing
>>> message [3], I'm wondering if there is a permission problem. Does
>>> the user that runs this Tomcat instance have permission to write
>>> in the content/thredds directory and/or the directory where the
>>> data files are located?
>>> 
>>> If that is the issue, another option is to run the THREDDS Data
>>> Manager (TDM) [2]. The TDM is a stand-alone process that can run
>>> in the background to crawl data directories and keep indexes up
>>> to date. It can also notify the TDS when indexes have been
>>> updated.
>>> 
>>> Let us know if any of that helps.
>>> 
>>> Ethan


Ticket Details
===================
Ticket ID: GHR-456072
Department: Support THREDDS
Priority: High
Status: Open