[THREDDS #YRL-997326]: Follow-up on TDS discussion

On 8/14/2012 9:27 AM, Don Murray wrote:
> New Client Reply: Follow-up on TDS discussion
>
> Hi Lansing-
>
> Thanks for looking into this and for providing your configuration
> information. Hoop is setting up a test machine and will get back to you
> what he finds out.
>
> One question is how you are testing that this works.  Are you using
> ToolsUI to check the data values to make sure the times are all there?
> Our problem has been with OPeNDAP access not listing all the times.
> Just want to make sure we're on the same page.
>
> Thanks.
>
> Don
>
> On 8/13/12 10:59 AM, Unidata THREDDS Support wrote:
>>> Thank you.  Let us know if you need specifics on configuration other
>>> than what Hoop has already posted.
>>>
>>> Don
>>>
>>> On 7/26/12 6:26 PM, Unidata THREDDS Support wrote:
>>>>> John-
>>>>>
>>>>> On 7/25/12 5:37 PM, Unidata THREDDS Support wrote:
>>>>>> This problem only affects NcML joinExisting aggregations.
>>>>> Thanks for that clarification.
>>>>>
>>>>>> If there's a problem with FMRC FeatureCollections, its a seperate 
>>>>>> problem that we dont know about.
>>>>> When Hoop first sent his messages to the thredds mailing list about the
>>>>> NcML aggregation not working, he was told by Ethan to use the
>>>>> FeatureCollection instead to fix the problem.  He did and the problem
>>>>> still persisted and reported the problem there.  Ethan suggested several
>>>>> things which did not solve the issue.  So, if you only tested the NcML
>>>>> aggregation, then perhaps you can review the conversation on the thredds
>>>>> list:
>>>>>
>>>>> https://www.unidata.ucar.edu/mailing_lists/archives/thredds/2012/msg00138.html
>>>>>
>>>>> and test the FeatureCollection as well.
>>>> Lansing will try to recreate the problem with FMRCs.
>>>>
>>>>
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: YRL-997326
>>>> Department: Support THREDDS
>>>> Priority: Normal
>>>> Status: Open
>>>>
>>> --
>>> Don Murray
>>> NOAA/ESRL/PSD and CIRES
>>> 303-497-3596
>>> http://www.esrl.noaa.gov/psd/people/don.murray/
>>>
>>>
>> Don,
>>
>> I've been working with some of the data files that Hoop previously provided 
>> for me (SST for a few years, with two different length versions for 2012) to 
>> reproduce the issue in fmrc that manifests.  Basically, I set up a 
>> collection with full years 2009-2011 and a partial year for 2012 (through 
>> May 13=4, 2012).  I fire up the tds, look at the collection, etc.  Then I 
>> switch the 2012 file for one that goes through May 15, 2012.  I've found a 
>> couple of interesting things.
>>
>> First off, it appears that fmrc is working in the current release of tds 
>> 4.2.  However, there is an error in updating catalog metadata, which results 
>> in a catalog page that appears out of date, i.e., it appears not to pick up 
>> the modified file.  However, clicking through to an access page (opendap in 
>> this case) reveals that the changes have been picked up by the tds.  This is 
>> also evidenced looking at the collection with toolsui.
>>
>> Second, there seem to be a couple of gotchas in play, perhaps environmental. 
>>  I have been battling some sort a shadow issue that manifests from time to 
>> time.  It is possible that you are seeing this in your installation, as 
>> well.  To see if this is the case, we would need to look at a fresh set of 
>> log files from the current stable tds release - particularly the 
>> featureCollectionScan.log and others in /content/thredds/logs.
>>
>> Here is the snippet from my catalog for the fmrc:
>>
>> <featureCollection featureType="FMRC" name="hoop_issue" harvest="true" 
>> path="hoop2">
>>       <metadata inherited="true">
>>         <serviceName>ncdods</serviceName>
>>         <dataFormat>netcdf</dataFormat>
>>         <documentation type="summary">Constantly changed and updated modeled 
>> sea surface temperatures</documentation>
>>       </metadata>
>>
>>       <collection spec="/C:/Users/madry/hoop2/sst.day.mean.#yyyy#.v2.nc$" 
>> name="modeled_SST" olderThan="1 min"/>
>>       <update startup="true" rescan="0 * * * * ? *" trigger="allow"/>
>>       <protoDataset choice="Penultimate" change="0 2 3 * * ? *" />
>>       <fmrcConfig regularize="false" datasetTypes="TwoD Best Files Runs 
>> ConstantForecasts ConstantOffsets" />
>>     </featureCollection>
>>
>> It would be good to establish whether the behavior that I have observed 
>> manifests in your configuration.
>>
>> -Lansing
>>
>> Ticket Details
>> ===================
>> Ticket ID: YRL-997326
>> Department: Support THREDDS
>> Priority: Emergency
>> Status: Open
>>
Don,

I used toolsui to "look under the hood" while I was doing my testing, 
then followed through with downloading the SST data via opendap.  
Essentially, I have years 2009-2011 plus 2012 through May 13th.  This 
gives a time series of length 1229.  Then, I switch the 2012 file for 
one that goes through May 14th.  The catalog page does not reflect this 
change under the "TimeCoverage:" information. However, clicking through 
to the opendap access shows a time series with 1230 steps.  So, it seems 
that the data is there, but it is not being described correctly on the 
catalog page.  This is a separate issue that we will address.

I did this again this morning using the latest 4.2 download.

-Lansing



Ticket Details
===================
Ticket ID: YRL-997326
Department: Support THREDDS
Priority: Emergency
Status: Open