[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problems with GRIB feature collections

Sounds good. We'll get this worked out sooner or later!

Sean


On Fri, Dec 8, 2017 at 3:34 PM, Roy Mendelssohn - NOAA Federal
<address@hidden> wrote:
> Let  me look at that.  Let me also go over the logs more carefully to make 
> certain that nothing is on one of them that I am missing.  Will also be 
> writing a python program to go over each file and check the times. Probably 
> not this weekend unless I got motivated, will let you know.
>
> Thanks for your help!!!
>
> -Roy
>
>> On Dec 8, 2017, at 2:28 PM, Sean Arms <address@hidden> wrote:
>>
>> Greetings Roy,
>>
>> Not sure if this is causing the issues or not, but I noticed that the
>> collection from 201604:
>>
>> http://thredds1.pfeg.noaa.gov/thredds/catalog/FNMOC/NAVGEM/onedegree/pressure/6hr/FNMOC_NAVGEM_1-degree_6-hourly_Pressure-201604/catalog.html?dataset=FNMOC/NAVGEM/onedegree/pressure/6hr/FNMOC_NAVGEM_1-degree_6-hourly_Pressure-201604
>>
>> shows
>>
>> Start: 2016-04-01T00:00:00Z
>> End: 2016-05-04T06:00:00Z
>>
>> The end time in May looks a little suspicious. Can you check those
>> files to see if there is actually data from May in that directory? I
>> know the docs say that the directories should only contain grids from
>> the same partition, but I am not sure what happens if that rule is
>> violated (i.e. a few May grids in the April directory). It's a shot in
>> the dark.
>>
>> Worst case - how big is the entire collection?
>>
>> Cheers,
>>
>> Sean
>>
>>
>> On Fri, Dec 8, 2017 at 10:07 AM, Roy Mendelssohn - NOAA Federal
>> <address@hidden> wrote:
>>> Hi Sean:
>>>
>>> So,  I shutdown the TDS.  I removed all the ncx* and gbx* files.  I removed 
>>> the grid cache.  So the restart should be completely clean.  I started the 
>>> TDM.  I started the TDS.  The TDM did its thing.  For the NAVGEM 1-day,  no 
>>> matter which of the best options I look at(Best,  or Full), the last time 
>>> still have an increment of  8 hours instead of 6,  and is off into the 
>>> future.  All of the other times are correct.
>>>
>>> Thanks,
>>>
>>> -Roy
>>>
>>>> On Dec 6, 2017, at 3:49 PM, Sean Arms <address@hidden> wrote:
>>>>
>>>> Hi Roy,
>>>>
>>>> I can't reproduce the issue here locally with the files and the
>>>> catalog you provided. Can you try to remove all of the .gbx9 and .ncx3
>>>> files for the one degree collection and have the TDM rebuild them from
>>>> scratch?
>>>>
>>>> Thanks!
>>>>
>>>> Sean
>>>>
>>>> On Wed, Dec 6, 2017 at 4:12 PM, Sean Arms <address@hidden> wrote:
>>>>> Hi Roy,
>>>>>
>>>>> It made it - thanks! Let me take a look and see what I come up with.
>>>>>
>>>>> Cheers!
>>>>>
>>>>> Sean
>>>>>
>>>>> On Wed, Dec 6, 2017 at 3:08 PM, Roy Mendelssohn - NOAA Federal
>>>>> <address@hidden> wrote:
>>>>>> Just sent an email with the file.  I know in the past sometimes files 
>>>>>> that make it through our mailer don't make it through yours.  Let me 
>>>>>> know if you get it,  otherwise I will send by a different method.
>>>>>>
>>>>>> -Roy
>>>>>>
>>>>>>> On Dec 6, 2017, at 1:36 PM, Sean Arms <address@hidden> wrote:
>>>>>>>
>>>>>>> Can I get a copy of the files from December 4th? If that is too large,
>>>>>>> I can work with the gbx9 and ncx3 files.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Sean
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Dec 6, 2017 at 2:30 PM, Roy Mendelssohn - NOAA Federal
>>>>>>> <address@hidden> wrote:
>>>>>>>> The latest time range is correct based on the files in the directory.  
>>>>>>>>   The files are in sub directories like 201712, so given that the xml 
>>>>>>>> defines "as directory"  it is doing that and is finding the correct 
>>>>>>>> times, if that helps at all.
>>>>>>>>
>>>>>>>> -Roy
>>>>>>>>
>>>>>>>>> On Dec 6, 2017, at 1:19 PM, Sean Arms <address@hidden> wrote:
>>>>>>>>>
>>>>>>>>> Hi Roy,
>>>>>>>>>
>>>>>>>>> This might take a bit. If we look at the time range on the "Best"
>>>>>>>>> collection, we get:
>>>>>>>>>
>>>>>>>>> Start: 2013-02-21T00:00:00Z
>>>>>>>>> End: 2017-12-09T06:00:00Z
>>>>>>>>>
>>>>>>>>> However, if we look at the time range on the "latest" (201712), we 
>>>>>>>>> get:
>>>>>>>>>
>>>>>>>>> Start: 2017-12-01T00:00:00Z
>>>>>>>>> End: 2017-12-04T18:00:00Z
>>>>>>>>>
>>>>>>>>> It's not clear where anything after 2017-12-04T18:00:00Z is coming
>>>>>>>>> from, but I think I have a way of tracking it down if it isn't a bug
>>>>>>>>> (again, it will just take a bit of time).
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Sean
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Dec 6, 2017 at 12:47 PM, Roy Mendelssohn - NOAA Federal
>>>>>>>>> <address@hidden> wrote:
>>>>>>>>>> Okay the TwoD datasets are showing.  I still see the same issue with 
>>>>>>>>>> the last time.  Can easily send you sample data if that would help.
>>>>>>>>>>
>>>>>>>>>> -ROy
>>>>>>>>>>
>>>>>>>>>>> On Dec 6, 2017, at 11:38 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hmmm. Can you try removing the ncx files to force the TDM to 
>>>>>>>>>>> regenerate them? Both the TDS and TDM will need to be restarted.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Sean
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 6, 2017 at 12:17 PM Roy Mendelssohn - NOAA Federal 
>>>>>>>>>>> <address@hidden> wrote:
>>>>>>>>>>> Adding that appears to have sent that dataset into never-neverland. 
>>>>>>>>>>>  Now I get no response.  DOn't see an error in the log file either.
>>>>>>>>>>>
>>>>>>>>>>> -Roy
>>>>>>>>>>>
>>>>>>>>>>>> On Dec 6, 2017, at 10:53 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Greetings Roy,
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry for the delay - I've been in a teleconference all morning.
>>>>>>>>>>>>
>>>>>>>>>>>> So the TDS may be picking up on the fact that some of those records
>>>>>>>>>>>> are encoded as forecasts. That's possible, and I'll keep that in 
>>>>>>>>>>>> mind.
>>>>>>>>>>>>
>>>>>>>>>>>> The xml snippet should look like this (change made to first line 
>>>>>>>>>>>> to add TwoD):
>>>>>>>>>>>>
>>>>>>>>>>>> <gribConfig datasetTypes="TwoD Best LatestFile">
>>>>>>>>>>>> <pdsHash>
>>>>>>>>>>>>    <useTableVersion>false</useTableVersion>
>>>>>>>>>>>> </pdsHash>
>>>>>>>>>>>> </gribConfig>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>
>>>>>>>>>>>> Sean
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Dec 6, 2017 at 11:05 AM, Roy Mendelssohn - NOAA Federal
>>>>>>>>>>>> <address@hidden> wrote:
>>>>>>>>>>>>> BTW - I should add that the 1-degree data is a special run that 
>>>>>>>>>>>>> they do for us,  and the way I understand it in their processing 
>>>>>>>>>>>>> terms the 00 and 12 files are considered real-time,  while the 06 
>>>>>>>>>>>>> and 18 files are considered "forecasts".  They have different 
>>>>>>>>>>>>> filenames because of that, for example:
>>>>>>>>>>>>>
>>>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>>>>>>>>>
>>>>>>>>>>>>> That is why in the regex for the time I have to skip two spaces.  
>>>>>>>>>>>>> In case this info helps any:
>>>>>>>>>>>>>
>>>>>>>>>>>>> wgrib -V 
>>>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>>>>>>>>> rec 1:21:date 2017120400 PRMSL kpds5=2 kpds6=102 kpds7=0 
>>>>>>>>>>>>> levels=(0,0) grid=240 MSL anl:
>>>>>>>>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>>>>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 
>>>>>>>>>>>>> num_in_ave 0 missing 0
>>>>>>>>>>>>> center 58 subcenter 0 process 18 Table 2 scan: WE:SN winds(N/S)
>>>>>>>>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>>>>>>>>      long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 
>>>>>>>>>>>>> mode 128 bdsgrid 1
>>>>>>>>>>>>> min/max data 95236 104736  num bits 10  BDS_Ref 9523.6  DecScale 
>>>>>>>>>>>>> -1 BinScale 0
>>>>>>>>>>>>>
>>>>>>>>>>>>> wgrib -V 
>>>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>>>>>>>>> rec 1:0:date 2017120418 PRMSL kpds5=2 kpds6=102 kpds7=0 
>>>>>>>>>>>>> levels=(0,0) grid=240 MSL anl:
>>>>>>>>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>>>>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 
>>>>>>>>>>>>> num_in_ave 0 missing 0
>>>>>>>>>>>>> center 58 subcenter 0 process 18 Table 3 scan: WE:SN winds(N/S)
>>>>>>>>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>>>>>>>>      long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 
>>>>>>>>>>>>> mode 128 bdsgrid 1
>>>>>>>>>>>>> min/max data 94770 104260  num bits 10  BDS_Ref 9477  DecScale -1 
>>>>>>>>>>>>> BinScale 0
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> BTW - Note the different table,  which is why I have include in 
>>>>>>>>>>>>> the xml the stuff about:
>>>>>>>>>>>>>
>>>>>>>>>>>>>       <pdsHash>
>>>>>>>>>>>>>          <useTableVersion>false</useTableVersion>
>>>>>>>>>>>>>       </pdsHash>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Roy
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Dec 6, 2017, at 6:09 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For the second question regarding "full" vs "best", this has to 
>>>>>>>>>>>>>> do with how the overall collection is aggregated. For the 
>>>>>>>>>>>>>> half-degree NAVGEM, you have monthly collections of grids, but 
>>>>>>>>>>>>>> none of the forecasts overlap. Therefore, the overall collection 
>>>>>>>>>>>>>> aggregation exposes all of the grids. For the one degree, it 
>>>>>>>>>>>>>> looks like the TDS thinks there was an overlap in the grids 
>>>>>>>>>>>>>> (i.e. two grids with the same valid time), and so one of the 
>>>>>>>>>>>>>> overlapped grids was "better" (i.e. closer to a runtime) and the 
>>>>>>>>>>>>>> other was dropped to make a "best" collection. However, I don't 
>>>>>>>>>>>>>> see that happening in any of the monthly collections listed, so 
>>>>>>>>>>>>>> it is probably a bug. In the xml for these two collections, can 
>>>>>>>>>>>>>> you add "TwoD" to the datasetTypes attribute of the gribConfig 
>>>>>>>>>>>>>> element? That will give us a bit more information.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> **********************
>>>>>>>>>>>>> "The contents of this message do not reflect any position of the 
>>>>>>>>>>>>> U.S. Government or NOAA."
>>>>>>>>>>>>> **********************
>>>>>>>>>>>>> Roy Mendelssohn
>>>>>>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>>>>>>> NOAA/NMFS
>>>>>>>>>>>>> Environmental Research Division
>>>>>>>>>>>>> Southwest Fisheries Science Center
>>>>>>>>>>>>> ***Note new street address***
>>>>>>>>>>>>> 110 McAllister Way
>>>>>>>>>>>>> Santa Cruz, CA 95060
>>>>>>>>>>>>> Phone: (831)-420-3666
>>>>>>>>>>>>> Fax: (831) 420-3980
>>>>>>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>>>>>>> "the arc of the moral universe is long, but it bends toward 
>>>>>>>>>>>>> justice" -MLK Jr.
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> **********************
>>>>>>>>>>> "The contents of this message do not reflect any position of the 
>>>>>>>>>>> U.S. Government or NOAA."
>>>>>>>>>>> **********************
>>>>>>>>>>> Roy Mendelssohn
>>>>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>>>>> NOAA/NMFS
>>>>>>>>>>> Environmental Research Division
>>>>>>>>>>> Southwest Fisheries Science Center
>>>>>>>>>>> ***Note new street address***
>>>>>>>>>>> 110 McAllister Way
>>>>>>>>>>> Santa Cruz, CA 95060
>>>>>>>>>>> Phone: (831)-420-3666
>>>>>>>>>>> Fax: (831) 420-3980
>>>>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>>>>
>>>>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>>>>> "the arc of the moral universe is long, but it bends toward 
>>>>>>>>>>> justice" -MLK Jr.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> **********************
>>>>>>>>>> "The contents of this message do not reflect any position of the 
>>>>>>>>>> U.S. Government or NOAA."
>>>>>>>>>> **********************
>>>>>>>>>> Roy Mendelssohn
>>>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>>>> NOAA/NMFS
>>>>>>>>>> Environmental Research Division
>>>>>>>>>> Southwest Fisheries Science Center
>>>>>>>>>> ***Note new street address***
>>>>>>>>>> 110 McAllister Way
>>>>>>>>>> Santa Cruz, CA 95060
>>>>>>>>>> Phone: (831)-420-3666
>>>>>>>>>> Fax: (831) 420-3980
>>>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>>>
>>>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>>>>>> -MLK Jr.
>>>>>>>>>>
>>>>>>>>
>>>>>>>> **********************
>>>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>>>> Government or NOAA."
>>>>>>>> **********************
>>>>>>>> Roy Mendelssohn
>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>> NOAA/NMFS
>>>>>>>> Environmental Research Division
>>>>>>>> Southwest Fisheries Science Center
>>>>>>>> ***Note new street address***
>>>>>>>> 110 McAllister Way
>>>>>>>> Santa Cruz, CA 95060
>>>>>>>> Phone: (831)-420-3666
>>>>>>>> Fax: (831) 420-3980
>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>
>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>>>> -MLK Jr.
>>>>>>>>
>>>>>>
>>>>>> **********************
>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>> Government or NOAA."
>>>>>> **********************
>>>>>> Roy Mendelssohn
>>>>>> Supervisory Operations Research Analyst
>>>>>> NOAA/NMFS
>>>>>> Environmental Research Division
>>>>>> Southwest Fisheries Science Center
>>>>>> ***Note new street address***
>>>>>> 110 McAllister Way
>>>>>> Santa Cruz, CA 95060
>>>>>> Phone: (831)-420-3666
>>>>>> Fax: (831) 420-3980
>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>
>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>> "From those who have been given much, much will be expected"
>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>> -MLK Jr.
>>>>>>
>>>
>>> **********************
>>> "The contents of this message do not reflect any position of the U.S. 
>>> Government or NOAA."
>>> **********************
>>> Roy Mendelssohn
>>> Supervisory Operations Research Analyst
>>> NOAA/NMFS
>>> Environmental Research Division
>>> Southwest Fisheries Science Center
>>> ***Note new street address***
>>> 110 McAllister Way
>>> Santa Cruz, CA 95060
>>> Phone: (831)-420-3666
>>> Fax: (831) 420-3980
>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>
>>> "Old age and treachery will overcome youth and skill."
>>> "From those who have been given much, much will be expected"
>>> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>>> Jr.
>>>
>
> **********************
> "The contents of this message do not reflect any position of the U.S. 
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>