Re: Problems with GRIB feature collections

Can I get a copy of the files from December 4th? If that is too large,
I can work with the gbx9 and ncx3 files.

Thanks!

Sean


On Wed, Dec 6, 2017 at 2:30 PM, Roy Mendelssohn - NOAA Federal
<address@hidden> wrote:
> The latest time range is correct based on the files in the directory.    The 
> files are in sub directories like 201712, so given that the xml defines "as 
> directory"  it is doing that and is finding the correct times, if that helps 
> at all.
>
> -Roy
>
>> On Dec 6, 2017, at 1:19 PM, Sean Arms <address@hidden> wrote:
>>
>> Hi Roy,
>>
>> This might take a bit. If we look at the time range on the "Best"
>> collection, we get:
>>
>> Start: 2013-02-21T00:00:00Z
>> End: 2017-12-09T06:00:00Z
>>
>> However, if we look at the time range on the "latest" (201712), we get:
>>
>> Start: 2017-12-01T00:00:00Z
>> End: 2017-12-04T18:00:00Z
>>
>> It's not clear where anything after 2017-12-04T18:00:00Z is coming
>> from, but I think I have a way of tracking it down if it isn't a bug
>> (again, it will just take a bit of time).
>>
>> Cheers,
>>
>> Sean
>>
>>
>> On Wed, Dec 6, 2017 at 12:47 PM, Roy Mendelssohn - NOAA Federal
>> <address@hidden> wrote:
>>> Okay the TwoD datasets are showing.  I still see the same issue with the 
>>> last time.  Can easily send you sample data if that would help.
>>>
>>> -ROy
>>>
>>>> On Dec 6, 2017, at 11:38 AM, Sean Arms <address@hidden> wrote:
>>>>
>>>> Hmmm. Can you try removing the ncx files to force the TDM to regenerate 
>>>> them? Both the TDS and TDM will need to be restarted.
>>>>
>>>> Cheers,
>>>>
>>>> Sean
>>>>
>>>> On Wed, Dec 6, 2017 at 12:17 PM Roy Mendelssohn - NOAA Federal 
>>>> <address@hidden> wrote:
>>>> Adding that appears to have sent that dataset into never-neverland.  Now I 
>>>> get no response.  DOn't see an error in the log file either.
>>>>
>>>> -Roy
>>>>
>>>>> On Dec 6, 2017, at 10:53 AM, Sean Arms <address@hidden> wrote:
>>>>>
>>>>> Greetings Roy,
>>>>>
>>>>> Sorry for the delay - I've been in a teleconference all morning.
>>>>>
>>>>> So the TDS may be picking up on the fact that some of those records
>>>>> are encoded as forecasts. That's possible, and I'll keep that in mind .
>>>>>
>>>>> The xml snippet should look like this (change made to first line to add 
>>>>> TwoD):
>>>>>
>>>>> <gribConfig datasetTypes="TwoD Best LatestFile">
>>>>>   <pdsHash>
>>>>>       <useTableVersion>false</useTableVersion>
>>>>>   </pdsHash>
>>>>> </gribConfig>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Sean
>>>>>
>>>>> On Wed, Dec 6, 2017 at 11:05 AM, Roy Mendelssohn - NOAA Federal
>>>>> <address@hidden> wrote:
>>>>>> BTW - I should add that the 1-degree data is a special run that they do 
>>>>>> for us,  and the way I understand it in their processing terms the 00 
>>>>>> and 12 files are considered real-time,  while the 06 and 18 files are 
>>>>>> considered "forecasts".  They have different filenames because of that, 
>>>>>> for example:
>>>>>>
>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>>
>>>>>> That is why in the regex for the time I have to skip two spaces.  In 
>>>>>> case this info helps any:
>>>>>>
>>>>>> wgrib -V 
>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>> rec 1:21:date 2017120400 PRMSL kpds5=2 kpds6=102 kpds7=0 levels=(0,0) 
>>>>>> grid=240 MSL anl:
>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 0 
>>>>>> missing 0
>>>>>> center 58 subcenter 0 process 18 Table 2 scan: WE:SN winds(N/S)
>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>         long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 mode 
>>>>>> 128 bdsgrid 1
>>>>>> min/max data 95236 104736  num bits 10  BDS_Ref 9523.6  DecScale -1 
>>>>>> BinScale 0
>>>>>>
>>>>>> wgrib -V 
>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>> rec 1:0:date 2017120418 PRMSL kpds5=2 kpds6=102 kpds7=0 levels=(0,0) 
>>>>>> grid=240 MSL anl:
>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 0 
>>>>>> missing 0
>>>>>> center 58 subcenter 0 process 18 Table 3 scan: WE:SN winds(N/S)
>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>         long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 mode 
>>>>>> 128 bdsgrid 1
>>>>>> min/max data 94770 104260  num bits 10  BDS_Ref 9477  DecScale -1 
>>>>>> BinScale 0
>>>>>>
>>>>>>
>>>>>> BTW - Note the different table,  which is why I have include in the xml 
>>>>>> the stuff about:
>>>>>>
>>>>>>          <pdsHash>
>>>>>>             <useTableVersion>false</useTableVersion>
>>>>>>          </pdsHash>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -Roy
>>>>>>
>>>>>>
>>>>>>> On Dec 6, 2017, at 6:09 AM, Sean Arms <address@hidden> wrote:
>>>>>>>
>>>>>>> For the second question regarding "full" vs "best", this has to do with 
>>>>>>> how the overall collection is aggregated. For the half-degree NAVGEM, 
>>>>>>> you have monthly collections of grids, but none of the forecasts 
>>>>>>> overlap. Therefore, the overall collection aggregation exposes all of 
>>>>>>> the grids. For the one degree, it looks like the TDS thinks there was 
>>>>>>> an overlap in the grids (i.e. two grids with the same valid time), and 
>>>>>>> so one of the overlapped grids was "better" (i.e. closer to a runtime) 
>>>>>>> and the other was dropped to make a "best" collection. However, I don't 
>>>>>>> see that happening in any of the monthly collections listed, so it is 
>>>>>>> probably a bug. In the xml for these two collections, can you add 
>>>>>>> "TwoD" to the datasetTypes attribute of the gribConfig element? That 
>>>>>>> will give us a bit more information.
>>>>>>>
>>>>>>
>>>>>> **********************
>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>> Government or NOAA."
>>>>>> **********************
>>>>>> Roy Mendelssohn
>>>>>> Supervisory Operations Research Analyst
>>>>>> NOAA/NMFS
>>>>>> Environmental Research Division
>>>>>> Southwest Fisheries Science Center
>>>>>> ***Note new street address***
>>>>>> 110 McAllister Way
>>>>>> Santa Cruz, CA 95060
>>>>>> Phone: (831)-420-3666
>>>>>> Fax: (831) 420-3980
>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>
>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>> "From those who have been given much, much will be expected"
>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>> -MLK Jr.
>>>>>>
>>>>
>>>> **********************
>>>> "The contents of this message do not reflect any position of the U.S. 
>>>> Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division
>>>> Southwest Fisheries Science Center
>>>> ***Note new street address***
>>>> 110 McAllister Way
>>>> Santa Cruz, CA 95060
>>>> Phone: (831)-420-3666
>>>> Fax: (831) 420-3980
>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>
>>>> "Old age and treachery will overcome youth and skill."
>>>> "From those who have been given much, much will be expected"
>>>> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>>>> Jr.
>>>>
>>>
>>> **********************
>>> "The contents of this message do not reflect any position of the U.S. 
>>> Government or NOAA."
>>> **********************
>>> Roy Mendelssohn
>>> Supervisory Operations Research Analyst
>>> NOAA/NMFS
>>> Environmental Research Division
>>> Southwest Fisheries Science Center
>>> ***Note new street address***
>>> 110 McAllister Way
>>> Santa Cruz, CA 95060
>>> Phone: (831)-420-3666
>>> Fax: (831) 420-3980
>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>
>>> "Old age and treachery will overcome youth and skill."
>>> "From those who have been given much, much will be expected"
>>> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>>> Jr.
>>>
>
> **********************
> "The contents of this message do not reflect any position of the U.S. 
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>