Re: Problems with GRIB feature collections

Hi Roy,

It made it - thanks! Let me take a look and see what I come up with.

Cheers!

Sean

On Wed, Dec 6, 2017 at 3:08 PM, Roy Mendelssohn - NOAA Federal
<address@hidden> wrote:
> Just sent an email with the file.  I know in the past sometimes files that 
> make it through our mailer don't make it through yours.  Let me know if you 
> get it,  otherwise I will send by a different method.
>
> -Roy
>
>> On Dec 6, 2017, at 1:36 PM, Sean Arms <address@hidden> wrote:
>>
>> Can I get a copy of the files from December 4th? If that is too large,
>> I can work with the gbx9 and ncx3 files.
>>
>> Thanks!
>>
>> Sean
>>
>>
>> On Wed, Dec 6, 2017 at 2:30 PM, Roy Mendelssohn - NOAA Federal
>> <address@hidden> wrote:
>>> The latest time range is correct based on the files in the directory.    
>>> The files are in sub directories like 201712, so given that the xml defines 
>>> "as directory"  it is doing that and is finding the correct times, if that 
>>> helps at all.
>>>
>>> -Roy
>>>
>>>> On Dec 6, 2017, at 1:19 PM, Sean Arms <address@hidden> wrote:
>>>>
>>>> Hi Roy,
>>>>
>>>> This might take a bit. If we look at the time range on the "Best"
>>>> collection, we get:
>>>>
>>>> Start: 2013-02-21T00:00:00Z
>>>> End: 2017-12-09T06:00:00Z
>>>>
>>>> However, if we look at the time range on the "latest" (201712), we get:
>>>>
>>>> Start: 2017-12-01T00:00:00Z
>>>> End: 2017-12-04T18:00:00Z
>>>>
>>>> It's not clear where anything after 2017-12-04T18:00:00Z is coming
>>>> from, but I think I have a way of tracking it down if it isn't a bug
>>>> (again, it will just take a bit of time).
>>>>
>>>> Cheers,
>>>>
>>>> Sean
>>>>
>>>>
>>>> On Wed, Dec 6, 2017 at 12:47 PM, Roy Mendelssohn - NOAA Federal
>>>> <address@hidden> wrote:
>>>>> Okay the TwoD datasets are showing.  I still see the same issue with the 
>>>>> last time.  Can easily send you sample data if that would help.
>>>>>
>>>>> -ROy
>>>>>
>>>>>> On Dec 6, 2017, at 11:38 AM, Sean Arms <address@hidden> wrote:
>>>>>>
>>>>>> Hmmm. Can you try removing the ncx files to force the TDM to regenerate 
>>>>>> them? Both the TDS and TDM will need to be restarted.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Sean
>>>>>>
>>>>>> On Wed, Dec 6, 2017 at 12:17 PM Roy Mendelssohn - NOAA Federal 
>>>>>> <address@hidden> wrote:
>>>>>> Adding that appears to have sent that dataset into never-neverland.  Now 
>>>>>> I get no response.  DOn't see an error in the log file either.
>>>>>>
>>>>>> -Roy
>>>>>>
>>>>>>> On Dec 6, 2017, at 10:53 AM, Sean Arms <address@hidden> wrote:
>>>>>>>
>>>>>>> Greetings Roy,
>>>>>>>
>>>>>>> Sorry for the delay - I've been in a teleconference all morning.
>>>>>>>
>>>>>>> So the TDS may be picking up on the fact that some of those records
>>>>>>> are encoded as forecasts. That's possible, and I'll keep that in mind.
>>>>>>>
>>>>>>> The xml snippet should look like this (change made to first line to add 
>>>>>>> TwoD):
>>>>>>>
>>>>>>> <gribConfig datasetTypes="TwoD Best LatestFile">
>>>>>>>  <pdsHash>
>>>>>>>      <useTableVersion>false</useTableVersion>
>>>>>>>  </pdsHash>
>>>>>>> </gribConfig>
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Sean
>>>>>>>
>>>>>>> On Wed, Dec 6, 2017 at 11:05 AM, Roy Mendelssohn - NOAA Federal
>>>>>>> <address@hidden> wrote:
>>>>>>>> BTW - I should add that the 1-degree data is a special run that they 
>>>>>>>> do for us,  and the way I understand it in their processing terms the 
>>>>>>>> 00 and 12 files are considered real-time,  while the 06 and 18 files 
>>>>>>>> are considered "forecasts".  They have different filenames because of 
>>>>>>>> that, for example:
>>>>>>>>
>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>>>>
>>>>>>>> That is why in the regex for the time I have to skip two spaces.  In 
>>>>>>>> case this info helps any:
>>>>>>>>
>>>>>>>> wgrib -V 
>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>>>> rec 1:21:date 2017120400 PRMSL kpds5=2 kpds6=102 kpds7=0 levels=(0,0) 
>>>>>>>> grid=240 MSL anl:
>>>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 0 
>>>>>>>> missing 0
>>>>>>>> center 58 subcenter 0 process 18 Table 2 scan: WE:SN winds(N/S)
>>>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>>>        long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 
>>>>>>>> mode 128 bdsgrid 1
>>>>>>>> min/max data 95236 104736  num bits 10  BDS_Ref 9523.6  DecScale -1 
>>>>>>>> BinScale 0
>>>>>>>>
>>>>>>>> wgrib -V 
>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>>>> rec 1:0:date 2017120418 PRMSL kpds5=2 kpds6=102 kpds7=0 levels=(0,0) 
>>>>>>>> grid=240 MSL anl:
>>>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 0 
>>>>>>>> missing 0
>>>>>>>> center 58 subcenter 0 process 18 Table 3 scan: WE:SN winds(N/S)
>>>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>>>        long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 
>>>>>>>> mode 128 bdsgrid 1
>>>>>>>> min/max data 94770 104260  num bits 10  BDS_Ref 9477  DecScale -1 
>>>>>>>> BinScale 0
>>>>>>>>
>>>>>>>>
>>>>>>>> BTW - Note the different table,  which is why I have include in the 
>>>>>>>> xml the stuff about:
>>>>>>>>
>>>>>>>>         <pdsHash>
>>>>>>>>            <useTableVersion>false</useTableVersion>
>>>>>>>>         </pdsHash>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> -Roy
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Dec 6, 2017, at 6:09 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>
>>>>>>>>> For the second question regarding "full" vs "best", this has to do 
>>>>>>>>> with how the overall collection is aggregated. For the half-degree 
>>>>>>>>> NAVGEM, you have monthly collections of grids, but none of the 
>>>>>>>>> forecasts overlap. Therefore, the overall collection aggregation 
>>>>>>>>> exposes all of the grids. For the one degree, it looks like the TDS 
>>>>>>>>> thinks there was an overlap in the grids (i.e. two grids with the 
>>>>>>>>> same valid time), and so one of the overlapped grids was "better" 
>>>>>>>>> (i.e. closer to a runtime) and the other was dropped to make a "best" 
>>>>>>>>> collection. However, I don't see that happening in any of the monthly 
>>>>>>>>> collections listed, so it is probably a bug. In the xml for these two 
>>>>>>>>> collections, can you add "TwoD" to the datasetTypes attribute of the 
>>>>>>>>> gribConfig element? That will give us a bit more information.
>>>>>>>>>
>>>>>>>>
>>>>>>>> **********************
>>>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>>>> Government or NOAA."
>>>>>>>> **********************
>>>>>>>> Roy Mendelssohn
>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>> NOAA/NMFS
>>>>>>>> Environmental Research Division
>>>>>>>> Southwest Fisheries Science Center
>>>>>>>> ***Note new street address***
>>>>>>>> 110 McAllister Way
>>>>>>>> Santa Cruz, CA 95060
>>>>>>>> Phone: (831)-420-3666
>>>>>>>> Fax: (831) 420-3980
>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>
>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>>>> -MLK Jr.
>>>>>>>>
>>>>>>
>>>>>> **********************
>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>> Government or NOAA."
>>>>>> **********************
>>>>>> Roy Mendelssohn
>>>>>> Supervisory Operations Research Analyst
>>>>>> NOAA/NMFS
>>>>>> Environmental Research Division
>>>>>> Southwest Fisheries Science Center
>>>>>> ***Note new street address***
>>>>>> 110 McAllister Way
>>>>>> Santa Cruz, CA 95060
>>>>>> Phone: (831)-420-3666
>>>>>> Fax: (831) 420-3980
>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>
>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>> "From those who have been given much, much will be expected"
>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>> -MLK Jr.
>>>>>>
>>>>>
>>>>> **********************
>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>> Government or NOAA."
>>>>> **********************
>>>>> Roy Mendelssohn
>>>>> Supervisory Operations Research Analyst
>>>>> NOAA/NMFS
>>>>> Environmental Research Division
>>>>> Southwest Fisheries Science Center
>>>>> ***Note new street address***
>>>>> 110 McAllister Way
>>>>> Santa Cruz, CA 95060
>>>>> Phone: (831)-420-3666
>>>>> Fax: (831) 420-3980
>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>
>>>>> "Old age and treachery will overcome youth and skill."
>>>>> "From those who have been given much, much will be expected"
>>>>> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>>>>> Jr.
>>>>>
>>>
>>> **********************
>>> "The contents of this message do not reflect any position of the U.S. 
>>> Government or NOAA."
>>> **********************
>>> Roy Mendelssohn
>>> Supervisory Operations Research Analyst
>>> NOAA/NMFS
>>> Environmental Research Division
>>> Southwest Fisheries Science Center
>>> ***Note new street address***
>>> 110 McAllister Way
>>> Santa Cruz, CA 95060
>>> Phone: (831)-420-3666
>>> Fax: (831) 420-3980
>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>
>>> "Old age and treachery will overcome youth and skill."
>>> "From those who have been given much, much will be expected"
>>> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>>> Jr.
>>>
>
> **********************
> "The contents of this message do not reflect any position of the U.S. 
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>