Re: Problems with GRIB feature collections

Hi Roy,

This might take a bit. If we look at the time range on the "Best"
collection, we get:

Start: 2013-02-21T00:00:00Z
End: 2017-12-09T06:00:00Z

However, if we look at the time range on the "latest" (201712), we get:

Start: 2017-12-01T00:00:00Z
End: 2017-12-04T18:00:00Z

It's not clear where anything after 2017-12-04T18:00:00Z is coming
from, but I think I have a way of tracking it down if it isn't a bug
(again, it will just take a bit of time).

Cheers,

Sean


On Wed, Dec 6, 2017 at 12:47 PM, Roy Mendelssohn - NOAA Federal
<address@hidden> wrote:
> Okay the TwoD datasets are showing.  I still see the same issue with the last 
> time.  Can easily send you sample data if that would help.
>
> -ROy
>
>> On Dec 6, 2017, at 11:38 AM, Sean Arms <address@hidden> wrote:
>>
>> Hmmm. Can you try removing the ncx files to force the TDM to regenerate 
>> them? Both the TDS and TDM will need to be restarted.
>>
>> Cheers,
>>
>> Sean
>>
>> On Wed, Dec 6, 2017 at 12:17 PM Roy Mendelssohn - NOAA Federal 
>> <address@hidden> wrote:
>> Adding that appears to have sent that dataset into never-neverland.  Now I 
>> get no response.  DOn't see an error in the log file either.
>>
>> -Roy
>>
>> > On Dec 6, 2017, at 10:53 AM, Sean Arms <address@hidden> wrote:
>> >
>> > Greetings Roy,
>> >
>> > Sorry for the delay - I've been in a teleconference all morning.
>> >
>> > So the TDS may be picking up on the fact that some of those records
>> > are encoded as forecasts. That's possible, and I'll keep that in mind.
>> >
>> > The xml snippet should look like this (change made to first line to add 
>> > TwoD):
>> >
>> > <gribConfig datasetTypes="TwoD Best LatestFile">
>> >    <pdsHash>
>> >        <useTableVersion>false</useTableVersion>
>> >    </pdsHash>
>> > </gribConfig>
>> >
>> > Cheers,
>> >
>> > Sean
>> >
>> > On Wed, Dec 6, 2017 at 11:05 AM, Roy Mendelssohn - NOAA Federal
>> > <address@hidden> wrote:
>> >> BTW - I should add that the 1-degree data is a special run that they do 
>> >> for us,  and the way I understand it in their processing terms the 00 and 
>> >> 12 files are considered real-time,  while the 06 and 18 files are 
>> >> considered "forecasts".  They have different filenames because of that, 
>> >> for example:
>> >>
>> >> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>> >> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>> >>
>> >> That is why in the regex for the time I have to skip two spaces.  In case 
>> >> this info helps any:
>> >>
>> >> wgrib -V 
>> >> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>> >> rec 1:21:date 2017120400 PRMSL kpds5=2 kpds6=102 kpds7=0 levels=(0,0) 
>> >> grid=240 MSL anl:
>> >>  PRMSL=Pressure reduced to MSL [Pa]
>> >>  timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 0 
>> >> missing 0
>> >>  center 58 subcenter 0 process 18 Table 2 scan: WE:SN winds(N/S)
>> >>  latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>> >>          long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 mode 
>> >> 128 bdsgrid 1
>> >>  min/max data 95236 104736  num bits 10  BDS_Ref 9523.6  DecScale -1 
>> >> BinScale 0
>> >>
>> >> wgrib -V 
>> >> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>> >> rec 1:0:date 2017120418 PRMSL kpds5=2 kpds6=102 kpds7=0 levels=(0,0) 
>> >> grid=240 MSL anl:
>> >>  PRMSL=Pressure reduced to MSL [Pa]
>> >>  timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 0 
>> >> missing 0
>> >>  center 58 subcenter 0 process 18 Table 3 scan: WE:SN winds(N/S)
>> >>  latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>> >>          long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 mode 
>> >> 128 bdsgrid 1
>> >>  min/max data 94770 104260  num bits 10  BDS_Ref 9477  DecScale -1 
>> >> BinScale 0
>> >>
>> >>
>> >> BTW - Note the different table,  which is why I have include in the xml 
>> >> the stuff about:
>> >>
>> >>           <pdsHash>
>> >>              <useTableVersion>false</useTableVersion>
>> >>           </pdsHash>
>> >>
>> >>
>> >> Thanks,
>> >>
>> >> -Roy
>> >>
>> >>
>> >>> On Dec 6, 2017, at 6:09 AM, Sean Arms <address@hidden> wrote:
>> >>>
>> >>> For the second question regarding "full" vs "best", this has to do with 
>> >>> how the overall collection is aggregated. For the half-degree NAVGEM, 
>> >>> you have monthly collections of grids, but none of the forecasts 
>> >>> overlap. Therefore, the overall collection aggregation exposes all of 
>> >>> the grids. For the one degree, it looks like the TDS thinks there was an 
>> >>> overlap in the grids (i.e. two grids with the same valid time), and so 
>> >>> one of the overlapped grids was "better" (i.e. closer to a runtime) and 
>> >>> the other was dropped to make a "best" collection. However, I don't see 
>> >>> that happening in any of the monthly collections listed, so it is 
>> >>> probably a bug. In the xml for these two collections, can you add "TwoD" 
>> >>> to the datasetTypes attribute of the gribConfig element? That will give 
>> >>> us a bit more information.
>> >>>
>> >>
>> >> **********************
>> >> "The contents of this message do not reflect any position of the U.S. 
>> >> Government or NOAA."
>> >> **********************
>> >> Roy Mendelssohn
>> >> Supervisory Operations Research Analyst
>> >> NOAA/NMFS
>> >> Environmental Research Division
>> >> Southwest Fisheries Science Center
>> >> ***Note new street address***
>> >> 110 McAllister Way
>> >> Santa Cruz, CA 95060
>> >> Phone: (831)-420-3666
>> >> Fax: (831) 420-3980
>> >> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>> >>
>> >> "Old age and treachery will overcome youth and skill."
>> >> "From those who have been given much, much will be expected"
>> >> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>> >> Jr.
>> >>
>>
>> **********************
>> "The contents of this message do not reflect any position of the U.S. 
>> Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> ***Note new street address***
>> 110 McAllister Way
>> Santa Cruz, CA 95060
>> Phone: (831)-420-3666
>> Fax: (831) 420-3980
>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected"
>> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>>
>
> **********************
> "The contents of this message do not reflect any position of the U.S. 
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>