[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Happy Friday!

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: Happy Friday!
  • Date: Fri, 26 Oct 2012 22:37:38 -0500
Thanks. I'll see if I can get to someone at the source that can address this. 

In the meantime, this filename ought to be there:
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20121026/00/monthly_grib_01/flxf.01.2012102600.201305.avrg.grib.grb2

On 10/26/2012 6:57 PM, John Caron wrote:

Hi Jack:
CFSR has known problems, which we surmise happened when the original GRIB1 files were translated into GRIB2. We are looking for ways to fix these problems on the server, assuming that they arent going to fix them at the source. But it would be certainly better if they were rewritten correctly. 

John
PS: I seemed to have lost this file, and its gone on the the ftp site. can you send me another link? 

On 10/11/2012 11:45 AM, Jack Settelmaier wrote:
Sorry for not responding sooner helpful gents, as I see in this thread I 
was asked a question.
In looking at the below PDS info, I see no valid reason why they would 
have this set as they do:
Grib2ProductDefinitionSection
Interval = (2012-08-27T00:00:00.000Z,2012-09-27T00:00:00.000Z) 

The file is/was a climate model run that ran on
2012-09-27T00:00:00.000Z, with forecast conditions that go out MONTHS in 
advance. I see no good reason for them to be referencing
2012-08-27T00:00:00.000Z--that looks like a flat out error.
It seems to me they need to encode that setting as you surmised it 
should be:
(2013-05-01T00:00:00.000Z, 2013-06-01T00:00:00.000Z)
So, should I attempt to find the "creator" of this GRIB to get this corrected, rather than you making changes to your system which would 
work if they encoded it properly?

On 9/28/2012 2:19 PM, John Caron wrote:

Hi all:
1. The CFSR dataset is known to have grib encoding issues, so its possible we are seeing some of that and/or bugs in the software. in particular, the translation of grib1 to grib2 had bugs in it. ive been unable to get authoritative info on how to fix this, but i do have 
machinery to do so if we can figure out what to do.
2. The 4.3 software is completely different from 4.2. We are only going to fix 4.3 grib problems. 4.2 doesnt handle time interval 
coordinates correctly.

3. In this file:
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01/flxf.01.2012092700.201305.avrg.grib.grb2 

I am seeing records like the one I have dumped below.

statistic type is "Average"

reference time is 2012-09-27T00:00:00.000Z

the "end of overall time interval" is  == 2012/09/27T0:0:0

forecast time is 8 months.

we are interpreting the time interval coordinate as:

 (0.000000, 1.000000) == (2012-08-27T00:00:00.000Z,
2012-09-27T00:00:00.000Z)

which seems wrong. ill have to look at this closer.
4. Note that this is an interval coordinate, meaning the "valid 
time"is a time interval. From this comment:
"The forecasts from that model run are for the mean conditions over 
the months of May 2013"

Im guessing the correct interval should be:

 (0.000000, 1.000000) == (2013-05-01T00:00:00.000Z,
2013-06-01T00:00:00.000Z)

so:
  - we do have a bug
  - the dates are encoded wrong
- have to figure out how to fix systematically. if theres a way to derive the correct answer from the info in the PDS below, can you see it? It will be unfortunate if we have to rely on the filename. 



---------
File=0 E:/data/work/ansari/flxf.01.2012092700.201305.avrg.grib.grb2 
Header=""

Grib2IndicatorSection
 Discipline = (0) Meteorological products
 Length     = 64370

Grib2IdentificationSection
Center = (7) US National Weather Service, National Centres for 
Environmental Prediction (NCEP)
 SubCenter     = (0) null
 Master Table  = 2
 Local Table   = 1
 RefTimeSignif = 1 (Start of forecast)
 RefTime       = 2012-09-27T00:00:00.000Z
 RefTime Fields = 2012-9-27 0:0:0
 ProductionStatus      = 0 (Operational products)
 TypeOfProcessedData   = 1 (Forecast products)

Grib2GridDefinitionSection hash=-1824839391 crc=4229097007
 Length             = 72
 Source  (3.0)      = 0 (Specified in Code table 3.1)
 Npts               = 72960
 Template (3.1)     = 40
(3.40) Grid definition template 3.40 - Gaussian latitude/longitude 
1: GDS length == 72
5: Section == 3
6: Source of Grid Definition (see code table 3.0) == 0 (table 3.0: Specified in Code 
table 3.1)
7: Number of data points == 72960
11: Number of octects for 
optional list of numbers == 0
12: Interpretation of list of numbers == 0 (table 3.11: There is no 
appended list)
 13: Grid Definition Template Number == 40
15: Shape of the Earth == 6 (table 3.2: Earth assumed spherical with 
radius of 6 371 229.0 m)
 16: Scale factor of radius of spherical Earth == 0
 17: Scaled value of radius of spherical Earth == 0
21: Scale factor of major axis 
of oblate spheroid Earth == 0
22: Scaled value of major axis 
of oblate spheroid Earth == 0
26: Scale factor of minor axis 
of oblate spheroid Earth == 0
27: Scaled value of minor axis 
of oblate spheroid Earth == 0
 31: Ni - number of points along a parallel == 384
 35: Nj - number of points along a meridian == 190
 39: Basic angle of the initial production domain == 0
43: Subdivisions of basic angle used to define extreme longitudes and 
latitudes, and direction increments == 0
 47: La1 - latitude of first grid point == 89277000
 51: Lo1 - longitude of first grid point == 0
 55: Resolution and component flags == 48
 56: La2 - latitude of last grid point == -89277000
 60: Lo2 - longitude of last grid point == 359062000
 64: Di - i direction increment == 938000
68: N - number of parallels 
between a pole and the Equator == 95
 72: Scanning mode == 0
73: List of number of points along 
each meridian or parallel. == -9999

Grib2ProductDefinitionSection
Interval = (2012-08-27T00:00:00.000Z,2012-09-27T00:00:00.000Z)
(4.8) Product definition template 4.8 - average, accumulation and/or extreme values or other statistically-processed values at a horizontal level or in a horizontal layer in a continuous or non-continuous time 
interval
1: PDS length == 58
5: Section == 4
  6: Number of coordinates values after Template == 0
8: Product Definition Template Number == 8
 10: Parameter category == 2
 11: Parameter number == 17
 12: Type of generating process == 2 (table 4.3: Forecast)
13: Background generating process identifier (defined by originating centre) == 0 (table ProcessId: Table ProcessId 
code 0 not found)
14: Analysis or forecast generating process identifier (defined by originating centre) == 98 (table ProcessId: Table 
ProcessId code 98 not found)
 15: Hours after reference time of data cutoff == 0
 17: Minutes after reference time of data cutoff == 0
18: Indicator of unit of time range == 3 (table 4.4: Month) 
 19: Forecast time in units defined by octet 18 == 8
23: Type of first fixed surface == 1 (table 4.5: Ground or water 
surface)
 24: Scale factor of first fixed surface == 0
 25: Scaled value of first fixed surface == 0
29: Type of second fixed surface == 255 (table 4.5: Missing) 
 30: Scale factor of second fixed surface == 0
 31: Scaled value of second fixed surface == 0
 35: Year - time of end of overall time interval == 2012
 37: Month - time of end of overall time interval == 9
 38: Day - time of end of overall time interval == 27
 39: Hour - time of end of overall time interval == 0
 40: Minute - time of end of overall time interval == 0
 41: Second - time of end of overall time interval == 0
42: n - number of time range specifications describing the time intervals used to calculate the statistically-processed field == 1 43: Total number of data values 
missing in statistical process == 0
47: Statistical process used to calculate the processed field from the field at each time increment during the time range == 0 (table 
4.10: Average)
48: Type of time increment between successive fields used in the statistical processing == 2 (table 4.11: Successive times processed have same start time of forecast, forecast time is incremented) 49: Indicator of unit of time for time range over which 
statistical processing is done == 3 (table 4.4: Month)
50: Length of the time range over which statistical processing is 
done, in units defined by the previous octet == 1
54: Indicator of unit of time for the increment between 
the successive fields used == 255 (table 4.4: Missing)
55: Time increment between successive fields, in units 
defined by the previous octet == 0
59: As octets 47 to 58, next 
innermost step of processing == -9999
71: Additional time range specifications, included in accordance with the value of n. Contents as octets 47 to 58, repeated as necessary == 
-9999

Grib2SectionDataRepresentation
Template = 40 (Grid point data - JPEG 2000 code stream 
format)
  NPoints            = 72960

Grib2SectionData
  Starting Pos       = 196
  Data Length        = 64170


On 9/28/2012 11:45 AM, Steve Ansari wrote:

Hi Jack,
Thanks for bringing this up. It appears that the GRIB decoders in the NetCDF for Java library are not detecting the time dimension correctly. 20120927 should be the runtime and 201305 should be the time, but the runtime date is showing up as the time dimension and 
the valid time of the model is not present.
I'm cc-ing John Caron and Robb at Unidata who are the GRIB masters at 
Unidata.
I'm using NCJ 4.2 in the Weather and Climate Toolkit, but I tested 
several files with NCJ 4.3 and got the same results.
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01 

John, Robb, thanks for your help!

Thanks!
Steve




On Fri, Sep 28, 2012 at 1:03 PM, Jack Settelmaier
<address@hidden <mailto:address@hidden>> wrote:
I'm poking around with displaying output from the climate (CFS) 
    model, and it's forecasts of mean monthly conditions.
So, in the below files, one gets the CFS model output from the Sep 26, 2012 18Z model run. The forecasts from that model run are for the mean conditions over the months of May and June, 
    2013, respectively.
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01/flxf.01.2012092700.201305.avrg.grib.grb2 ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01/flxf.01.2012092700.201306.avrg.grib.grb2
When I Save a single KMZ after viewing each dataset, your software apparently assigns a "time stamp" based on first date in the filename, or from pulling the model initial time, NOT the valid time as I'd like. So what I end up with in these two files http://www.srh.noaa.gov/rtimages/srh/stsd/Jack/CFS_500HAnom4.kmz http://www.srh.noaa.gov/rtimages/srh/stsd/Jack/CFS_500HAnom5.kmz
are two images that have the same "time" in Google, even though their valid time is a month apart. Maybe if I made a loop over the 2 grib files it would have inserted a "time step" that would allow for some sort of animation in Google, but I'm still not 
    sure it would help show me the valid time.
This is pretty specific to the CFS files and their nuanced forecasts and such, but is there something that could be done to 
    better the result?