20060125: 20060124: CONDUIT data stream

Jennie,

See embedded responses......




>From: address@hidden (Jennie L. Moody)
>Organization: UCAR/Unidata
>Keywords: 200601251548.k0PFmW7s005926

>On Jan 24, 15:23, Unidata Support wrote:
>> Subject: 20060124: CONDUIT data stream
>> 
>> Jennie,
>> 
>> I think you are confusing a graphic of grid #104 (for NAM) with
>> grid #4 which is a global .5x.5 degree lat/lon grid.
>
>Oops, your're right, my mistake.
>
>> 
>> The GRID #4 is the same lat/lon projection as GRID #3, just with
>> row/columns at .5 degree rather than 1 degree (eg 4x as many points).
>> GRIB2 is much better in regard to file size for the larger grid
>> size. In fact, NCEP will be transitioning all of its data to
>> GRIB2, except that which they have to provide to the GTS under WMO
>> agreements.
>> 
>> NCEP provides libraries to decode GRIB2 (as well as convert GRIB2 to GRIB1).
>  
>> See: http://www.nco.ncep.noaa.gov/pmb/codes/GRIB2/
>
>Okay, this all is fine, and maybe even convinces me that I should just start
>right now working with the 0.5x0.5 deg grid, especially if I can get it
>decoded.  So, two more questions.  If I start running GEMPAK on this new 
>machine to view these grids (and or the IDV), will they read the GRIB2
>files, or do they have decoders that will allow me to import the grids into
>the application?  Even though I am running NASA trajectory code, I also want t
> o 
>make grid products, cross sections, etc.  I have always only used McIDAS befor
> e,
>but Tom encouraged me to look at either of these (by the way, this is all goin
> g
>to be running on a new Dell 610 Workstation running RH Enterprise 4, its a 64
>bit machine, I should be able to build GEMPAK in this environment, right?



Yes, GEMPAK is able to decode the GRIB2 grids into GEMPAK grid files for 
display.
The IDV should also be able to read the GRIB2 through the java decoders/thredds 
interface.


>
>As I note, the main application I have for these grids is to convert them to 
>netcdf grids for reading them into the a NASA trajectory program.
>Is there still any kind of gribtonc decoder that will automatically decode the
>GRIB2 files into netcdf?  Ultimately, since I am running a reverse domain
>filling trajectory model, I could try to use a pattern action to just grab
>the winds, temperatures, heights I need and might be able to just pipe
>the data to a gribtonc decoder to get back netcdf files. Either way, I would 
>not try to suck up 180 hours, but would just use 72 or maybe 96.  



I don't think gribtonc has been maintained in quite some time. Robb and
the thredds group are doing things through Java. Given the voluminous size
of a NetCDF file, I think the tredds interface is going direct from GRIB,
but Robb would be able to tell you what he can do as far as GRIB2 to NetCDF.



>> 
>> Currently, we are sending the same GFS data at 2 resolutions
>> (grid #3 and grid #4) for F000-F084. That consumes a lot of user bandwidth.
>> Now that NCEP is running the GFS model at T384 through 180 hours, they will
>> be able to produce the top resolution grid #4 through 180 hours, so
>> CONDUIT will want to transmit that when available. At that time, the complet
> e
>> 0-180 GFS data set will be redundant on grid 3 so it makes sense to remove
>> that and use the bandwith for other data sets. The GRIB2 .5 degree data set 
> will 
>> have been available for over a year and a half, giving users adequate time t
> o 
>> transition legacy code to the higher resolution.
>
>Well, wouldn't you know that when I start to want to work with this, my need a
> rises
>right at the transition point.  Is there a specific date set for when NCEP exp
> ects to make this 
>change?  From the documents on the web, it sounds like the moratorium on model
>  changes
>gets lifted in March/April time frame, and then there is a 30-day implementati
> on?
>I am trying to do work for a NASA/NCAR aircraft field campaign over the N.E. P
> acific 
>running from mid-April to mid-May.  We will try to have products running to pr
> ovide some support
>to another campaign that runs in the month of March (again, NASA and NCAR/NSF)



The moratorium expiration will allow NCEP to begin producing the GRID #4 data 
set through 
180 hours probably in the March-April time frame. Given that the combined volume
of GRIDS #3 and #4 to 180 hours would be 4GB, that would be problematic. We'll 
discuss
the CONDUIT items at the meeting at the AMS to get user feedback on how soon 
they want the
data once its available.

My guess is that NCEP would keep producing the 1 degree, and possibly extend 
that to 384 hours
if the GFS T-resolution supports that. That would then make the 2.5 degree data 
set we
currently distribute from 180 to 384 hours 'so last week'. So, you shouldn't 
have any problem with
the availability of GRID #3 on the ftp servers. It would be a good plan to 
decode the fields/levels/times
you need to netCDF though, since you may be talking about 10-20GB for a NetCDF 
file otherwise.


> .
>
>> 
>> If you don't have the resources to transition to grid #4, or convert the GRI
> B2
>> to GRIB1 using the NCEP program mentioned above, you should also be aware th
> at
>> NCEP has introduced additional levels/parameters etc to the GFS output
>> (both GRID #3 and GRID #4), so you should ensure that legacy applications yo
> u might
>> be using with GFS grid #3 are able to handle those newer parametere/levels
>> (or at least not die during inopportune moments)!
>Okay, this leads to my final question.  The only way I know how to "see" all t
> he
>parameter levels that are being sent out in these models is to look at the 
>conduit data being received in the past 72 hours.
>http://motherlode.ucar.edu/cgi-bin/ldm/conduit_reception.csh
>I checked this last week, and this week as well, and so far, I haven't even se
> en
>any 0.5 degree model data being recieved (all the grids show up in red)
>http://motherlode.ucar.edu/cgi-bin/ldm/statusgen?SL.us008001/ST.opnt/MT.gfs_CY
> .12/RD.20060124/PT.grid_DF.gr2/fh.0000_tl.press_gr.0p5deg
>These are listing files with a grid number of #000, so maybe thats the problem




GRIB2 doesn't have a method for specifying a predefined grid number (#4 in this 
case)
when the grid is defined within the message, so that results in the #000 
identification.

The web page shows red for the GFS004 since the data is being stored on disk as 
single 
files rather than separate fields, so the cgi can't see the status of the 
individual fields. 
They are being received though. The fields are the same as GRID #3.



Also, 
> .  By the way,
>what is the SPEC_SSI designation prepended to the AVN, is this a special initi
> alization using
>sea-surface temperatures (I may be confabulating, ie., making up a memory, but
>  I thought I
>remembered reading something about a new initialization), and its only designa
> ted on the
>initial run, so that would make sense.  Anyway, is this an indicator of instab
> ility in
>the delivery of these grids (somethings broken?, things often break?), or have
>  these grids
>not officially been turned on yet (your note indicates that they should have b
> een
>being delivered via conduit for a year and a half).  Maybe I'm just a jinx, an
> d they are missing
>because I am looking for them ;-) ?


The GFS (spectral model) initialization grids are identified as the spectral 
statistical 
interpolation SSI. That is just the model ID in the GRIB identifying the 
interpolation to
the wave numbers.


>
>Looking forward to answers, and figuring out a path forward.
>Thanks Steve,

No worries,

Steve Chiswell
Unidata User Support



>
>Jennie
>> 
>> Steve Chiswell
>> Unidata User Support
>> 
>> 
>> 
>> 
>> 
>> >From: address@hidden (Jennie L. Moody)
>> >Organization: UCAR/Unidata
>> >Keywords: 200601242007.k0OK727s003856
>> 
>> >Hey Steve,
>> >
>> >I talked with Tom Yoksas the end of last week about my need to
>> >gain access to GFS runs through the idd CONDUIT feed for an
>> >upcoming spring campaign.  I have some non-unidata applications
>> >(not IDS, GEMPAK, McIDAS) that I want to run using grids.  I have
>> >been trying to sort out just what grids are available on what
>> >projections, and in what format (I have some pre-processor code
>> >to read the grib1 format, but have never used the grib2).   
>> >
>> >Before I jump in and drown, I have been trying to read everything
>> >that seems relevant regarding the CONDUIT feed.  I just saw a
>> >link today to an agenda for a meeting that will be held next week
>> >in Atlanta(AMS).  It mentions some up-coming changes, or
>> >development plans for CONDUIT, changes that will be implemented
>> >by NCEP.  These have me a bit confused.  I will quote here from
>> >what I read at 
>> >http://www.unidata.ucar.edu/projects/CONDUIT/CONDUIT_development_plans.html
>> >
>> >GFS Grids: 30 days pending availability
>> >
>> >    * Increase GFS 0.5 degree global grids in GRIB2 from 84 hours
>> >to 180 hours.
>> >    * The 0.5 degree GFS data has been available in CONDUIT from
>> >0 to 84 hours since September 2004. The extension of the number
>> >of forecast hours available at ~37 MB per forecast time will
>> >yield a net 1.3 GB increase per model run.
>> >    * Remove redundant GFS 1.0 degree global grids in GRIB1 from
>> >0 to 180 hours at ~23 MB per forecast time will yield a net 1.4
>> >GB decrease per model run.
>> >    * NCEP plans to produce the 0.5 degree data set to 180 hours
>> >following the lifting of moratorium at a time to be determined. 
>> >
>> >So, this raises a few questions for me.
>> >Does it mean that I should not plan on working with (which on my
>> >end, means writing code to access the GFS data on a given grid
>> >with a given resolution) the GFS 1.0 degree global grids, since
>> >it sounds like they could actually GO AWAY (be removed) in the
>> >March-April-May time frame (exactly when I would like to be
>> >looking at/using them?)
>> >
>> >My understanding of the GFS grids available on CONDUIT (all
>> >gleaned from Unidata web pages at this point) leads me to believe
>> >that the 1 degree data and the 0.5 degree data are on pretty
>> >different projections (the graphic you link to that shows model
>> >grids indicates that GFS 003 is mercator, and GFS 004 is
>> >conformal or something like that).  Also, as I noted, these two
>> >different GFS grids (the 1 degree and the 0.5 degree) are in two
>> >different formats, Grib1 and Grib2.  
>> >
>> >I was thinking that I would go ahead and set up a new ldm on my
>> >new workstation, and get just one feed going to pull in the GFS 1
>> >degree mercator grid files.  This would be a really bad idea if
>> >these grids are being sunset.
>> >
>> >Please let me know your thoughts, and if I have misunderstood
>> >things (I hope I have, I would really like to proceed down the
>> >path noted above)!
>> >
>> >Thanks.  
>> >Jennie
>> >address@hidden
>> >
>> --
>> ****************************************************************************
>> Unidata User Support                                    UCAR Unidata Program
>> (303)497-8643                                                  P.O. Box 3000
>> address@hidden                                   Boulder, CO 80307
>> ----------------------------------------------------------------------------
>> Unidata WWW Service              http://my.unidata.ucar.edu/content/support 
>> ----------------------------------------------------------------------------
>> NOTE: All email exchanges with Unidata User Support are recorded in the
>> Unidata inquiry tracking system and then made publicly available
>> through the web.  If you do not want to have your interactions made
>> available in this way, you must let us know in each email you send to us.
>> -- End of excerpt from Unidata Support <address@hidden>
>
>
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.