Re: FW: Re: NCEP North American Reanalysis (fwd)

On Thu, 21 Oct 2004, Dan Swank wrote:

> Almost have it.
> The latest version indeed corrects the gribtocdl bug.
> I was able to create the narr-b .nc file going through both steps.
> Next i tried to convert the narr-a files, which have more then 400 grib
> records each.
> gribtocdl wrote this record:
>
>         float  03tot_atm(record,y,x) ;
>                03tot_atm:long_name = "Total ozone at entire atmosphere
> considered as a single layer" ;
>                03tot_atm:GRIB_parameter_number = 237 ;
>                03tot_atm:GRIB_level_flag = 200 ;
>                03tot_atm:units = "Kg/m2" ;
>                03tot_atm:_FillValue = -9999.f ;
>                03tot_atm:navigation = "nav" ;
>
> Gribtonc doesn't seem to like the variable name starting with a number
> zero, and throws a syntax error.

dan,

in netcdf no var can start with a number, guess that parameter hasn't been
used before.  the problem has been fixed and tested on the file you gave
as a test. sorry but you have to download the latest release:

decoders-3.0.6.tar.Z

robb...

ps the size of netcdf output file grew to ~625mg from ~56 mg input file.
that;s about right


> This is also strange becuase the NARR doesn' tcontain any ozone fields.
>
> Suggestions?  A sample narr-a file to try:
> http://nomads.ncdc.noaa.gov/tmp/narr-a_221_20010101_0000_000.grb
>
> -Dan
>
>
>
> Robb Kambic wrote:
>
> >On Thu, 21 Oct 2004, Dan Swank wrote:
> >
> >
> >
> >>Robb,
> >>
> >>No luck.  I've recompiled the decoders package again and gribtocdl does
> >>the same thing.
> >>We have:
> >>netcdf-3.5.1          (built from source)
> >>udunits-1.11.7       ( Used binary installation since our system had
> >>trouble compiling this from source)
> >>decoders-2.3.8n    ( built from source)
> >>
> >>
> >>
> >
> >Dan,
> >
> >that's a really old distribution of the decoders package. Your problem
> >sounds like a bug that was fixed back at that time. the latest decoders
> >package is decoders-3.0.5.tar.Z hopefully the newest distribution will fix
> >your problem.
> >
> >robb...
> >
> >
> >
> >
> >>And the output from the compilation can be found here
> >>http://nomads.ncdc.noaa.gov/tmp/decoders-2.3.8n.logs.tar
> >>
> >>OS:
> >>Red Hat Linux release 7.3 (Valhalla)
> >>
> >>We also have a
> >>Red Hat Linux release 8.0 (Psyche)
> >>We can try this if you think it will really make much of a difference.
> >>Avbl but we dont' have root user on either of these so installing
> >>tpackages can be a bit tricky.
> >>
> >>If you need anymore information let me know.
> >>
> >>-Dan
> >>
> >>
> >>
> >>Robb Kambic wrote:
> >>
> >>
> >>
> >>>no need to include the others now...
> >>>
> >>>
> >>>On Wed, 20 Oct 2004, Dan Swank wrote:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>Getting a little closer...
> >>>>
> >>>>Set the Env var and now udunits works as expected... however gribtocdl
> >>>>still gets stuck in an infinate loop and don't seem to work right.
> >>>>when i ran it on the narr-b GRIB it got to the following section
> >>>>
> >>>>
> >>>>// Navigation
> >>>>nav_model = "GRIB1" ;
> >>>>grid_type_code = 3 ;
> >>>>grid_type = "Lambert conformal projection" ;
> >>>>grid_name = " " ;
> >>>>grid_center = 7 ;
> >>>>grid_number = 221 ;
> >>>>i_dim = "lon" ;
> >>>>j_dim = "lat" ;
> >>>>Nx = 349 ;
> >>>>Ny = 277 ;
> >>>>La1 = 1.000000 ;
> >>>>Lo1 = -145.500000 ;
> >>>>Lov = -107.000000 ;
> >>>>Dx = 32463.000000 ;
> >>>>Dy = 32463.000000 ;
> >>>>// Pole in proj. plane = North ;
> >>>>centers = 1 ;
> >>>>Latin1 = 50.000000 ;
> >>>>Latin2 = 50.000000 ;
> >>>>Splat = 0.000000 ;
> >>>>SPlon = 0.000000 ;
> >>>>ResCompFlag = 0 ;
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>if you notice the cdl i made didn't have lon and lats.  there must be
> >>>something wrong with your gribtocdl build.  for a test, i build the
> >>>decoders package on a linux box fedora 9 and gribtocdl created the cdl
> >>>fine.  remake the distribution, if you can try it on another box. whats
> >>>your os?
> >>>
> >>>robb...
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>lon = -145.50,-146.50,-147.50,-148.50,-149.50,-150.50,-151.50,-152.50,
> >>>>     -153.50,-154.50,-155.50,-156.50,-157.50,-158.50,-159.50,-160.50,
> >>>>     -161.50,-162.50,-163.50,-164.50,-165.50,-166.50,-167.50,-168.50,
> >>>>
> >>>>Then continues decrementing the longitue values and don't stop.  I was
> >>>>forced to CNTL-C it and remove the lon section completely, putting the
> >>>>curly bracket at the end.  Know why it might do this?
> >>>>After setting the env var, gribtonc successfully re-created narr.nc from
> >>>>the narr.cdl you supplied.  However when I tried my CDL from above i get
> >>>>the following:
> >>>>
> >>>>./narr-b_221_20010101_0000_000.grb
> >>>>ncgen: narr-b2.cdl line 6: dimension length must be positive
> >>>>ncgen: narr-b2.cdl line 7: dimension length must be positive
> >>>>ncgen: narr-b2.cdl line 73: Nx not declared as a variable, fatal error
> >>>>
> >>>>Does gribtocdl require any ENV vars to be set?
> >>>>
> >>>>Appreciate the help.
> >>>>-Dan
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>Robb Kambic wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>On Wed, 20 Oct 2004, Dan Swank wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>I copied the narr.cdl file from the FTP and tried:
> >>>>>>
> >>>>>>gribtonc -v -l ./log -e ./error ./narr.cdl
> >>>>>>narr-b_221_20010101_0000_000.nc < narr-b_221_20010101_0000_000.grb
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>here's the process
> >>>>>
> >>>>># creates cdl file narr.cdl
> >>>>>% gribtocdl -v narr-b_221_20010101_0000_000.grb > narr.cdl
> >>>>>
> >>>>># creates netcdf file narr.nc  in verbose mode, logging to screen
> >>>>>% gribtonc -vl - narr.cdl narr.nc < narr-b_221_20010101_0000_000.grb
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Result
> >>>>>>Segmentation fault
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>explained below, set UDUNITS_PATH
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>and
> >>>>>>Oct 20 19:31:05 gribtonc[32594]: Starting Up
> >>>>>>in the ./log file.
> >>>>>>
> >>>>>>Any idea whats going on?  It is likely gribtonc (or more likely one of
> >>>>>>its dependancies) is not installed correctly on our system (RedHat 7.3)
> >>>>>>Also, what did you use to create this CDL file?  The data in cdl seems
> >>>>>>like a translation of the information in the grib PDS, reworked into a
> >>>>>>format that ncgen can understand.
> >>>>>>Any way i can get anymore debug information regarding this?
> >>>>>>
> >>>>>>Only hunch is that it is not happy with the udunits package
> >>>>>>
> >>>>>>-> /usr/local/udunits-1.11.7/udunits-1.11.7/bin/udunits
> >>>>>>udunits(3): Couldn't open units database "/upc/udunits/etc/udunits.dat":
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>ahh, that's the problem. udunits can't find udunits.dat  either place
> >>>>>udunits.dat in dir /upc/udunits/etc/ or set environment var
> >>>>>
> >>>>>% setenv UDUNITS_PATH /your/udunits/path/udunits.dat
> >>>>>
> >>>>>replace /your/udunits/path/ with appropriate path
> >>>>>
> >>>>>robb...
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>No such file or directory
> >>>>>>Segmentation fault
> >>>>>>
> >>>>>>But, while building the unidata decoders package it only seemed to want
> >>>>>>the .dat .a and .h files within these packages.
> >>>>>>Would this be the source of the problems?
> >>>>>>
> >>>>>>-Dan
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>Robb Kambic wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Thanks russ for the clarification.  i was assuming you were familar 
> >>>>>>>with
> >>>>>>>the decoders process. if i can answer any more questions let me know. 
> >>>>>>>i'll
> >>>>>>>try to be more descriptive.
> >>>>>>>
> >>>>>>>robb...
> >>>>>>>
> >>>>>>>
> >>>>>>>On Tue, 19 Oct 2004, Dan Swank wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>Russ
> >>>>>>>>
> >>>>>>>>Actually, i was, at first, trying
> >>>>>>>>GRIB -( gribtocdl )-> CDL -( gribtonc )-> NetCDF
> >>>>>>>>Which i now understand is completely wrong, thanks for the help.
> >>>>>>>>As you have noticed, we are completely unfamiliar with these programs.
> >>>>>>>>
> >>>>>>>>Attempting it the correct way now, i'll let you know how it goes.
> >>>>>>>>
> >>>>>>>>-Dan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>Russ Rew wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Robb,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>i downloaded the file from the last message, created a cdl, and 
> >>>>>>>>>>decoded
> >>>>>>>>>>the grb file. there might be something wrong with Dan's  decoders 
> >>>>>>>>>>build or
> >>>>>>>>>>it could be a platform issue. this was done on a solaris box 5.9  
> >>>>>>>>>>The
> >>>>>>>>>>narr.cdl file is attached and the files narr.cdl, narr.grb, and 
> >>>>>>>>>>narr.nc are
> >>>>>>>>>>in the Unidata's ftp dir at
> >>>>>>>>>>
> >>>>>>>>>>ftp unidata.ucar.edu
> >>>>>>>>>>
> >>>>>>>>>>% cd pub/contrib
> >>>>>>>>>>% mget narr*
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>Thanks Robb.  The file sizes are:
> >>>>>>>>>
> >>>>>>>>>-rw-rw-r--   1 rkambic  ustaff   3486764 Oct 19 13:32 narr.nc
> >>>>>>>>>-rw-rw-r--   1 rkambic  ustaff      8560 Oct 19 13:32 narr.cdl
> >>>>>>>>>-rw-rw-r--   1 rkambic  ustaff   1398914 Oct 19 13:32 narr.grb
> >>>>>>>>>
> >>>>>>>>>so the netCDF file is about 2.5 times as big as the GRIB file.
> >>>>>>>>>
> >>>>>>>>>I'm guessing the source of the problem may come from using
> >>>>>>>>>
> >>>>>>>>>GRIB -> (via gribtocdl) -> CDL -> (via ncgen) -> netCDF
> >>>>>>>>>
> >>>>>>>>>(Using gribtocdl to generate a very large CDL file and then using 
> >>>>>>>>>ncgen
> >>>>>>>>>to convert that into a netCDF file.)
> >>>>>>>>>
> >>>>>>>>>I think Robb used the following tools instead:
> >>>>>>>>>
> >>>>>>>>>GRIB -> (via gribtocdl) -> CDL
> >>>>>>>>>GRIB and CDL -> (via gribtonc) -> netCDF
> >>>>>>>>>
> >>>>>>>>>(Using gribtocdl to generate a small CDL file describing structure of
> >>>>>>>>>the desired netCDF file and then using gribtonc to convert the GRIB
> >>>>>>>>>data into the netCDF file.)
> >>>>>>>>>
> >>>>>>>>>--Russ
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>--
> >>>>>>>>Dan Swank <address@hidden>
> >>>>>>>>NOMADS programmer
> >>>>>>>>STG, Incorporated - Government Contractor
> >>>>>>>>151 Patton Avenue, Room 514
> >>>>>>>>Asheville, NC 28801
> >>>>>>>>Phone: 828-271-4007
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>===============================================================================
> >>>>>>>Robb Kambic                               Unidata Program Center
> >>>>>>>Software Engineer III                     Univ. Corp for Atmospheric 
> >>>>>>>Research
> >>>>>>>address@hidden            WWW: http://www.unidata.ucar.edu/
> >>>>>>>===============================================================================
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>--
> >>>>>>Dan Swank <address@hidden>
> >>>>>>NOMADS programmer
> >>>>>>STG, Incorporated - Government Contractor
> >>>>>>151 Patton Avenue, Room 514
> >>>>>>Asheville, NC 28801
> >>>>>>Phone: 828-271-4007
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>===============================================================================
> >>>>>Robb Kambic                                 Unidata Program Center
> >>>>>Software Engineer III                       Univ. Corp for Atmospheric 
> >>>>>Research
> >>>>>address@hidden              WWW: http://www.unidata.ucar.edu/
> >>>>>===============================================================================
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>--
> >>>>Dan Swank <address@hidden>
> >>>>NOMADS programmer
> >>>>STG, Incorporated - Government Contractor
> >>>>151 Patton Avenue, Room 514
> >>>>Asheville, NC 28801
> >>>>Phone: 828-271-4007
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>===============================================================================
> >>>Robb Kambic                                   Unidata Program Center
> >>>Software Engineer III                         Univ. Corp for Atmospheric 
> >>>Research
> >>>address@hidden                WWW: http://www.unidata.ucar.edu/
> >>>===============================================================================
> >>>
> >>>
> >>>
> >>>
> >>--
> >>Dan Swank <address@hidden>
> >>NOMADS programmer
> >>STG, Incorporated - Government Contractor
> >>151 Patton Avenue, Room 514
> >>Asheville, NC 28801
> >>Phone: 828-271-4007
> >>
> >>
> >>
> >>
> >
> >===============================================================================
> >Robb Kambic                             Unidata Program Center
> >Software Engineer III                           Univ. Corp for Atmospheric 
> >Research
> >address@hidden                  WWW: http://www.unidata.ucar.edu/
> >===============================================================================
> >
> >
>
> --
> Dan Swank <address@hidden>
> NOMADS programmer
> STG, Incorporated - Government Contractor
> 151 Patton Avenue, Room 514
> Asheville, NC 28801
> Phone: 828-271-4007
>
>

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================