[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: Fw: decoders] Grib1 Indexer and GDS info (fwd)
This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
- Subject: RE: Fw: decoders] Grib1 Indexer and GDS info (fwd)
- Date: Thu, 11 Sep 2008 11:29:31 -0600 (MDT)
Pelin,attached is Grib1ExtractRawData.java class. Put it in the grib1 dir and compile. it will extract the first record if not parm # is given or all the records of a certain parm, ie
% java -Xmx256m ucar/grib/grib1/Grib1ExtractRawData RUC.grib1 > RUC1rec.grib1
or % java -Xmx256m ucar/grib/grib1/Grib1ExtractRawData RUC.grib1 7 > RUCparm7.grib1can you create a file either 1 record or one with a reasonable # of parms so i can use it as a test case.
RObb... On Wed, 10 Sep 2008, Bali, Pelin wrote:
Robb, I can attach the original ecmwf grib file if you want me to do so. It is about 35MB... When I zip it, it didn't make much difference. Please let me know how you want to proceed. If I read the dx and dy values by using readShort method in the Grib1GridDefinitionSection.java this is how grid definition section looks like in the grib index file: GDSkey = 1015536804 grid_type = 0 grid_name = Latitude/Longitude Grid grid_shape_code = 0 grid_shape = spherical grid_radius_spherical_earth = 6367.47 Nx = 720 Ny = 171 La1 = 65.0 Lo1 = 180.0 ResCompFlag = 128 Winds = Relative La2 = -20.0 Lo2 = 179.5 Dx = -3.071 Dy = -3.071 ScanningMode = 0 -------------------------------------------------------------------- GDSkey = 3838746175 grid_type = 0 grid_name = Latitude/Longitude Grid grid_shape_code = 0 grid_shape = spherical grid_radius_spherical_earth = 6367.47 Nx = 720 Ny = 171 La1 = 65.0 Lo1 = 180.0 ResCompFlag = 128 Winds = Relative La2 = -20.0 Lo2 = 179.5 Dx = -3.071 Dy = -3.071 ScanningMode = 0 -------------------------------------------------------------------- GDSkey = 1388487965 grid_type = 0 grid_name = Latitude/Longitude Grid grid_shape_code = 0 grid_shape = spherical grid_radius_spherical_earth = 6367.47 Nx = 360 Ny = 181 La1 = 90.0 Lo1 = 180.0 ResCompFlag = 128 Winds = Relative La2 = -90.0 Lo2 = 179.0 Dx = -6.141 Dy = -6.141 ScanningMode = 0 If I read the same variables by using readInt2 method in the Grib1GridDefinitionSection.java, I got the following values: GDSkey = 1015536804 grid_type = 0 grid_name = Latitude/Longitude Grid grid_shape_code = 0 grid_shape = spherical grid_radius_spherical_earth = 6367.47 Nx = 720 Ny = 171 La1 = 65.0 Lo1 = 180.0 ResCompFlag = 128 Winds = Relative La2 = -20.0 Lo2 = 179.5 Dx = 0.5 Dy = 0.5 ScanningMode = 0 -------------------------------------------------------------------- GDSkey = 3838746175 grid_type = 0 grid_name = Latitude/Longitude Grid grid_shape_code = 0 grid_shape = spherical grid_radius_spherical_earth = 6367.47 Nx = 720 Ny = 171 La1 = 65.0 Lo1 = 180.0 ResCompFlag = 128 Winds = Relative La2 = -20.0 Lo2 = 179.5 Dx = 0.5 Dy = 0.5 ScanningMode = 0 -------------------------------------------------------------------- GDSkey = 1388487965 grid_type = 0 grid_name = Latitude/Longitude Grid grid_shape_code = 0 grid_shape = spherical grid_radius_spherical_earth = 6367.47 Nx = 360 Ny = 181 La1 = 90.0 Lo1 = 180.0 ResCompFlag = 128 Winds = Relative La2 = -90.0 Lo2 = 179.0 Dx = 1.0 Dy = 1.0 ScanningMode = 0 Sincerely, Pelin Bali -----Original Message----- From: Robb Kambic [mailto:address@hidden] Sent: Tuesday, September 09, 2008 5:25 PM To: Bali, Pelin Cc: decoders Subject: RE: Fw: decoders] Grib1 Indexer and GDS info (fwd) On Thu, 28 Aug 2008, Bali, Pelin wrote:Robb, I've been registered to address@hidden and I was sending my questions to 'address@hidden' all the time. I don't knowhowit is being handled from there and how it falls into your spam. I started to think that you were ignoring my messages. <:) After sending this message, I made a little change in my code to read Grib1GridDefinitionSection.params field to get HashMap of the parameters. This helped me to get all the information I needed.However,in the grib1 index file, the Dx and Dy fields of the grid definition section was still displaying wrong numbers. My colleague and I started looking into the Grib1GridDefinitionSection.java. In the section where the information obtained from octets 24-25 (Dx) and octets 26-27(Dy or np), they were being read as "short". We changed them to read by using GribNumbers.int2 method (please see the following code snippet). It started spitting out the correct numbers. Sincerely, Pelin Bali.Pelin, Here's my findings so far, the only time readSHort returns a wrong value is the missing value scenario. ie when byte 1 and 2 = 255. readShort returns -0.0010 and GribNumbers.int2 return -9999. Is this what you are seeing? If not could you make available a file that i can test. thanks, robb...case 202 : // octets 18-20 (La2 - latitude of last grid point) lat2 = GribNumbers.int3(raf) / 1000.0; // octets 21-23 (Lo2 - longitude of last grid point) lon2 = GribNumbers.int3(raf) / 1000.0; // octets 24-25 (Dx - Longitudinal Direction Increment ) //------------------------------------------------------ // This is where we made the change. //------------------------------------------------------- //dx = raf.readShort() / 1000.0; dx = GribNumbers.int2(raf) / 1000.0; // octets 26-27 (Dy - Latitudinal Direction Increment ) // Np - parallels between a pole and the equator if (type == 4) { // np = raf.readShort(); np = GribNumbers.int2(raf); } else { // dy = raf.readShort() / 1000.0; dy = GribNumbers.int2(raf) / 1000.0; } //----------------------------------------------------------- // End of our changes. // ---------------------------------------------------------- // octet 28 (Scanning mode) See Table 8 scan = raf.read(); // octet 29-32 reserved reserved = raf.readInt(); if (P_VorL != 255) { if( NV == 0 || NV == 255 ) { getPL(raf); } else { getPV(NV, raf); } } break; // end Latitude/Longitude grids -----Original Message----- From: Robb Kambic [mailto:address@hidden] Sent: Wednesday, August 27, 2008 3:20 PM To: Bali, Pelin Cc: decoders Subject: Re: Fw: decoders] Grib1 Indexer and GDS info (fwd) Bali, your support msg has been sitting in my spam folder for months now, i just clean it out so that;s the reason for delay. There is an onlinesupporton unidata's home page and it's more dependable. I going to address your msg in a general manner until we get to thesamelevel. The Grib decoder package was meant to only be a low level API, the data resolution that you are looking for would be easier to use a ToolsUI program the uses the Grib package. Actually i think you should be able to do the refinement w/o doing any coding. The ToolsUI program is located at: https://www.unidata.ucar.edu/software/netcdf-java/ It will read in the file and it has options to select areas of data,thedocumentation is a little scarce but intuitive. The is also NCML that will let you manipulate the output files. On the other hand: The grib package provides a dump, indexer and getData program that can be used as templates to get the info you were looking for. There are javadocs on all these routines on the decoders page. There is also a display package IDV that could do the data processing. Hope this helps. RObb...---------- Forwarded message ---------- Date: Tue, 5 Aug 2008 10:59:21 -0400 From: "Bali, Pelin" <address@hidden> To: address@hidden Subject: decoders] Grib1 Indexer and GDS info I am using the Grib1Indexer to read the Grib file. I need to get the information lat, long, and nxny (145 X 73) as indicated below. Itriedto use Index.GdsRecord, however I wasn't successful. Although I could get start points of lat and long, I wasn't able to get neither resolution nor the end points of lat & long by using gdsRecord. Could you please show me some that direction on how to get thisinformation?Thanks in advance. ------------ rec 98:1554711:date 2008080400 TMP kpds5=11 kpds6=105 kpds7=2 levels=(0,2) grid=16 2 m above gnd 24hr fcst: TMP=Temp. [K] timerange 0 P1 24 P2 0 TimeU 1 nx 145 ny 73 GDS grid 0 num_in_ave0missing 0 center 1 subcenter 0 process 45 Table 1 scan: WE:SN winds(N/S) latlon: lat -90.000000 to 90.000000 by 2.500000 nxny 10585 long 0.000000 to 360.000000 by 2.500000, (145 x 73) scan 64 mode 128 bdsgrid 1-- The following is how I use the grib indexer to get the grib data. RandomAccessFile raf = new RandomAccessFile(filePath, "r"); Grib1Indexer indexer = new Grib1Indexer(); PrintStream ps = new PrintStream(indexPath); indexer.writeFileIndex(raf, ps, false ); Index index = new Index(); boolean rc = index.open(indexPath); ArrayList gribIndexRecords = index.getGribRecords(); ArrayList<GdsRecord> gdsRecords = index.getHorizCoordSys(); // // Build a hashmap for the GDS keys. // java.util.HashMap gdsMap = new java.util.HashMap(); for (int a = 0; a < gdsRecords.size(); a++) { GdsRecord gdsRecord = gdsRecords.get(a); gdsMap.put(gdsRecord.gdsKey, gdsRecord); } for (Iterator it = gribIndexRecords.iterator (); it.hasNext (); ) { zz++; GribRecord gribRecord = (GribRecord)it.next(); if (gribRecord != null) { // // Select only temperature. // if (gribRecord.paramNumber == 11 ) { if (gribRecord.levelType1 == 105) { if (gribRecord.levelValue1 == 2.0) { if (gribRecord.forecastTime == 24) { Grib1Data grib1data = new Grib1Data(raf); if (gdsMap.get(gribRecord.gdsKey) != null) { GdsRecord gds = (GdsRecord)gdsMap.get(gribRecord.gdsKey); System.out.println("Dx: " + gds.dx); System.out.println("Dy: " + gds.dy); System.out.println("Grid Shape Code: " + gds.grid_shape_code); System.out.println("Grid Type: " + gds.grid_type); System.out.println("La1: " + gds.La1); System.out.println("LaD: " + gds.LaD); System.out.println("Latin1: " + gds.latin1); System.out.println("Latin2: " + gds.latin2); System.out.println("Lo1: " + gds.Lo1); System.out.println("LoV: " + gds.LoV); System.out.println("Major Axis Earth " + gds.major_axis_earth); System.out.println("Minor Axis Earth " + gds.minor_axis_earth); System.out.println("Nx: " + gds.nx); System.out.println("Ny: " + gds.ny); System.out.println("Radius Spherical Earth: " + gds.radius_spherical_earth); System.out.println("Resolution: " + gds.resolution); System.out.println("Winds: " + gds.winds); } float[] arr = grib1data.getData(gribRecord.offset1, gribRecord.decimalScale, gribRecord.bmsExists); } // hour 24 } // 2m } //above ground } // end of temperature } // grib record is not null } // end for Sincerely, Pelin Bali WeatherPredict Consulting, Inc. 3200 Atlantic Avenue, Suite 114 Raleigh, NC 27604 Phone (Direct): 919-239-8833 Phone (Main) : 919-876-3633 Fax : 919-876-4469 "Imagination will often carry us to worlds that never were. But without it we go nowhere." - Carl Sagan (Cosmos - 1980)=============================================================================== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research address@hidden WWW: http://www.unidata.ucar.edu/======================================================================================================================================================= ======= Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research address@hidden WWW: http://www.unidata.ucar.edu/ ======================================================================== =======
=============================================================================== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research address@hidden WWW: http://www.unidata.ucar.edu/ ===============================================================================
/*
* Copyright 1997-2004 Unidata Program Center/University Corporation for
* Atmospheric Research, P.O. Box 3000, Boulder, CO 80307,
* address@hidden.
*
* This library is free software; you can redistribute it and/or modify it
* under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation; either version 2.1 of the License, or (at
* your option) any later version.
*
* This library is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
* General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with this library; if not, write to the Free Software Foundation,
* Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/
// $Id: Grib1ExtractRawData.java,v 1.33 2006/04/28 16:19:14 rkambic Exp $
package ucar.grib.grib1;
import ucar.grib.*;
import ucar.unidata.io.RandomAccessFile;
import java.io.IOException;
import java.io.DataOutputStream;
/*
* Grib1ExtractRawData.java 1.0 09/11/2008
* @author Robb Kambic
*
*/
import java.util.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* A class that scans a GRIB file to extract product information.
*/
public final class Grib1ExtractRawData {
/*
* raf for grib file
*/
/** _more_ */
private ucar.unidata.io.RandomAccessFile raf = null;
/*
* the header of Grib record
*/
/** _more_ */
private String header = "GRIB";
/**
* Pattern to extract header.
*/
private static final Pattern productID =
Pattern.compile("(\\w{6} \\w{4} \\d{6})");
// *** constructors *******************************************************
/**
* Constructs a <tt>Grib1Input</tt> object from a raf.
*
* @param raf with GRIB content
*
*/
public Grib1ExtractRawData(RandomAccessFile raf) {
this.raf = raf;
}
/**
* scans a Grib file to gather information that could be used to
* create an index or dump the metadata contents.
*
* @param parmNumber
* @throws NotSupportedException
* @throws IOException if raf does not contain a valid GRIB record
*/
public final boolean scan( int parmNumber)
throws NotSupportedException, IOException {
long start = System.currentTimeMillis();
// stores the number of times a particular GDS is used
HashMap gdsCounter = new HashMap();
Grib1ProductDefinitionSection pds = null;
Grib1GridDefinitionSection gds = null;
long gdsOffset = 0;
DataOutputStream dos = new DataOutputStream( System.out );
//System.out.println("file position =" + raf.getFilePointer());
while (raf.getFilePointer() < raf.length()) {
if (seekHeader(raf, raf.length())) {
// Read Section 0 Indicator Section
Grib1IndicatorSection is = new Grib1IndicatorSection(raf);
//System.out.println( "Grib record length=" +
is.getGribLength());
// EOR (EndOfRecord) calculated so skipping data sections is
faster
long EOR = raf.getFilePointer() + is.getGribLength()
- is.getLength();
long SOR = raf.getFilePointer() - is.getLength();
// skip Grib 2 records in a Grib 1 file
if( is.getGribEdition() == 2 ) {
//System.out.println( "Error Grib 2 record in Grib1 file" )
;
raf.seek(EOR);
continue;
}
if( parmNumber == -1 ) { // extract only 1st record
raf.seek( SOR );
byte[] oneRecord = new byte[ (int)is.getGribLength() ];
raf.read( oneRecord );
dos.write( oneRecord, 0, oneRecord.length );
dos.flush();
break;
}
long dataOffset = 0;
try { // catch all exceptions and seek to EOR
// Read Section 1 Product Definition Section PDS
pds = new Grib1ProductDefinitionSection(raf);
if (pds.getLengthErr()) {
raf.seek(EOR);
continue;
}
if( parmNumber == pds.getParameterNumber() ) {
raf.seek( SOR );
byte[] oneRecord = new byte[ (int)is.getGribLength() ];
raf.read( oneRecord );
dos.write( oneRecord, 0, oneRecord.length );
dos.flush();
}
raf.seek( EOR );
continue;
} catch( Exception e ) {
//.println( "Caught Exception scannning record" );
e.printStackTrace();
raf.seek(EOR);
continue;
}
} // end if seekHeader
//System.out.println( "raf.getFilePointer()=" +
raf.getFilePointer());
//System.out.println( "raf.length()=" + raf.length() );
} // end while raf.getFilePointer() < raf.length()
//System.out.println("GribInput: processed in " +
// (System.currentTimeMillis()- start) + " milliseconds");
dos.close();
return true;
} // end scan
/**
* Grib edition number 1, 2 or 0 not a Grib file.
* @throws NotSupportedException
* @throws IOException
* @return int 0 not a Grib file, 1 Grib1, 2 Grib2
*/
public final int getEdition() throws IOException, NotSupportedException {
int check = 0; // Not a valid Grib file
long length = (raf.length() < 4000L)
? raf.length()
: 4000L;
if ( !seekHeader(raf, length)) {
return 0; // not valid Grib file
}
// Read Section 0 Indicator Section to get Edition number
Grib1IndicatorSection is = new Grib1IndicatorSection(raf); // section 0
return is.getGribEdition();
} // end getEdition
/**
* _more_
*
* @param raf _more_
* @param stop _more_
*
* @return _more_
*
* @throws IOException _more_
*/
private boolean seekHeader(RandomAccessFile raf, long stop)
throws IOException {
// seek header
StringBuffer hdr = new StringBuffer();
int match = 0;
while (raf.getFilePointer() < stop) {
// code must be "G" "R" "I" "B"
char c = (char) raf.read();
hdr.append((char) c);
if (c == 'G') {
match = 1;
} else if ((c == 'R') && (match == 1)) {
match = 2;
} else if ((c == 'I') && (match == 2)) {
match = 3;
} else if ((c == 'B') && (match == 3)) {
match = 4;
Matcher m = productID.matcher(hdr.toString());
if (m.find()) {
header = m.group(1);
} else {
//header = hdr.toString();
header = "GRIB1";
}
//System.out.println( "header =" + header.toString() );
return true;
} else {
match = 0; /* Needed to protect against "GaRaIaB" case. */
}
}
return false;
} // end seekHeader
/**
* Outputs first record of raw data to STDOUT
* or parms based on discipline, category, and number
* @param args filename
* @throws IOException
**/
public static void main(String args[])
throws IOException, NotSupportedException {
String fileName;
int number = -1;
if (args.length == 1) {
fileName = args[0];
} else if( args.length == 2) {
fileName = args[0];
number = Integer.parseInt( args[1] );
} else {
System.out.println( "Not correct number of parms, either 1 or 2" );
return;
}
RandomAccessFile raf = new RandomAccessFile( fileName, "r" );
Grib1ExtractRawData erd = new Grib1ExtractRawData( raf );
erd.scan( number );
}
} // end Grib1ExtractRawData
- Prev by Date: [IDV #DGE-160974]: Problem reading grib1 files with bitmap present
- Next by Date: RE: Fw: decoders] Grib1 Indexer and GDS info (fwd)
- Previous by thread: RE: Fw: decoders] Grib1 Indexer and GDS info (fwd)
- Next by thread: RE: Fw: decoders] Grib1 Indexer and GDS info (fwd)
- Index(es):