[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: trouble decoding Bufr msg

Mitch,
Here's my status, I can decode the MOS Guidance files up to the point of the actual data in section 4, the data section. I can get the length of section 4, skip one octet, then my decoding process gives me wrong values. 

I attached a sample file:

                      Header : JSML33 KWNO 150000
                Reference Time : 2008-08-15T00:00:00Z
                  Nominal Time : 2008-08-15T00:00:00Z
                     Center Id : 7
                 Sub Center Id : 0
                   Center Name : US National Weather Service (NCEP)
                      Category : 0 Surface data - land
                  Sub Category : 0 Unknown
                  Master table : NCEPtable-ABD.diff
                  isCompressed : true


here's the list of descriptors I decode:
First descriptors :[0-1-63, 0-5-2, 0-6-2, 0-4-1, 0-4-2, 0-4-3, 0-4-4, 0-8-21, 0-1-32, 0-2-200, 0-4-24, 0-8-13, 0-12-21, 0-8-13, 0-12-22, 0-8-13, 0-12-104, 0-12-106, 0-11-1, 0-11-200, 0-11-3, 0-11-4, 0-60-1, 0-60-2, 0-60-3, 0-60-58, 0-60-59, 0-60-60, 0-60-61, 0-60-62, 0-60-63, 0-60-150, 0-60-151, 0-60-152, 0-60-20, 0-60-21, 0-60-22, 0-60-91, 0-60-4, 0-60-7, 0-60-10, 0-60-13, 0-4-31, 0-60-90, 0-60-5, 0-60-8, 0-60-11, 0-60-14, 0-60-16, 0-4-31, 0-60-90, 0-60-6, 0-60-9, 0-60-12, 0-60-15, 0-60-17, 0-4-31, 0-60-90, 0-60-144, 0-60-31, 0-60-33, 0-60-35, 0-60-145, 0-4-31, 0-60-93, 0-60-100, 0-60-108, 0-60-109, 0-60-103, 0-60-104, 0-20-11, 0-60-42, 0-60-43, 0-60-44, 0-60-80, 0-60-81, 0-60-46, 0-60-47, 0-60-48, 0-60-94, 0-60-49, 0-60-112, 0-60-118, 0-60-113, 0-60-114, 0-60-115, 0-60-190, 0-60-57, 0-60-55, 0-60-116, 0-60-117, 0-60-54, 0-60-96, 0-60-23, 0-60-24, 0-60-25, 0-60-92, 0-60-148, 0-60-149] 
--------------------------------------------------------------------

The first descriptor is 0-1-63, ie
  0-01-063    ICAO location identifier                            CCITT IA5 0
            0         64  from tableB

That states this is char data, 8 chars long.
The length of section 4 is 818664 octets seems correct since the file is 822602 bytes. 


so i read the first 8 chars and I get wrong values. I get non char data
and for the 2nd descriptor 0-1-5 lat, i get:

0-5-2 Latitude size =12285 varCount =1
 -1.7044362E7 3976000.0 type values
so there is something wrong with the way I'm interpreting the compressed char data or ... So my question is how is the char data entered in compressed format. My decoder can interpret almost all other bufr data so i know that there must be something small that I"m not accounting for? 

Thanks for your help,
RObb...

On Fri, 29 Aug 2008, Mitchell Weiss wrote:


Robb,
Just to repeat what Becky has already stated, our lab is responsible for encoding BUFR messages, but the decoding process is something we do not do. However, this does not mean that we don't check the decoded output of our BUFR messages, but we need help from AWIPS decoder specialist in order to this. My suggestion for dumping a BUFR file for a specific file is to do the following.
1. Have an AWIPS decoder person decode a BUFR file into a NETCDF formated file. 2. You can then use a utility called NCDUMP to dump data values from the NETCDF files. NCDUMP has a number of very interesting and powerful options for dumping 
a NETCDF file. They can be found at
http://www.unidata.ucar.edu/software/netcdf/docs/ncdump-man-1.html

I'm not sure if you have access to the NCDUMP utility but if you do, I would
suggest going this route.

I hope this helps

Mitch

Rebecca Cosgrove wrote:

Robb,
Before I send you to an AWIPS decoder person, let me pass you on to Mitch Weiss. He actually worked on the code we use to code up our MOS BUFR products and he's now heavily involved in the GFS LAMP BUFR products. I think he will be much more knowledgable about the bits and bytes of our BUFR products than I am. I told him you were using the JSML33 KWNO product. I hope he can help you work things out. 

Becky

Robb Kambic wrote:

On Fri, 29 Aug 2008, Rebecca Cosgrove wrote:


Robb,
I'm afraid I don't even know what CCITT means. But let me see if this helps 

Hi Becky,

CCITT just means 8 bit char data. From the Bufr table B at NCEP 0-1-63 is:
 0-01-063    ICAO location identifier                            CCITT IA5
0            0         64  ICLI

http://www.emc.ncep.noaa.gov/mmb/data_processing/bufrtab_tableb.htm
you at all. The first thing we're putting in that file is the call letter record. For the JSML33 product, the first few call letters are 
K1H2    K2WX    K8D3    K9V9    KAAA    KAAO    KABR    KACB
We treat each call letter as an 8 digit "thing".  So it's "K1H2    ".



i get that data correctly in my decoder.
GFS MOS GUIDANCE FOR K1H2 K2WX K8D3 K9V9 KAAA KAAO KABR KACB KACQ KADC KADG KADU KAEL KAFK KAIA KAID KAIG KAIO KAIT KAIZ KALN KALO KAMN KAMW KANE KANJ KANW KAPN KAQP KARB KARR KARV KASX KATW KATY KAUH KAUM KAUW KAWG KAXA KAXN KAZO KBAX KBBB KBBW KBDE KBEH KBFF KBFW KBIE KBIS KBIV KBJI KBKX KBLV KBMG KBMI KBNW KBRD KBRL KBTL KBUU KBWG KC09 KC75 KCAD KCAV KCBF KCBG KCCY KCDD KCDJ KCDR KCFV KCGI KCGX KCID KCIN KCIU KCKC KCKN KCLI KCMI KCMX KCMY KCNC KCNK KCNU KCOQ KCOU KCPS KCQM KCSQ KCUT KCVG KCVX KCWA KCWI KD07 KDBQ KDDC KDEC KDEH KDET KDIK KDKB KDLH KDLL KDMO KDNS KDNV KDPA KDSM KDTL KDTW KDUH KDVL KDVN KDXX KDYT KEAR KEAU KEBS KEFT KEGV KEHA KEHR KELO KEMP KENL KENW KEOK KERY KESC KEST KETB KETH KEVM KEVV KEWK KEYE KFAM KFAR KFBL KFCM KFEP KFET KFFL KFFM KFFT KFGN KFKA KFLD KFNB KFNT KFOA KFOD KFOE KFOZ KFPK KFRM KFSD KFSE KFSW KFTK KFWA KFWC KGBD KGBG KGCK KGEZ KGFK KGHW KGLD KGLR KGNA KGOV KGPZ KGRB KGRI KGRR KGSH KGUS KGYL KGYY KHCD KHCO KHDE KHEI KHIB KHLC KHNB KHNR KHON KHOP KHSB KHSI KHTL KHUF KHUT KHYR KHYS KHYX KIAB KIBM KICL KICT KIEN KIGQ KIIB KIJX KIKK KIKV KILL KIML KIMT KIND KINL KIOW KIRK KIRS KISN KISQ KISW KIWD KIXD KJEF KJKJ KJKL KJLN KJMR KJMS KJOT KJVL KJXN KJYG KJYM KJYR KLAF KLAN KLBF KLBL KLDM KLEX KLJF KLNK KLNR KLOT KLOU KLOZ KLRJ KLSE KLVN KLWC KLWD KLWV KLXL KLXN KLXT KMBG KMBL KMBS KMCD KMCI KMCK KMCW KMDH KMDW KMDZ KMFI KMGG KMGN KMHE KMHK KMIB KMIC KMIE KMIW KMJQ KMKC KMKE KMKG KMKT KMLE KMLI KMML KMNM KMOP KMOT KMOX KMPZ KMQB KMQT KMRJ KMSN KMSP KMTC KMTO KMTW KMUT KMVE KMVN KMWA KMWM KMXO KMZH KN60 KODX KOEB KOEO KOFF KOFK KOGA KOJC KOKK KOLU KOLY KOLZ KOMA KONA KONL KONZ KORB KORC KORD KOSC KOSH KOTG KOTM KOVL KOVS KOWA KOXV KOZW KP28 KP58 KP59 KP75 KPAH KPBH KPDC KPEA KPHN KPHP KPIA KPIR KPKD KPLN KPNM KPNT KPOF KPPF KPPQ KPQN KPRG KPTK KPWC KPWK KRAC KRAP KRCA KRDK KRDR KRFD KRGK KRHI KRMY KRNH KROS KROX KRPD KRPJ KRQB KRRL KRRT KRSL KRST KRWF KRYV KRZN KSAR KSAW KSAZ KSBM KSBN KSDA KSDF KSET KSFD KSGF KSGS KSHL KSJX KSLB KSLH KSLN KSLO KSME KSNY KSPI KSPW KSQI KSTC KSTE KSTJ KSTL KSTP KSUE KSUS KSUW KSUX KSZL KTAZ KTBN KTEW KTIP KTNU KTOB KTOP KTQE KTTF KTVC KTVF KTWM KUES KUGN KUIN KULM KUNO KUNU KVIH KVOK KVPZ KVTI KVTN KVVV KVWU KVYS KWLD KXVG KYIP KYKN 45001 45002 45003 45004 45006 45007 45008 DISW3 LSCM4 PILM4 ROAM4 SGNW3 STDM4
We don't actually have code to decode our BUFR. We send the products to the AWIPS developers here in the NWS and they have a decoder.
Do you have an awips contact person so i can send a detail msg with the metadata from the file. 

Thanks for your help,
RObb...





If me giving you
the call letters doesn't clear anything up, can you write me an email that describes the whole situation as it stands right now as if you hadn't asked me anything. I might be able to send this on to someone who can help you out, but they'll kind of need the questions from the beginning. 

Becky

Robb Kambic wrote:


any progress...

Thanks,
Robb...




On Wed, 27 Aug 2008, Rebecca Cosgrove wrote:


Robb,
We're having problems with our supercomputer, so I can't get on and take a look at any of this for you today. I hope to be able to take a look tomorrow afternoon. 

Sorry for the delay, but I will get back to you.
Becky

Robb Kambic wrote:

Becky,
I also did an octal dump on the msg, it appears my decoding is correct upto actually decoding the data. since the first descriptor is a text one, i wondering if my reading compress String routine is correct? I'm includeding it, basically it just reads a char at a time, up to the width. the width of 0-1-63 is 64 bits so i'm reading 8 chars. Could you send me the code of how you read compressed text data? Any ideas here? 

Thanks,
RObb...
private void getCompressString(DescriptorTableB des) throws IOException { 
    BufrData bd = bufrdatas.get(des.getKey());
    for (int j = 0; j < dds.getNumberDatasets(); j++) {
        StringBuffer sb = new StringBuffer();
        for (int i = 0; i < ((des.getWidth() + xWidth) / 8); i++) {
          sb.append((char) bits2UInt(8));
        }
        bd.setValue( sb.toString() );
    }
  }







On Tue, 26 Aug 2008, Robb Kambic wrote:



Hi,
I was wondering if you could go into more detail with this particular Bufr msg. I've been working on this one Bufr msg for about 2 wks and have read much documentation but the answer eludes me. I already had the table, i was trying to verify that it was correct. It seems the table is correct, so are the descriptors correct? I'm have never seen a CCITT type descriptor in a compressed file, ie 0-1-63 is there anything special about CCITT in a compressed file? If the descriptors are correct, then i'm assuming that i've decoded the file correctly up to the data section. i get 818,664 for the data section length, is that correct? What data values do you get for the first field. I know i'm asking a lot here but ... 

thanks for your time,
RObb...



Here is the metadata i get from  the file, is this correct?


--------------------------------------------------------------------
                          BUFR Product
                        Header : JSML33 KWNO 250000
                Reference Time : 2008-08-25T00:00:00Z
                  Nominal Time : 2008-08-25T00:00:00Z
                     Center Id : 7
                 Sub Center Id : 0
                   Center Name : US National Weather Service (NCEP)
                      Category : 0 Surface data - land
                  Sub Category : 0 Unknown
                  Master table : NCEPtable-ABD.diff
                  isCompressed : true
-------------------------------------------------------------------- First descriptors :[0-1-63, 0-5-2, 0-6-2, 0-4-1, 0-4-2, 0-4-3, 0-4-4, 0-8-21, 0- 1-32, 0-2-200, 0-4-24, 0-8-13, 0-12-21, 0-8-13, 0-12-22, 0-8-13, 0-12-104, 0-12- 106, 0-11-1, 0-11-200, 0-11-3, 0-11-4, 0-60-1, 0-60-2, 0-60-3, 0-60-58, 0-60-59, 0-60-60, 0-60-61, 0-60-62, 0-60-63, 0-60-150, 0-60-151, 0-60-152, 0-60-20, 0-60 -21, 0-60-22, 0-60-91, 0-60-4, 0-60-7, 0-60-10, 0-60-13, 0-4-31, 0-60-90, 0-60-5 , 0-60-8, 0-60-11, 0-60-14, 0-60-16, 0-4-31, 0-60-90, 0-60-6, 0-60-9, 0-60-12, 0 -60-15, 0-60-17, 0-4-31, 0-60-90, 0-60-144, 0-60-31, 0-60-33, 0-60-35, 0-60-145, 0-4-31, 0-60-93, 0-60-100, 0-60-108, 0-60-109, 0-60-103, 0-60-104, 0-20-11, 0-6 0-42, 0-60-43, 0-60-44, 0-60-80, 0-60-81, 0-60-46, 0-60-47, 0-60-48, 0-60-94, 0- 60-49, 0-60-112, 0-60-118, 0-60-113, 0-60-114, 0-60-115, 0-60-190, 0-60-57, 0-60 -55, 0-60-116, 0-60-117, 0-60-54, 0-60-96, 0-60-23, 0-60-24, 0-60-25, 0-60-92, 0 
-60-148, 0-60-149]
-------------------------------------------------------------------- 

for instanous for year descriptor 0-4-1 i get values:

0-4-1 Year size =12285 varCount =1
 3.0784312E7 2.5368012E7 4.727478E7
I notice these files use many class 60 descriptors, is this correct? 


thanks for your help,
RObb...
=============================================================================== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research 
address@hidden           WWW: http://www.unidata.ucar.edu/
===============================================================================
=============================================================================== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research 
address@hidden           WWW: http://www.unidata.ucar.edu/
===============================================================================
=============================================================================== Robb Kambic Unidata Program Center 
Software Engineer III               Univ. Corp for Atmospheric Research
address@hidden           WWW: http://www.unidata.ucar.edu/
===============================================================================
=============================================================================== Robb Kambic Unidata Program Center 
Software Engineer III               Univ. Corp for Atmospheric Research
address@hidden           WWW: http://www.unidata.ucar.edu/
=============================================================================== 



===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================

Attachment: SurfaceForecast_L33.bufr
Description: Binary data