20051115: Accessing bufr data via noaaport (cont.)

>From: "Alliss, Randall J." <address@hidden>
>Organization: NGC
>Keywords: 200511151537.jAFFbu7s016602 LDM McIDAS-XCD bufr

Hi Randy,

>I have enclosed two files. They are both the pqact.conf files. One on
>noaapnew and the other on noaapxcd.

OK.  I see that the one on noaapxcd does nothing other than send the
products to McIDAS-XCD ingesters:

PPS|DDS|IDS     ^.*     PIPE    /home/oper/mcidas/bin/ingetext.k        NTXT
HDS|NNEXRAD     ^.*     PIPE    /home/oper/mcidas/bin/ingebin.k NBIN

This means that all of your configuration setup for saving bufr
files will need to be done in McIDAS-XCDland.

>I also ran the notifyme command which sent output to screen. I tried
>piping to a file with > look  but that did not work.

The output from 'notifyme -vxl-' goest to STDERR, not STDOUT.

If you want to save the notifyme output to a file, you would do the
following:

notifyme -vxl bufr.log -f HDS -o 3600 -p '^[IJ]'

The 'l' (ell) in '-vxl' is where to write the log output.  The default
is for the output to be sent to the syslogd daemon.  This example
writes the output to a file named 'bufr.log' in the current directory.

>It appears there are bufr files there.

OK.  This tells me that you NOAAPORT ingestion is not broken.  By the way,
you should see LOTS of bufr files in the HDS stream.

>Rebecca is not here but she set the processing up via SSEC instructions.

The setup you show is similar to mine.  I wrapped the invocation of
ingetext.k and ingebin.k in a Bourne shell script.  The major reason
for doing this was for setting Unix environment variables used by
McIDAS (e.g., MCPATH, etc.).  I suppose that the SSEC instructions had
you set those environment variables in the shell of your LDM user.  I
don't like this approach since I try to keep McIDAS configuration out
of the LDM environment.  Putting everything into a shell script keeps
things more tidy (in my opinion).

>We were not specifically trying to retain the bufr or grib data when she
>set this up last january.

And the contents of pqact.conf.noaapxcd show this.

>I do not know the answer on how we configured XCD data monitor to
>process the bufr products. I do know that the statdisp shows many of the
>decoders running which means we are getting all the surface/upper air
>etc data.

OK. It is the DMBIN data monitor that needs to be configured for
processing the bufr data.  A quick look at McIDAS-XCD on my home
machine shows the following:

<as 'mcidas'; I have the Unidata community run things as 'mcidas', not 'oper'>

decinfo.k LIST

 ...

Processing Data Monitor: DMBIN        is inactive
=========================================================================
Decoder       Status      Configuration File
----------------------------------------------------------------------
GRBFILER      Active      GRBFILER.CFG


Two things to note here:

- my DMBIN is set to be inactive

- DMBIN's actions are controlled by entries in GRBFILER.CFG.

You should logon to your XCD box as 'oper' and run:

DECINFO LIST

to see if DMBIN processing has been turned on.  If it hasn't, then run:

DECINFO SET DMBIN ACTIVE

The next step is to check out the configuration of your GRBFILER.CFG.
Find GRBFILER.CFG and check to see how DECODETYPE has been set.  It
is likely that it is set to the default:

DECODETYPE=1              # determines which types of data get decoded and 
filed                          # if set to 1 then both GRIB and BUFR
                          # if set to 2 then only GRIB
                          # if set to 3 then only BUFR

If you want to save only BUFR files, you would _presumably_ (I have not
tried this) set DECODETYPE to '3'.

NOTE: if you change a value in GRBFILER.CFG, you will need to 'bounce'
DMBIN.  You do this by inactivating it and then reactivating it:

DECINFO SET DMBIN INACTIVE
<wait until you do _not_ see a dmbin.k process running (use ps -eaf | grep 
dmbin)>
DECINFO SET DMBIN ACTIVE

You need to do this since XCD data monitors read their configuration files
at startup only.

By the way, I have a hunch that DMBIN processing has not been activated
on your machine.

>Can you tell us what needs to be edited in those .conf files in order to
>retain the bufr data.

Comment: I find using McIDAS-XCD to file grib/bufr messages to be
clunky.  I would suggest using the LDM directly through appropriate
pqact.conf entries.  This can be done in a variety of ways, and in
all ways you have to decide how to name the output files.

The first thing to do is get an idea of what the bufr product headers
look like using notifyme:

cd ~ldm/logs
notifyme -vxl bufrfiles.log -o 3600 -p "^[IJ]"

Let this run for awhile (how long depends on how many header lines you
will want to look at; I suggest at least 15 minutes), and then kill
the process.  Now, look through the log file output to get a feel
for the various bufr products and their headers.  Here is an
example:

 ...

Nov 16 15:21:52 notifyme[82237] INFO: cc881cba980be40920d234a25ff3d867      225 
20051116143936.766     HDS 61250138  IUPT43 KBOU 161434
Nov 16 15:21:52 notifyme[82237] INFO: 82dc4f23effd594e960e9e7a95adf45d      225 
20051116143938.768     HDS 61250159  IUPT43 KBOU 161434
Nov 16 15:21:52 notifyme[82237] INFO: 5a37620f6bc0496416d4f18128004e41      225 
20051116143938.769     HDS 61250160  IUPT43 KBOU 161434
Nov 16 15:21:53 notifyme[82237] INFO: 7bb4e3edea623ea2aa49340bd12881a0      369 
20051116143950.018     HDS 61250289  IUAD01 EGRR 161438 RRB

 ...

The headers are:

IUPT43 KBOU 161434
IUPT43 KBOU 161434
IUPT43 KBOU 161434
IUAD01 EGRR 161438 RRB

 ...

JSAT98 KKCI 161435
JSMT71 KWNH 160000
JSMT73 KWNH 160000
JSMT74 KWNH 160000

The headers contain the product type (e.g., IUPT43), originating center
(e.g., KBOU), time (e.g., 161434), etc.  The meaning of the product type
is, I believe, defined in the WMO manual on codes.  Since I am no expert
on these codes, I will not pretend to be able to guide you here.

Anyway, back to creating an LDM pqact.conf action to file the products.
I would do this on the XCD machine, not the NOAAPORT one.  To capture
all of the products into files whose names contain the product type,
originating center, and time, add the following entry in pqact.conf
(on noaapxcd):

HDS     ^([IJ].....) (....) (......)
        FILE    -close  -overwrite
        data/bufr/\1_\2_\3.bufr

This will write bufr files into the ~ldm/data/bufr subdiretory,
and the names will look something like:

IUPT43_KBOU_161434.bufr
IUPT43_KBOU_161434.bufr

etc.

NOTES:

- some grib messages have additional fields that may be important
  to you.  Adjust your pqact.conf action accordingly.

- this is only one idea of how to do this; what will work best for
  you will become more clear as you get familiar with the LDM
  
- in pqact.conf entries, some whitespace MUST BE TABS!  This
  is _very_ important (hence the capital letters :-).  Read
  the above action as follows:

newline
HDS<tab>^([IJ].....)<space>(...)<space>(......)
<tab>FILE<tab>-close<tab>-overwrite
<tab>data/bufr/\1_\2_\3.bufr

- absolutely NO line in a pqact.conf file can begin with a space.

Once you have settled upon which bufr files you want to FILE, and the
naming scheme for those files, you will need to decide how many days
of data you want to keep on disk (i.e., worry about data scouring).
You can scour the files easily using the LDM 'scour' utility:

- edit ~ldm/etc/scour.con and add an entry to the end that says
  what directory(ies) you want to scour and how many _days_ of
  data you want to keep in that(those) directory(ies)

- add an entry to your LDM user's crontab that runs scour once per day

Here is an example:

- add the following line to ~ldm/etc/scour.conf to scour the ~ldm/data/bufr
  directory and keep 2 days of files in it:

~ldm/data/bufr                  2

  (by convention, the whitespace between the directory name and the number
  of days to keep are tabs)

- add an entry to your LDM user's crontab file:

0 1 * * * bin/ldmadmin scour > /dev/null 2>&1

That's pretty much it.  It is up to you to decide what you want to keep;
how you want the output named; and how long you want to keep the data
saved.

>Thanks,

No worries.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.