20010118: More on pqact slowness

Art,

The usage below for dcgrib is apparently specifying a
verbose 2 level, eg "-v 2". For dcgrib, just use "-v", while
dcgrib2 can use "-v N". (the difference is that dcgrib came
from gribtonc..an old unidata program which just uses -v and -x;
while dcgrib2 I rewrote to conform to the "dc" conventions that all the NCEP
decoders use. I do know of one instance where dcshef will closedown when
a product larger than the allocated buffer occurs.  I haven't seen
any problems with dcprof, wo any logs you have on that would be helpful.

As for resource differences between dcgrib and dcgrib2:

dcgrib2 will keep up to 5 output files open, so there is less opening
and closing of files when sending lots of different types of products
from pqact....however, you can still use the decoder with separate actions
if you feel that one instance of the decoder cannot write to disk
fast enough. 

dcgrib2 also takes advantage of the grib packing in GEMPAK, so that
gribs that are not part of a larger domain and that don't use the
bit masking (eg eta, ngm, AWIPS AVN grids ...not the thinned octets)
are able to be written directly to the gempak data file rather than expanding 
each grid point value with the scale and offset before writing (as dcgrib did).
Dcgrib2 uses the GEMPAK GB library as does NAGRIB.

I created the $GEMTBL/grid/gribkey.tbl file to allow for template naming
(similar to the $GEMTBL/config/datatype.tbl file) so that the decoder
could better separate the different models, subcenters, and grids out
since this can be difficult especially for those who don't know the
WMO patterns very well.

Performance wise, dcgrib2 should be much faster than dcgrib for grids
that don't have to get expanded. Others such as the RUC use the bit
masking so the array must be filled first, and the tiled grids have to be
expanded so they can be inserted into the larger domain- so not as much
benefit there. The file opening and closing may be slower with one invocation
rather than separate invocations - but you can still use dcgrib2 for that.
If there are more than five different output files for data at the same time, 
the
oldest written to file is closed. Typically, I see that the NGM and AVN on 
NOAAPORT
come in with the different grids (202, 201, 213, 207, 211 etc) interspersed-
which might be one drawback of a single pattern.

Steve Chiswell
Unidata User SUpport




>From: "Arthur A. Person" <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200101182326.f0INQ1e09726

>On further examination, I see a lot of "Broken pipes" showing up in my
>ldmd.log files, e.g.
>
>Jan 18 23:19:28 navier pqact[25973]: child 2433 exited with status 1
>Jan 18 23:19:30 navier pqact[25973]: child 2438 exited with status 1
>Jan 18 23:19:32 navier pqact[25973]: pbuf_flush (12) write: Broken pipe
>Jan 18 23:19:32 navier pqact[25973]: pipe_dbufput:
>/opt1/gempak/NAWIPS/bin/sol/dcgrib-v2-ddata/gempak/logs/dcgrib.log-g
>/opt1/gempak/N
>Jan 18 23:19:32 navier pqact[25973]: pipe_prodput: trying again
>Jan 18 23:19:32 navier pqact[25973]: child 2439 exited with status 1
>Jan 18 23:19:32 navier pqact[25973]: child 2440 exited with status 1
>Jan 18 23:19:34 navier pqact[25973]: pbuf_flush (13) write: Broken pipe
>Jan 18 23:19:34 navier pqact[25973]: pipe_dbufput:
>/opt1/gempak/NAWIPS/bin/sol/dcgrib-v2-ddata/gempak/logs/dcgrib.log-g
>/opt1/gempak/N
>Jan 18 23:19:34 navier pqact[25973]: pipe_prodput: trying again
>Jan 18 23:19:34 navier pqact[25973]: child 2441 exited with status 1
>Jan 18 23:19:36 navier pqact[25973]: child 2442 exited with status 1
>
>I'm guessing this is more likely related to the problem since even when
>my system lightly loaded today, my pqact is still running behind.  Can you
>give me some ideas what might be causing the broken pipes?  It seems to
>occur for dcgrib, dcshef, and dcprof only.
>
>                                    Thanks.
>
>                                      Art.
>
>On Thu, 18 Jan 2001, Arthur A. Person wrote:
>
>> Hi...
>>
>> I started a test dcgrib2 up the other day in preparation of switching to
>> gempak 5.6.a and in the process discovered that my pqact is apparently
>> having trouble keeping up with the amount of data flowing through the
>> various options in pqact.conf.  I'm wondering if breaking out the pqact
>> tasks among several instances with separate pqact.conf files will help...
>> could you comment on how pqact does its job?  Does it get a product and
>> hand it off to each action asynchronously, or does it wait for each
>> hand-off to finish?  Any other comments on pqact's ability to keep up with
>> a big load, short of finding more processor resources, would be helpful.
>>
>> Also, does dcgrib2 take more resources than its predecesor?  It took a lot
>> more processor while running but I suppose it's possible that it may have
>> just been handling more data since I ran it in universal mode with all
>> GRIB data being piped into it.
>>
>> I'm running a dual sparc ultra-2 200mhz server with ldm V5.1.2.
>>
>>                                   Thanks.
>>
>>                                     Art.
>>
>
>Arthur A. Person
>Research Assistant, System Administrator
>Penn State Department of Meteorology
>email:  address@hidden, phone:  814-863-1563
>