20010211: XCD GRID decoding, SOUNDINGS HODOGRAPH (cont.)

>From: "James D. Marco" <address@hidden>
>Organization: Cornell
>Keywords: 200102071732.f17HW4L24435 McIDAS-X XCD ADDE FOUSDISP

Jdm,

Sorry I couldn't get to this earlier...

>       Thanks. You mentioned the pqact and I checked.  One of the hacks
>I made last week was to add GRID and PPS types to the pqact. I got this
>out of the xcd_run script and was unsure if this needed to be specified
>in the new version.

The idea in xcd_run was to allow users to specify textual input to
ingetext.k in a variety of ways (two of which are DDS and PPS) and
input to ingebin.k in an equal variety of ways (HRS, HDS, GRID).

>It is NOT in the old.

And you should only specify one way.  What is going on is that all textual
data gets sent to ingetext.k, and all binary data gets sent to ingebin.k.
xcd_run is simply a script wrapper around startxcd.k, ingebin.k, and
ingetext.k.  It is used to set needed McIDAS environment variables and
then run the specific McIDAS command.

>Anyway, here is the first couple entries (head -20 pqact.conf):
>
>WMO     ^([^H][A-Z])([A-Z][A-Z])([0-9][0-9]) (....) ([0-3][0-9])([0-2][0-9])
>        FILE    /var/data/mcidas/WMO/\2/\4/\6/\1\3.wmo
># Entries for XCD decoders
>DDPLUS|IDS      ^.*     PIPE    xcd_run DDS
>#GRID           ^.*     PIPE    xcd_run GRID
>HRS             ^.*     PIPE    xcd_run HRS
>#PPS            ^.*     PIPE    xcd_run PPS                                 
><deleted...>

I would go ahead and delete the #GIRD and #PPS lines as these might lead
to confusion down the road.

>After comenting the GRID and PPS file types out, per your suggestion of
>checking the pqact.conf, the ingetext.k processes dropped from 4 to the
>expected 2.

Good.

>I am assuming that some data streams were being incorrectly 
>decoded twice.

ALL of the textual data was being sent into the text ingester, ingetext.k.
Both of these guys were writing into the daily spool file (*.XCD) and
both were attempting to manipuate the pointers into the file.  This
is bad...

>It will take a while to get some of the model runs in, but 
>there are now a couple GRID54* files in the ~data/xcd directory.  Dare I 
>assume this is good?

Yes.  GRID files in the range 5401 to 5500 are AVN grids.

>I am at home, so I will check tommorow when I get
>in to work.

OK.

>(McIDAS gui still doesn't run under eXceed or MacX.)

At some point I would love to figure that one out.

>The Fkey
>menu doesn't give me the level of controll I need, and, I'm not that
>knowledgable with mcidas commands to issue them directly without fudging
>around with them.

As far as GRIDs go, the Fkey menu is severely lacking.  Once your decoding
is working correctly, we should touch base about you running a cron-initiated
uwgrid.sh invocation that will create from the XCD decoded GRIDs a set
of GRID files that match what was once in the Unidata-Wisconsin datastream.
This will allow the Fkey menu to work as it was designed.

>I can certainly put personnal accounts on the rest of the machines for you,
>if you need them. The mcidas account is the same on all 7 servers.  

It would be good for me to be able to login to the machine running the
LDM that runs XCD.  That is IF the login is still needed.

>Ahhh...the structure is a bit less than simple. And, I will making it more
>complex, for a couple reasons...not just because I like complexity.
>The primary machine is overloaded with LDM, Decoding a large number of 
>files, McIDAS display hosting for NT/eXceed, GemPAK display hosting and 
>decoding, etc. When I run ldmadmin scour, data is often interrupted or 
>garbled, and, sometimes one of the McIDAS displays (30 frame loops, 
>autoupdate) will hang.

Do you think this is because the CPU is hammered?

>(There are three autoupdate displays for 
>different data.) Any heavy job will do this, actually.  

This comment does imply that it is a loading problem.

>       I am attempting to distribute some of the LDM and decoding to
>a second machine. Hopefully, this will distribute the system load.

OK.

>Generally all 7 mcidas servers are set up with the same account, so you can
>telnet around.  However, only two machines are in service. Lab machines are
>pending upgrade to be placed back in active service, though they are used
>for other things.  

OK.

>Over all:
>       o  vorticity.cit.cornell.edu - you have access to mcidas
>               RH 6.2
>               Main LDM ingestion for Map Room, MAD Lab 
>               McIDAS Server for 3 NT/Exceed looping/auoto-update displays
>               Gempak Server for NSAT and GARP in Map Room 
>       
>       o  visibility.cit.cornell.edu - you have access to mcidas
>               Secondary LDM ingestion
>               NWS Fax Downloads (scheduled)
>               SETUP: LDM, Mcidas, Gempak  
>               This is the one I am working on.
>
>Note: All machines have wrappers, so you can only access them on telnet
>from vorticity.
>Locally, there is other access for cornell accounts. But I think you can
>handle telnet,
>unlike some of the other users.

Right.

>(mcidas)REDIRECT LIST
>...
>GRID000* /var/data/mcidas                                              
>GRID010* /var/data/mcidas                                              
>GRID5* /var/data/mcidas/                                               
>GRID8* /home/mcidas/data/tutorial                                      
>GRID* /var/data/mcidas                                                 
>HRS.SPL /var/data/mcidas/                                              
>IDXALIAS.DAT /var/data/mcidas/                                         
>LWPATH.NAM .
>...

These look correct.

>Note: I added GRID* and moved all GRID files to /var/data/mcidas
>       /var/data/xcd is a link to /var/data/mcidas

The GRID* was probably not needed and probably not wanted.  All of the
XCD-produced GRID files are in the GRID5* range.  What you may want to
do is remove the GRID* REDIRECTion and fill out the ones for GRID files
that uwgrid.sh can produce:

REDIRECT ADD GRID000* "/var/data/mcidas
REDIRECT ADD GRID0010 "/var/data/mcidas
REDIRECT ADD GRID010* "/var/data/mcidas
REDIRECT ADD GRID0110 "/var/data/mcidas

>On the model stuff: Running from Fkey gives this:
>Command (on one line):
>       FOUSDISP T OLAY +00 INT=00:30 DAY=2001042 MODE=X GRIDF=132 
>       GRA=12 SF=YES PRO=CONF 
>       OUT=PLO GU=GRAPHIC BLANK=NO               
>Error:
>       pipe read: Connection reset by peer

Yikes!!!!  The pipe read error is the same one I am running into when
trying to serve sounding data from Linux.

>       FOUSDISP: Unable to execute PTDISP       command
>       FOUSDISP - Done
>       FOUSDISP failed, RC=1

I just tried serving FOUS14 data from the same server at UNCA that I
am having problems with (sounding serving), and it serves FOUS14 data
just fine.  FOUSDISP plots FOUS14 data which, even though it is data
from a model, is stored in a McIDAS MD file and is part of the RTPTSRC
dataset.  I wonder what the pipe read error you are seeing is telling
us about ADDE service on Linux!?

Tom

>From address@hidden Tue Feb 13 14:02:31 2001
>Subject: Re: 20010211: XCD GRID decoding, SOUNDINGS HODOGRAPH (cont.) 

Tom, 

At 09:26 PM 2/12/01 -0700, you wrote:
>>From: "James D. Marco" <address@hidden>
>>Organization: Cornell
>>Keywords: 200102071732.f17HW4L24435 McIDAS-X XCD ADDE FOUSDISP
<deleted...>
>As far as GRIDs go, the Fkey menu is severely lacking.  Once your decoding
>is working correctly, we should touch base about you running a cron-initiated
>uwgrid.sh invocation that will create from the XCD decoded GRIDs a set
>of GRID files that match what was once in the Unidata-Wisconsin datastream.
>This will allow the Fkey menu to work as it was designed.
Sounds good. Some of the older members of the department will be pleased.

<...deleted...>
>The GRID* was probably not needed and probably not wanted.  All of the
>XCD-produced GRID files are in the GRID5* range.  What you may want to
>do is remove the GRID* REDIRECTion and fill out the ones for GRID files
>that uwgrid.sh can produce:
>
>REDIRECT ADD GRID000* "/var/data/mcidas
>REDIRECT ADD GRID0010 "/var/data/mcidas
>REDIRECT ADD GRID010* "/var/data/mcidas
>REDIRECT ADD GRID0110 "/var/data/mcidas
Done.


>>On the model stuff: Running from Fkey gives this:
>>Command (on one line):
>>      FOUSDISP T OLAY +00 INT=00:30 DAY=2001042 MODE=X GRIDF=132 
>>      GRA=12 SF=YES PRO=CONF 
>>       OUT=PLO GU=GRAPHIC BLANK=NO               
>>Error:
>>      pipe read: Connection reset by peer
>
>Yikes!!!!  The pipe read error is the same one I am running into when
>trying to serve sounding data from Linux.

Uhhh...I don't think this is your error.  This is a semi-normal
nonsensical message from some programs using the socket interface.

After the sockets do an initial connect, a child is spawned and 
the port number is adjusted, internally to the kernel networking,
to free the parent socket for the next incoming connection request.
Otherwise only one program could ever connect to a port number.

Some client programs interpret this as an error and report this as a 
serious sounding message.  In fact, the connection was maintained 
throughout the process by the socket interface.  The connect() 
process usually spawns a child for accept(). For some brief instant
of time, there are actually two active ports. The remote clients often 
interpret this hand off as an event and kick out the above message.  It 
isn't an error...normal Un-ix multitasking stuff at the socket interface
is just being reported.

But, it does indicate that the network interface is being used.  Of course,
this is normal even on the same Unix machine, (sockets are easy to use
for IPC. Much easier than the shared memory, message queues and semaphores
of Sys V unix.) 

I was under the impression that this was how the ADDE worked?  

Thus, the LOCAL-DATA in the LOCDATA.BAT is redundant for sessions
originating on the same machine.  You should be able to substitute the
hostname for
LOCAL-DATA.  When you do, you will see this same message on the local
machine.
   
>
>>      FOUSDISP: Unable to execute PTDISP       command
>>      FOUSDISP - Done
>>      FOUSDISP failed, RC=1
>
I think this is the real error. Looks like a script error or a bad call,
or just plain bad communications between two programs:
        FOUSDISP and PTDISP
I am assuming that FOUSDISP is a wrapper around PTDISP.

>I just tried serving FOUS14 data from the same server at UNCA that I
>am having problems with (sounding serving), and it serves FOUS14 data
>just fine.  FOUSDISP plots FOUS14 data which, even though it is data
>from a model, is stored in a McIDAS MD file and is part of the RTPTSRC
>dataset.  I wonder what the pipe read error you are seeing is telling
>us about ADDE service on Linux!?
>
>Tom
Absolutly.
Hmmm...OK. This would be consist ant with what I am seeing. There are several
flags to the socket() call when the sockets are instanciated.  These are
macros in the /usr/include files....uhhh, socket.h, I think.
(Well, close.../usr/include/sys/socket.h) Red Hat does some weird stuff 
with their Protected Interface stuff...These are pi_socket.h and others.

ADDE has been pretty workable in the past, so, I suspect that the
RH people of taking liberties with the include files.
There are really only two major kinds of socket interfaces:
        AF_UNIX (or AF_LOCAL)
        AF_INET
        A local interface and a remote interface basically. (Pretty 
much corresponding to the two types of connections in LOCDATA.BAT.)

Of course, you know this stuff, soo, I'm just thinking out loud. But I
suspect the error is in there.

                        Gotta run, some personal support issues...

If I don't send it now, you won't get it till the morrow.
                                                        jdm