20030904: 20030827: LDM-6 installation at UNCA (cont.)

Matt,

First off, here is some general information about what is
carried on NOAAPORT (eg the source of the data you are asking):

1) ETA is sent for 0-60 hours at 0Z and 12Z, and through 48 hours
   at 6Z and 18Z. You will not see any grids out to 84 hours on NOAAPORT
   for the ETA. 

2) AVN Global thinned grids are sent for 0-72 hours, while certain other
   regional projections are available through 120 hours.

In gempak, you can see the AVN on the global thinned grids
in $MODEL/YYYYMMDDHH_thin.gem (which is aliased to "AVN" in
$GEMTBL/config/datatype.tbl). For the CONUS region, you can view
AVN on the 80KM grid #211 through 120 hours in $MODEL/YYYYMMDDHH_avn211.gem
which is aliased to AVN211 in the configuration table. You can add
AVN211 to the $GEMTBL/nmap/mod_res.tbl entries as appropriate for various 
products.

Steve Chiswell
       
            
>From: Matt Rosier <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200309050019.h850JuLd017895

>Tom,
>
>Do you know how we can correct the problem of not receiving (or not decoding 
>perhaps) Eta data past 60 hrs or so and AVN data past 84 hrs?
>
>
>Thanks,
>
>Matt
>
>
>> -----Original Message-----
>> From: Tom Yoksas [mailto:address@hidden]
>> Sent: Wednesday, August 27, 2003 2:41 PM
>> To: ahuang
>> Cc: address@hidden; address@hidden;
>> address@hidden; Matthew J. Rosier
>> Subject: 20030827: LDM-6 installation at UNCA (cont.)
>>
>>
>> >From:  ahuang <address@hidden>
>> >Organization:  UNCA
>> >Keywords:  200307301533.h6UFXPLd024978 LDM McIDAS upgrade
>>
>> Hi Alex,
>>
>> I am CCing Matt Rosier on this note in case he has been delegated
>> the responsibility of looking after the LDM and McIDAS installations
>> on UNCA machines.  Here goes:
>>
>> It is time to update you on changes I made on storm2...
>>
>> >Thank you for e-mail and your willingness to help us out, so you
>> are not that
>> >busy after all :-))  Are you sure it will only take half of an hour?
>>
>> As you can see, it took me almost an entire month to get to all of the
>> updates that were needed on storm2, so I guess I was more busy than
>> I originally implied.  I can tell you, however, that the LDM upgrade
>> did take less than a half an hour.  In fact, I seem to remember that
>> upgrading from LDM-6.0.13 (which Leigh had installed) to LDM-6.0.14
>> and adding reporting of realtime statistics took on the order of
>> 10 minutes.
>>
>> >Your e-mail also gave the reason that why storm2 was 100% full
>> and crashed,
>> >since the data volume had exploded in July.  We had to clean up
>> manually to
>> >get storm2 back two days ago, so far it is running fine.
>>
>> In going through the McIDAS upgrade to v2003, I found a what was probably
>> the real reason that storm2 filled up.  It seems that the files being
>> created by McIDAS-XCD (the decoder component of the Unidata McIDAS
>> distribution) were not getting scoured.  Moreover, the output point
>> source data files (MDXX files) were all full and had been so since
>> sometime in June.  This was causing the McIDAS-XCD decoding processes
>> to continually fail and use up CPU in doing essentially nothing.
>>
>> The culprit in the failure of scouring XCD-produced data was the
>> non-existence of ~mcidas/util/mcscour.sh.  The ~mcidas/util
>> directory did not exist, so mcscour.sh in the directory couldn't
>> exist.  The scouring was being handled by a cron entry for the
>> user 'mcidas' that was continually failing.  Here is one of the
>> emails being sent by the OS to 'mcidas' that tells of the problem:
>>
>>   From address@hidden  Sat Jun  7 01:00:04 2003
>>   Date: Sat, 7 Jun 2003 01:00:01 -0400
>>   From: address@hidden (Cron Daemon)
>>   To: address@hidden
>>   Subject: Cron <mcidas@storm2> /home/mcidas/util/mcscour.sh
>>   X-Cron-Env: <SHELL=/bin/sh>
>>   X-Cron-Env: <HOME=/home/mcidas>
>>   X-Cron-Env: <PATH=/usr/bin:/bin>
>>   X-Cron-Env: <LOGNAME=mcidas>
>>
>>   /bin/sh: line 1: /home/mcidas/util/mcscour.sh: No such file or directory
>>
>> This tells us that the scouring failure started at least on June 7.
>>
>> I corrected the scouring failures by moving the scouring activities
>> to the 'ldm' account (it could have been done in the 'mcidas' account,
>> but I felt that consolidating and simplifying the McIDAS installation
>> was a good thing to do) by copying mcscour.sh from the ~mcidas/workdata
>> directory to ~ldm/decoders.  mcscour.sh, a Bourne shell script, then
>> had to be edited to set various Unix environment variables used by
>> McIDAS, and an entry was made in 'ldm's crontab file to run the scouring:
>>
>> #
>> # Scour McIDAS data files
>> #
>> 00 01 * * * decoders/mcscour.sh
>>
>> Since all of the McIDAS point data files (/data/mcidas/data/MDXX*)
>> were full or corrupted, I deleted them and resetup McIDAS-XCD decoding:
>>
>> <as 'mcidas' done while the LDM was not running>
>>
>> cd /data/mcidas/data
>> rm MDXX*
>> rm *.RA*
>> rm *.IDX
>> rm *.IDT
>>
>> cd ~mcidas/workdata
>> tl.k                   <- to verify that XCDDATA was set; it was
>> batch.k XCD.BAT
>> batch.k XCDDEC.BAT
>>
>> <review/update the McIDAS file REDIRECT entries in
>> ~mcidas/workdata/LWPATH.NAM>
>>
>> <as 'ldm'>
>> ldmadmin start                <- restart the LDM
>>
>> >I know you can make the storm2 more reliable, so please go ahead
>> and upgrade
>> >LDM in storm2.
>>
>> >I also wants students to use McIDAS and
>> >GEMPAK using a generic student account that has been set up.
>>
>> If I knew the name of the generic student account, I would logon as that
>> user and make sure that the account is setup correctly.  Is the
>> name of the generic account 'unca-mcidas'?
>>
>> >I do hope that a few critical things can be maintained:
>> >
>> >1.  storm2 prints out DIFAX-like maps by one of three cron
>> jobs, one for
>> >vacation, one for weekends, and the third for weekdays.   These maps are
>> >printed out by an IP printer (HP LJ5000).   This is very
>> essential to our
>> >instruction, so please don?t change it.
>>
>> I did not touch this setup, so it should not have been affected.
>>
>> >2.  storm2 is the LDM server, and it also has McIDAS and GEMPAK (by ntl
>> >command).  Other LINUX PC?s have McIDAS and GEMPAK.   I now seem to have
>> >problems to use ntl and mcidas commands on other PC?s, and I am
>> trying to
>> >figure it out, because it was working before the departure of
>> Leigh Orf in
>> >July.  I may ask for your help later, it should be easy.
>>
>> I can't speak on the GEMPAK problems, but I can speculate on the McIDAS
>> problems.  One problem I found on storm2 was the setup of the remote
>> ADDE server.  The v2002 remote server required the creation of three
>> files in the /etc/xinetd.d directory: mcserv, mcidasz, and mccompress.
>> These files existed and contained entries that worked correctly for
>> versions of RedHat Linux previous to 8 (i.e., things worked on 6.x and
>> 7.x).  There was one entry, however, in all of these files that is not
>> liked by RedHat 8.0 and 9.0.  I corrected this problem, and the remote
>> server on storm2 started working again.  The McIDAS v2003 remote ADDE
>> server installation script takes care of this problem.  More on
>> v2003 below.
>
>> >3.  storm2 is setup to stop and start LDM while booting, in
>> case of power
>> >failure.   Leigh spent some hours on getting this to work, and I
>> am not sure
>> >about this, maybe new LDM can do a better job in closing LDM completely.
>>
>> I just took a look at the startup script that Leigh put in place for the
>> LDM.  If pressed, I would modify this script to check the integrity of
>> the LDM queue before trying to start the LDM.  Right now, the script
>> simply starts the LDM with no checks.   It is our observation that
>> the LDM queue can get damaged in cases where there is a power failure
>> or the system is rebooted without first stopping the LDM with:
>>
>> <as 'ldm'>
>> ldmadmin stop
>>
>> The queue is not guaranteed to get corrupted in these cases, but it
>> is the most likely time for it to happen.
>>
>> The other thing I did to the LDM configuration was to setup all ldm-mcidas
>> decoders to log to the ~ldm/logs/ldm-mcidas.log file, and change the
>> crontab entry that rotates this file to keep 4 online instead of 2.
>> Moving of where the decoders log helps to clean up the LDM log file,
>> ~ldm/logs/ldmd.log.  We really like to see decoder logging put in
>> a file other than ldmd.log so that it is easier to use ldmd.log
>> to diagnose LDM problems.  The change was a minor one, so there is
>> no need to worry about it :-).  For reference, I made the logging
>> changes in all of the pqact.conf files that are used on storm2:
>>
>> pqact.conf
>> pqact.conf.vacation
>> pqact.conf.weekend
>> pqact.conf.weekday
>>
>> I also copied two files to ~ldm/etc, SATANNOT and SATBAND, and I adjusted
>> all pqact.conf* pnga2area entries to use these files.
>>
>> >4.  I would only like to keep 7 days of data, I hope 120 GB
>> space in storm2
>> >  is enough, if not, 5 days of data are acceptable too.  I don't want any
>> >  maps and data archived.
>>
>> I setup McIDAS-XCD to keep 7 days of decoded point source data.  I see
>> that it is not currently decoding model output data.  The entry in
>> ~ldm/etc/pqact.conf that would allow the decoding is commented out, and
>> the entry in the McIDAS-XCD configuration file
>> ~mcidas/workdata/DECINFO.DAT that would tell the XCD supervisor routine
>> to run grid decoding is not enabled.
>>
>> Do you want McIDAS to decode model output data?
>>
>> If yes, I would recommend only keeping one data of that data online.
>> The format of the McIDAS GRID files is not compressed like it is in
>> GEMPAK, so the GRID* files take up a LOT of room (like up to 7 GB per
>> day).
>>
>> >5.  I can be physically at storm2 for your instructions or something, if
>> >   you need me.
>>
>> All of the work that was needed could be done remotely.
>>
>> >You now have my blessing to perform the upgrade on storm2.
>> Thank you again,
>> >and hope your summer has been wonderful.   My summer is OK, and
>> UNCA starts
>> >August 20.
>>
>> I am sorry I couldn't complete the upgrade of the McIDAS side of things
>> until today, a full 1 week after the fall UNCA session started.  I
>> can report, however, that things are now working nicely.
>>
>> While on storm2, I decided to do more than upgrade the LDM and get
>> McIDAS working again:  I decided to upgrade McIDAS to the latest
>> version, v2003.  I also decided to upgrade the ldm-mcidas decoders
>> (the decoders for the image products in the Unidata-Wisconsin datastream)
>> to the latest release, ldm-mcidas-2003.
>>
>> While trying to upgrade of McIDAS to v2003, I found that the development
>> environment on storm2 was not complete.  Missing were:
>>
>> yacc, curses, X development, and things these depended on.  I installed
>> all of the needed stuff from RPMS to get the build to work.  Here is the
>> list of rpm files that I installed:
>>
>> XFree86-devel-4.3.0-2.i386.rpm
>> byacc-1.9-25.i386.rpm
>> fontconfig-devel-2.1-9.i386.rpm
>> freetype-devel-2.1.3-6.i386.rpm
>> ncurses-devel-5.3-4.i386.rpm
>> pkgconfig-0.14.0-3.i386.rpm
>>
>> After installing these, I was able to successfully build McIDAS-X/-XCD
>> v2003:
>>
>> <as 'mcidas'>
>> cd mcidas2003/src
>> make all VENDOR=-g77          <- use gcc/g77 compiler combination
>> make install.all VENDOR=-g77
>>
>> I did the installation part of the build/install with the LDM turned
>> off.
>>
>> I then followed the upgrade steps listed in:
>>
>> ABCs of Upgrading from a Previous Distribution
>> http://my.unidata.ucar.edu/content/software/mcidas/2003/mcx/upgrad
>e_notes.html#upgrade_top
>
>Based on the remote ADDE server problems I had previously, I decided
>to uninstall and the reinstall the ADDE remote server stuff.  This had
>to be done as 'root':
>
><as 'root'>
>cd ~mcidas
>sh ./mcinet2002.sh uninstall mcadde
>sh ./mcinet2003.sh install mcadde
>
>I verified that the v2003 ADDE remote stuff is working by pointing my
>McIDAS session at your server for the RTPTSRC, RTIMAGES, and CIMSS
>datasets and loading imagery, loops, doing plots and contours of a
>variety of different data, and also plotting soundings.  Everything
>looks solid at the moment.
>
>>Let me know if you have any questions.
>
>One observation of something I would change if given carte blanche.
>Right now, the scripts run out of 'ldm's cron that create the maps
>that you referenced in 1. above are located in the ~ldm/bin directory.
>Since ~ldm/bin is a link to ~ldm/runtime/bin, and since this runtime
>link gets changed for each new LDM installation, I would _strongly_
>recommend that the shell scripts:
>
>start_maps.csh
>vacation_maps.csh
>weekday_maps.csh
>weekend_maps.csh
>
>to a permanent directory like ~ldm/util.  If this move is done, the
>crontab entries that run these scripts will need to be altered
>slightly.  Please let me know if you would like me to make the
>recommended changes.
>
>OK, I think that is enough for now.  Please let me know if you
>would like me to finish off items I listed above.
>
>>Alex
>>828-232-5157 (O) 828-258-0292 (H)
>
>Cheers,
>
>Tom
>--
>+-----------------------------------------------------------------------------
>+
>* Tom Yoksas                                             UCAR Unidata Program 
>*
>* (303) 497-8642 (last resort)                                  P.O. Box 3000 
>*
>* address@hidden                                   Boulder, CO 80307 
>*
>* Unidata WWW Service                             
>http://www.unidata.ucar.edu/*
>+-----------------------------------------------------------------------------
>+
>
>