19990623: McIDAS Comments and Problems

>From: Bryan Rockwood <address@hidden>
>Organization: Creighton
>Keywords: 199906240351.VAA10949 McIDAS NLDN DAY ADDE NT

Bryan,

>       Greetings and long time no pester!

Here we go :-)

>It has been a rather quiet
>summer so far, here at Creighton, and we decided to try some different
>stuff.  The LDM (which had been running Linux) recently died so we went
>ahead and started using Solaris x86 here in the lab.

From my point of view this is a good move.

>From what I can tell
>of the LDM, it seems to agree with the Solaris a lot more then it did with
>the Linux OS.

Yes.  We have heard a few commets from sites using Linux regarding the
the amount of time it takes the LDM to stop after an 'ldmadmin stop' command
is executed.  It appears that Linux is very agressive about memory buffering,
but it does not handle large product queues well.

>Currently, we have XCD ingest the entire DDPlus stream,
>McIDAS, Post Process stuff running, NIDS, NLDN, and the FSL2 datastream.

Hopefully, you or someone else at Creighton has received the warnings
that the non-imagery components of the Unidata-Wisconsin datastream
(aka MCIDAS) are going away on July 1.  Good thing you are running XCD!

>The main thing that I did notice is that:
>
>1)  The hard drive does not thrash around nearly as bad as it did with the
>Linux.  I can only chalk this up to better memory management.

What kind of disk(s) do you have in your system (e.g. IDE, EIDE, UIDE,
SCSI)?

>2)  The NFS stuff seems to be faster then the Linux implementation as
>well.

I believe that Solaris 2.7 (both SPARC and x86) use NFS3.  Linux uses
NFS2.

>To keep the OS's constant, we turned our McIDAS workstation machines to
>Solaris as well.

The only drawback to Solaris x86 is the color management in CDE.  In
order to run McIDAS and Netscape together, Netscape should be started
second so that McIDAS can get enough colors.  If you are using CDE, you
probably have already run into instances where you need to go into
the Color manager (accessed from the Style Manager GUI) to change how
many colors are used by CDE and how many are used for applications.

>They appear to be a little slower then their Linux
>counterparts were, but still more then usable.

I have found that Solaris x86 on a 200 Mhz Pentium II is noticably slower
than Linux (but not terrible) and requires more memory.  The memory overhead
is again in CDE.

>The LDM has been up for
>the last two weeks with no downtime, which is more then I can say for the
>old Linux server.

Hmm...  Sites relying on Linux are not reporting that their systems crash
or that the LDM stops unexpectedly.  If this was happening on your system,
it may have been due to some sort of a configuration error.

>I don't know if I had some things configured wrong in
>Linux, but it is a moot issue now as the software is running very stable.

Sounds like the move to Solaris was a good one.

>I must also compliment you on the online documentation.  Before, I was
>always under some kind of pressure constraints to get the system setup in
>a hurry, which probably led to my problems from before.  This time, being
>summer and all, I followed everything step by step, basically taking about
>three days to sit down, read all the docs, and make sure I got it all
>right.  Things are working exactly like they should

Super.

>(with two exceptions
>as you will read below) which can only be attributed to the online
>information at the Unidata site. 

You just burst my bubble :-(

>Sorry for the rant, but I figured that you guys would like to hear some
>positive feedback every once in a while :-).

Actually YES.  I really appreciate it when people write in to correct
something in the online documentation or request a clarification.  This
helps everyone down the road.

>Now, for the pester part.

Ready...

>I have two issues that are giving me a
>headache.  The first has to do with the NLDN data and the client machines.
>When I go to do an analysis of the lightning data by selecting
>DISPLAY->OBSERVATIONS->PLOT/CONTOUR and choosing Lightning, plot, all
>flashes, and current.  I get an error saying NO DATA AVAILABLE.  If I go
>to the McIDAS GUI Console, I see that it loaded the US Conformal map and
>then tried to overlay the lightning data, but stops with the error:
>--MDX-No Data Found Check Map & Sort Specs

This is telling us that MDX failed to match the data request that it was
given.  This can happen in a few different ways.  First, if there is no
data, the failure will be as expected.

Second, if the DAY key formats specified in the MDX request (the
NLDNPLT macro simply runs MDX after setting up a number of keys like
DAY in the string table) do not match the DAY format in the data file,
then the SORT condition MDX uses will fail.  What I am getting at here
is the following:

If the ldm-mcidas decoder is filing data into MDXX files using a DAY
format of CCYYDDD, then the MDX command line that specifies DAY
must use that format.  If it uses YYDDD, the match will fail since
things like 99176 does not equal 1999176.

So, the question is what is the form of the DAY key in the NLDN MD
file (I am assuming, of course, that the file does exist; that it
is accessible to the client machine; that it is readable; and, finally
that it does have current data in it)?

>I tried to fix this on my own, but to no avail.  At first, I thought that
>the syskey.tab file might be corrupt in the /var/data/mcidas directory,
>and replaced it.

This was good thinking.

>I then restarted the LDM but still received the same
>error.  Next, I thought that the time on the workstations might be off,
>but after setting that, still nothing.

The time would have to be _way_ off for this to have been the problem.

>The final thing on this weird
>problem is I tried to overlay the lightning data on something current like
>a satellite loop or current temp observation.  Low and behold, the data
>did appear.  This is bizarre and I don't know where to go from here.

I am now heavily betting that the DAY key format in the NLDNPLT request is
different than the DAY key in the MD file.

>If you want, the login info is at the bottom of this letter.

Super, I will do this.  First, however, given that you have installed
the ADDE remote server on your machine, I can do some snooping from
here:

DATALOC ADD RTPTSRC whistler.creighton.edu

Group Name                    Server IP Address
--------------------         ----------------------------------------
RTPTSRC                      WHISTLER.CREIGHTON.EDU

<LOCAL-DATA> indicates that data will be accessed from the local data directory.
DATALOC -- done

DSINFO POINT RTPTSRC

        Dataset Names of Type: POINT in Group: RTPTSRC

Name         NumPos   Content
------------ ------   --------------------------------------
AIRCRAFT        10    Aircraft data
FOUS14          10    FOUS14 data
LIGHTNING       10    Lightning data
PROF6MIN        10    6-Minute Profiler data
PROFHOURLY      10    Hourly Profiler data
PTSRCS         100    All point data in MDXX files
SFCHOURLY       10    SFC Hourly
SHIPBUOY        10    Ship and Buoy data
SYNOPTIC        10    SYNOPTIC data
UPPERMAND       10    Upper Air (Mandatory)
UPPERSIG        10    Upper Air (Significant)

DSINFO -- done

PTLIST RTPTSRC/PTSRCS.ALL FORM=FILE
Pos      Description                        Schema  NRows NCols  Date
------   --------------------------------   ------  ----- ----- -------
     4   SAO/METAR data for   23 JUN 1999   ISFC       72  4500 1999174
     5   SAO/METAR data for   24 JUN 1999   ISFC       72  4500 1999175
    14   Mand. Level RAOB for 23 JUN 1999   IRAB        8  1300 1999174
    15   Mand. Level RAOB for 24 JUN 1999   IRAB        8  1300 1999175
    24   Sig.  Level RAOB for 23 JUN 1999   IRSG       16  6000 1999174
    25   Sig.  Level RAOB for 24 JUN 1999   IRSG       16  6000 1999175
    34   SHIP/BUOY data for   23 JUN 1999   ISHP       24  2000 1999174
    35   SHIP/BUOY data for   24 JUN 1999   ISHP       24  2000 1999175
    44   NGM MOS for day      23 JUN 1999   FO14       38   600 1999174
    45   NGM MOS for day      24 JUN 1999   FO14       38   600 1999175
    54   SYNOPTIC data for    23 JUN 1999   SYN         8  6000 1999174
    55   SYNOPTIC data for    24 JUN 1999   SYN         8  6000 1999175
    64   PIREP/AIREP data for 23 JUN 1999   PIRP       24  1500 1999174
    65   PIREP/AIREP data for 24 JUN 1999   PIRP       24  1500 1999175
    74   NLDN data for        23 Jun 1999   NLDN     1000  1000   99174
    75   NLDN data for        24 Jun 1999   NLDN     1000  1000   99175
PTLIST: Done

From this list, I can tell that your ldm-mcidas NLDN decoder, nldn2md,
is filing data products with a DAY key of YYDDD.  The next question
is how the client appliction NLDNPLT is asking for the data.

I used the ADDE NLDN plot routine to try and display the data:

NLDNDISP LIG OLAY 20
NLDNDISP - Begin
NLDNDISP - Done

No flashes were plotted.  To see if there is any data in your NLDN file,
I once again used PTLIST.  I found that the file for today (DAY 1999175)
only has one record in it.  Yesterday's file, however, has lots of
flashes, and, yet, I was unable to plot them.  I will have to logon
to verify that this is due to a mismatch in the form of the DAY key.

>The second problem stems from the ADDE server.  Everything, as told above,
>is working great except the NIDS data.  I currently am ingesting the NIDS
>data and filing it under both the McIDAS directory in the AREA format and
>the /var/data/ldm/nexrad/*DATATYPE* directory structure (partially so I
>wouldn't have to change all those .CFG files :-)).

I have reworked the NIDS and NOWrad server code recently so that there
is only one configuration file for NIDS data files and one for NOWrad
data files.  Through an implementation of what I call replacables, the
syntax of the data location and filemask make finding files much
simplier.

>I have tried to do an
>IMGLIST RTNIDS/BREF1 but get the error:
>Imglist: error generating list of files
>I had hoped the .CFG files in the McIDAS workdata directory would be set
>incorrectly so that it would be a simple fix, but they are fine.  I then
>tested by loading the SSEC GUI only to find that it exhibited the same
>problem, too.  I double checked and the mcidas account has full access to
>the /var/data/ldm/.... directory so it should not be a permission problem.
>I am just about completely setup for ADDE, and this is my
>last problem.  Do you have any idea on this one?

I am still betting on the syntax of the configuration in the .CFG files.

>I have pulled so much hair out that I am looking like my grandfather

I will look at this when I logon in a couple of minutes.

>To end this novel of a letter, I was bumming around the FTP site today
>looking at some info on the older distributions when I came across the NT
>directory.

Oh no...

>I take it that this is the fabled NT release of McIDAS that
>works with OpenNT that SSEC has been talking so much about?

Yes, it is.  It is pure SSEC code; it has none of the Unidata additions/
modifications/bugfixes in it.

>I was just
>curious if you guys have had a chance to play around with it a great deal

Not yet, but I will coming up.

>and wondered what did you think?

The code is there so that a site that volunteered could play with it on
NT.

>We are looking at getting some more
>computers this next year and if the NT stuff were any good, I would try
>and talk the department into investing into the NT/Interix combo to run
>McIDAS.  Just curious what the overall opinion is.

If you had this combo (you need NT, Interix, an X server, and McIDAS-NT)
now you could play with this distribution and report back on what
you think about it.  Interested?

>That raps it up.  Thanks for your time.  

I will be logging on not in a couple of minutes, but, rather, in a couple
of hours (forgot about a meeting).  I will get back to you on what I
find.

Tom Yoksas