20000329: Info on hardware for LDM ingestion and decoding (cont.)

>From: "Neil R. Smith" <address@hidden>
>Organization: Dept. Atmospheric Science, TAMU
>Keywords: 200003202042.NAA27616 Unidata platforms LDM decoders GEMPAK McIDAS 
>DEC Alpha True64

Neil,

>1. What's the current word on the DEC(Compaq) Alpha and True64?  I saw
>some folks having problems building McIDAS.

I support McIDAS on DEC OSF/1 4.0 D.   I have helped a site build it
on OSF/1 4.0 E, but I have no experience with True64.  In fact, since
we don't have access to such a machine here at the UPC, none of us
have experience with True64.  One additional note, DEC's support of
Java is not at the forefront.  Given our Java development efforts,
I could not recommend the True64 platform with a clear consience.

>2. The Alpha SPECfp's seem to stomp on anything else around, making
>them real attractive.

What about things like compiler cost?

>But what are good measures to look for in
>evaluating for a dedicated ingest/decoder platform?  If I/O, what
>specifically, can you look at?

Ingest/decode activities of the LDM and various decoders (GEMPAK,
McIDAS-XCD, netCDF decoders, etc.) are pretty tame.  They do
not need the hotest compute machine around in order to work.  In
fact, we have been having _very_ good success with relatively cheap
PCs running Solaris x86.  Our latest purchase was two of the following:

January 26, 2000

Base System:  Dual Intel Pentium III 700 Mhz Coppermine CPU w/256 KB Cache
                ASUS P2BDS Dual CPU SCSI ATX BX AGP Motherboard
                Single Edge Connector Pentium II Slots
              One AGP, Four PCI and Two ISA expansion slots
              (2) 16550 serial ports and (1) hight-speed parallel port
              AGP Video Support and Two Stacked USB Connectors
              Integrated PCI Ultra DMA Hard/Floppy Drive Controller
              Teac 1.44 MB Floppy Drive
              Qty 2 - High-end ball-bearing CPU Fan
              Enlight 8902 Full Tower Case with 300 Watt UL LIsted Power Supply

Memory:       756 MB SDRAM (Ztech to Provide ECC Registered Memory)

Hard Drive:   Qty 1 - 18.2 GB IBM 10,000 rpm U2W SCSI hard drive w/2 MB Cache
              Qty 1 - Enhance Technology ER4610 Removable Hard Drive Rack
                w/lock and fan

Controller:   On-board Adaptec Ultra 2 Wide Chipset
              On-board Bus Mastering Ultra DMA IdE controller supporting 4
                devices

Monitor:      21" KDS VS-21E SVGA monitor

Video Card:   8 MB ATI Xpert 98 ABP 2x video accelerator

Multimedia:   Creative Labs 52X IDE CD-ROM

Network:      3Com 905B-TX PCI network adapter with driver software

Software:     N/A (customer supplied)

I/O:          Keytronics PS2 Windows 95 keyboard
              Logitech 3-button PS2 mouse

Warranty      Technical Reference and Manuals
and Support:  Lifetime Toll-Free technical support
              24 hours of testing
              3 year Parts and Depot Labor Warranty

Price:        $5,295
 
Contact:      Lyndon V. Hanson
              Vice President - Contour Systems Group
              contoursys.com

On a machine much slower than these, we are ingesting all four NOAAPORT
channels, all of the IDD data, CONDUIT data, NIDS data, and we are
running the full suite of our decoders.  In addition, we are running
the McIDAS ADDE remote server and letting more and more people access
GINI imagery from it.  The machine is really not even breathing hard!

>3. Re optimizing any platform, is it worth the effort to
>put the queue files on a dedicated SCSI controller/disk, say 10k rpm, 
>separate from the "data" directory where product and decoded data are
>written to?  -considering that the ldm is to feed ~5-8 downstream
>sites and the OS will NFS-serve /data to 40-50 intra-subdomain clients.
>The ldm platform would have 100Mb ethernet.

Right up until the point about NFS serving 40-50 intra-subdomain
clients, I would say no.  Given that you are going to be supporting
huge amounts of NFS traffic, then I would think that you might want to
keep as much activity off of the data file system as possible.  For
more authoritative answer on this, however, I will have to consult our
system administrator (who is not here today).

>What is the impact of impending NIDs offering on the IDD for an ldm
>platform that will take it and serve it downstream?

Well, that is a good question indeed!  Given that there are on the
order of 160 NEXrads (out of which 140 may be available), and given
that there will be around ten NIDS products per NEXrad, and given that
the frequency of products for NEXrads running in "storm mode" is high
(one of each type of product every five minutes), there could be up to
35 GB of NIDS data available per day!  It may make the most sense to
not try and transmit all of the NIDS products through the IDD given
that the set of stations that a site would probably be interested in
would be a small subset of the sites available.  For this kind of
data it might make more sense to access it through data servers like
McIDAS ADDE.

The real impact on the current LDM is not so much in the amount of
data; it is the number of products.  New work at the UPC on the LDM
product queue will help to eliminate problems that users are now
experiencing (LDM induced latencies).  The sheer volume of NIDS data,
however, will undoubtedly present some new challanges.

>Thanks, -Neil

Have you checked into higher end PC plaforms?

Tom Yoksas