Re: LDM upgrade

Gini,

>Date: Thu, 16 Jun 2005 12:46:00 -0500
>From: Virginia Galvin <address@hidden>
>Organization: NOAA/NWS/TOC
>To: address@hidden,
>To: Virginia Galvin <address@hidden>
>Subject: LDM upgrade 

The above message contained the following:

> TOC will be upgrading TOC-LDM from ldm.6.0.14 to ldm 6.1.0 on Monday June 20
> TOC-LDM is now running RHLinux AS (Linux 2.4.21-27.0.2.ELsmp) and
> TOC-LDM  now runs with  a 2GB queue on LDM 6.0.14.

I belive that's a 32-bit operating-system, so 2 gigabytes might be the
maximum possible size for the product-queue without upgrading the
system.

Executing the pqmon(1) utility will show you the age of the oldes
product in the product-queue, so you can determine if you actually need
a larger queue.

> Last time TOC compiled LDM because we are use non-standard directory 
> path (/home/ldm/runtime)--
> so I expect we should do the same this time?

Yes.  Feel free to ask me to lead you through it (see below).

> Is there any special configuration we need to do to ensure rstats will 
> report up to 300 data products bins?

The rtstats(1) utility in version 6.1 of the LDM package will handle
4000 bins by default -- so you'll be fine.

> We are going to LDM 6.1.0 to maintain consistency with MAX-LDM, but  can 
> you tell us what would  be the advantage of going to latest LDM instead?

I've attached that portion of the CHANGE_LOG that deals with changes 
from version 6.1 to 6.3.  I think the most important items for you are
the following:

        Added "--enable-logging=localn" option to configure(1) script to
        support the use of a different logging facility and permit the
        running of two LDM-s simultaneously.

        Modified LDM configuration-file parser to prevent invalid
        entries like

            REQUEST IDS/DDPLUS .* host.domain

        from being misinterpreted as requesting data-products of
        feedtype "IDS" that match the regular-expression "/DDPLUS".
        Such entries are now detected as being invalid.

        Added "rpc" subpackage -- replacing use of the native RPC
        library -- to work-around a bug in the AIX 5.1 ONC RPC 4.0
        implementation, which fails when receiving large (~10 MB)
        RPC messages (i.e., most NIMAGE data wasn't being received).

        Modified product-queue (pq) module:

            Corrected bug in pq(3) module when inserting a data-product
            that has the same insertion-time as an already existing
            data-product.  The insertion-time in now incremented by
            one microsecond to ensure unique keys in the time-map
            rather than using the byte-offset of the data-product.
            Hopefully, this will eliminate the rare occurrence in which
            a data-product is missed by a product-queue reader because
            it has the same insertion-time as the previous data-product
            in the product-queue but has a smaller byte-offset.

            Corrected fClr() and fMask() macros in file "pq/fbits.h"
            so that they correctly handle the case where the "flag"
            variable is smaller than the variable in question.  This
            removes a problem creating product-queues with data-sections
            larger than (2^32)-1 bytes on systems where sizeof(size_t)
            == 8 and sizeof(unsigned) == 4.

        Made all programs that use regular-expressions convert all
        externally-specified pathological regular-expressions into
        non-pathological ones.

        Modified top-level LDM server:

            Removed latent bug that caused file-descriptor table to
            fill-up -- preventing additional connections -- if fork(2)s
            failed for a while.

        Modified downstream (i.e., receiving) LDM:

            Changed the way the "last" data-product creation-time is
            saved.  Before, the creation-time of the most recently
            received data-product was used.  Now, that time is used
            only if it is more recent than the saved time.  This
            should reduce problems caused by the sequential arrival of
            data-products with creation-times that are non-monotonic.

            Improved error-messages when the LDM can't connect to an
            upstream LDM.

        Modified programs to reduce artificially-induced latencies:

            Modified the toClients() function in the "pqing" module so
            that the "arrival" argument is ignored and the creation-time
            of the data-product is set within the function itself.
            This should reduce the (apparent) latency on systems doing
            data-product ingestion.

            Modified the pqinsert(1) utility so that the data-product
            creation- time is set just prior to inserting the
            data-product into the product-queue.  This should reduce any
            (apparent) latency.

        Modified rtstats(1):

            The latency field in the data-product it creates is now
            formatted "%g" from a floating-point value.

        Modified ldmadmin(1) script:

            Moved the configuration section of ldmadmin(1) into a
            separate file (etc/ldmadmin-pl.conf).  A consequence of this
            is that if the environment variable LDMHOME is not set, then
            $HOME is used.

            Fixed bug in ldmadmin(1) so that "ldmadmin watch -f
            'IDS|DDPLUS'" now works.

            Made "ldmadmin clean" abort if the LDM system is running.

        Ported package to
            
                Darwin (alias Mac OS X)
                SunOS 5.10 (alias Solaris 10)

            The package has poor performance under SunOS 5.10.  Sun is 
            investigating.

        Modified configure(1) script:

            Made the script try to create an LDM system that supports
            product-queue sizes up to (2^32)-1 bytes.

            Added "--disable-max-size" option so that the user can
            ensure that the resulting LDM only supports smaller
            product-queue sizes (to use a previously-existing
            product-queue, for example).

(Yes, I've been busy :-)

> Will you or Steve Emmerson be available on Monday if we have problems?

I'll be available.

> How is the  best way to reach you if we need you?

Email.  If you don't get a timely response, then phone me at work at
303-497-8648.  Or just phone me first thing.

I usually work at home in the morning until rush-hour traffic has
subsided.  When is the earliest that you might call?

> Thanks
> gini

Regards,
Steve Emmerson

Attachment: t.t
Description: t.t