[IDD #ADB-747400]: 20220822: Request advice on ldm setup for file transfer

Hi Dave,

First, I drove your email into our inquiry tracking system so that
other Unidata staff can see the inquiry and chime-in if/when needed.

re:
> I am attempting to configure 3 standalone AWIPS instances to sync ISC
> data from each instance.   I would like to use LDM to achieve this
> file transfer.  What is the best way to set this configured.
> 
> Below are the details coin
> 
> Below is workflow snip:
> 
> 1) Instance A saves an ISC file.  A script on Instance A runs pqinsert
> on the file name *ISCPackage_RandomNumber.zip*

OK.  Unless you specify otherwise, the feedtype that will be assigned
to the product upon product queue insertion is EXP.  You can change
this with a flag to 'pqinsert' if you would rather that the product
be classified in a different feedtype.

re:
> 2) Instance A has an ALLOW for INSTANCE B AND C

Very good.

re:
> 3) Instance B has a request for ISCPackage.*

OK.

re:
> 4) Instance B pqact.conf will store the file in /tmp/ and then be acted
> on it.

OK. You didn't mention instance C...

re:
> Is ALLOW and REQUEST the most efficient way to send/receive these ISC
> files.  Should I use ACCEPT?

One can send products "out of band" using 'ldmsend', but one must remember
two things before doing this:

- there needs to be an ACCEPT line in the downstream machine(s) LDM 
  configuration files

  Remember, any time a change is made in the LDM configuration file, the
  LDM on that machine will need to be stopped and restarted.

- 'ldmsend' uses LDM 5 protocols

  The LDM 5 protocol is more or less the upstream machine offering the
  product to the downstream, and then the downstream acknowledging that
  it does, in fact, want the product, and then the upstream actually
  sends the product.

  The LDM 6 protocol is used when the downstream LDM makes a standing
  REQUEST to be feed products (specifying the feedtype and an
  extended regular expression that descibes the product/set of products);
  the upstream LDM is configured to ALLOW the REQUEST; and then any time
  one of the products requested is inserted into the upstream's LDM
  queue, the product is sent.

  The LDM 6 way of moving products is much more efficient than the LDM 5
  way.

re:
> Current setup
> 
> LDMD.CONF
> ##########################################################
> ## CLOUD INSTANCE ISC SETUP
> ##     ALLOW ANY downstream LDM-s to request data-products from your LDM
> ##     REQUEST ANY Request data-products from upstream LDM-s.
> ############################################################
> ALLOW   ANY     ^10.27.10.188   .*
> REQUEST ANY     "ISCPackage.*"  10.27.10.188

I am assuming that this ldmd.conf snippet is on the downstream.  The
only thing that would be needed on the upstream is an ALLOW that will
allow either/both B or C to REQUEST products.

re:
> PQACT.CONF
> #####################################
> #ISC Traffice instance to instance
> ######################################
> ANY     (ISCPackage.*)
>         FILE    -log -close     /tmp/\1

This looks fine.

So, what about machine C?  For a really robust setup, you could
setup C to also REQUEST the products from A, and then setup both
B and C to REQUEST from each other.  You will not need to worry 
about the downstreams processing the same product more than once
since duplicate products should be detected and rejected by the
downstream LDMs thus preventing duplicate products from being
inserted into the queue.  Identification of a duplicate products
is done by comparing a products MD5 signature with products already
in the LDM queue, and then throwing the ones away that are already
in the queue.

One thing to be on the lookout for:

If one runs 'pqinsert' outside of the LDM process group, then upstream
LDM will not notice the product has been inserted until a 30 second
timeout has occurred *when there is no other activity in the queue*.
If the machine is busy, then the product will be processed along with
the other products that have been received/inserted into the queue,
so the "artificial" 0 - 30 second will not be a factor.  I can explain
this at more length/better if in a Meet if you are interested.

> Thanks
> Dave
> 

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ADB-747400
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.