[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #UUO-431027]: LDM to S3

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [LDM #UUO-431027]: LDM to S3
  • Date: Mon, 20 May 2019 08:31:22 -0600
Hi Greg,

re:
> Long time LDM users, we are trying to write to S3 vs our local LDM box.
> 
> This command works, files are written locally:
> 
> EXP   ^(uwis-obs.*)
> FILE  -flush  -close /data/vendor/obs/\1
> 
> We've installed the AWS CLI, but this command isn't writing to S3.
> 
> EXP   ^(uwis-obs.*)
> PIPE  -close aws s3 cp s3://bucketname/vendor/obs/
> 
> Am I missing something obvious?

Assuming that the action that runs 'aws' is properly formatted (i.e.,
tabs as white space where needed), then the likely reason that this 
is not working is that the Python script 'aws' was/is not found in the
PATH of the user running the LDM when the LDM was started, or that
the Python interpreter is also not in the PATH in scope.  The is one
other thing that is different in your action from one that we have
used successfully: 

# Sync all GRB products to AWS
DIFAX   ^/data/cspp-geo/GRB-R/(.*)
        PIPE    -close
        /opt/bin/aws s3 cp - s3://goesingest/\1

In our case:

- we explicitly tell the LDM where the 'aws' script is located

- the directory containing the v3 Python executable is in the
  PATH of our LDM user

Question:

- have you run the action you are using in your pattern-action file
  successfully by hand?

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: UUO-431027
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.