[McIDAS #PKJ-573885]: mcidas: problem using imgcopy

Hi Patrick,

re:
> I am having a problem using the MCIDAS imgcopy command. I suspect it's an 
> issue
> of 'user error'.
>
> *** What I am trying to do is copy data from a remote server to a local
> directory using command line only.

OK.  This should be easy enough...

> *** What I've done:
> 
> I've mapped the remote and local data sets:
> 
> dataloc.k ADD MSG 140.90.195.93

Looks good.

> dsserve.k ADD LOCAL/MSG AREA DIR='/data/scratch/msgglob09i*'  TYPE=IMAGE 
> “M8
> Latest MSG-1 band 9 10.8 um REALTIME

I believe that this is your problem.  The server that will write output files
in AREA format (the ones you want to copy to) needs to know how to name files 
using
AREA naming conventions.  Your ADDE dataset specification does not contain the
needed information to do this.  The DIRFILE= keyword specification for an output
dataset namespace does not contain enough information to uniquely name files.
For instance, a DIRFILE= that looks like:

DIR='/data/scratch/msgglob09i/AREA3*'

might, at first glance, seem to uniquely specify an output name space of
/data/scratch/mgsglob09i/AREA3000, /data/scratch/mgsglob09i/AREA3001, ...
/data/scratch/mgsglob09i/AREA3999, but it does not.  In fact the set of
names specified in this example is _anything_ beginning with AREA3.

What you need to do is tell DSSERVE that you want to create a specific output
namespace.  Here is one example:

dsserve.k ADD LOCAL/MSG AREA 3000 3999 \"M8 Latest MSG-1 band 9 10.8 um REALTIME

The unfortunate part about defining a dataset like this is that McIDAS does not
one to use the DIRFILE= keyword in conjunction with the range of output AREA
file numbers, 3000-3999 in this case.  Instead, you have to tell McIDAS where
the files exist in one of two ways:

MCPATH
REDIRECT

> I've created the directory /data/scratch/msgglob09i and can write to the
> directory as the user running mcidas.

Since you want to write output files to the /data/scratch/msgglob09i directory,
you would specify this directory as the first directory in your MCPATH.  For
example (that assumes that the HOME directory for 'rtoper' is /home/rtoper):

setenv MCDATA /data/scratch/msgglob09i
setenv MCPATH 
${MCDATA}:/home/rtoper/mcidas/data:/usr/local/mcidas/data:/usr/local/mcidas/help

In lieu of a file REDIRECTion, McIDAS will write output to the first writable
directory in the MCPATH set of directories that is in scope when a command is 
run.
So, with this environment setup, you could run:

imgcopy.k MSG/MSGGLOB03I.1 LOCAL/MSG.1

and the resultant output will be the file /data/scratch/msgglob09i/AREA3000

The other way to 'teach' McIDAS about where it should look for files is
by use of the REDIRECT facility:

REDIRECT sets up a match between a file name regular expression and a
specific directory that that/those file(s) will be found/written in.  For
instance, if your MCPATH was defined as:

setenv MCPATH 
/home/rtoper/mcidas/data:/usr/local/mcidas/data:/usr/local/mcidas/help

and you wanted to read/write AREA3000-AREA3999 files in 
/data/scratch/msgglob09i,
you would define the needed file REDIRECTion as follows:

redirect.k ADD AREA3\* \"/data/scratch/msgglob09i

File REDIRECT definitions get saved in the file LWPATH.NAM which should be
located in your McIDAS working directory (which is recommended to be
~user/mcidas/data (e.g., /home/rtoper/mcidas/data) for users other than 'mcidas'
and /home/mcidas/workdata for the user 'mcidas'.

> I can see the remote MSG data sets fine;
> 
> Image file directory listing for:MSG/MSGGLOB03I
> Pos Satellite/         Date       Time      Center      Res (km)   Image_Size
> sensor                                 Lat  Lon    Lat   Lon
> --- -------------  ------------  --------  ---- ----  ----- ----- ------------
> 1  METEOSAT8     26 FEB 07057  22:00:00     0    0
> Band: 3    1.6 um Near IR - Surface, cloud phase     3.02  3.00  3712 x 3712

Very good.  (For the records, access to the MSG dataset on 140.90.195.93 
requires
one to LOGON specifying a user name and project ID that has been assigned by
those maintaining McIDAS on 140.90.195.93).

> *** When I attempt to copy from MSG/MSGGLOB03I to LOCAL/MSG I get an error:
> 
> imgcopy.k LOCAL/MSGTMP.1  LOCAL/MSG.1
> Beginning Image Data transfer, bytes= 616660
> Transferring AREA data outbound, bytes= 616740
> imgcopy.k: MCAOUT rc= -1
> imgcopy.k: done

A couple of things here:

- you are not trying to copy from the remote ADDE dataset that you referred to
  previously

- the LOCAL/MSG dataset is not defined in a way that McIDAS will know how to
  name the output file

> *** I am sure I am missing something but not sure what. I'd appreciate any 
> help!

You have run into one of the areas of McIDAS that requires more intimate 
knowledge
of how things work.  Please let me know if my explanation about the output
namespace above was clear.

> *** Here is a data dump of everything I've set for this scenario:
> 
> setenv MCHOME /usr/local/mcidas
> setenv MCDATA $MCHOME/mcidas2006/data
> setenv MCSRC $MCHOME/mcidas2006/src
> setenv MCPATH $HOME/:$MCHOME/:$MCDATA/:$MCSRC
> setenv MCGUI $MCHOME/bin
> unsetenv MCTABLE_READ
> unsetenv MCTABLE_WRITE
> setenv PATH $PATH/:$MCPATH/:$MCGUI
> 
> running this as user 'rtoper'

I have several recommendations for how to define the various environmental
variables:

- you don't need MCSRC

- I strongly recommend you define MCDATA to be /home/rtoper/mcidas/data

- I recommend that your MCPATH be defined as:

  setenv MCPATH ${MCDATA}:$MCHOME/data:$MCHOME/help

- I recommend that you define MCTABLE_READ and MCTABLE_WRITE:

  setenv MCTABLE_WRITE $MCDATA/MCTABLE.TXT
  setenv MCTABLE_READ $MCTABLE_READ;$MCHOME/data/ADDESITE.TXT  

- since there should be no McIDAS executables in your $MCPATH directories,
  I see no reason to include MCPATH in your PATH definition

> file perms for /data/scratch/msgglob09i:
> drwxr-xr-x  2 rtoper fd 4096 Feb 27 17:58 msgglob09i
> 
> dataloc.k ADD GER 140.90.195.64
> dataloc.k ADD GWR 140.90.195.66
> dataloc.k ADD MSG 140.90.195.93
> dataloc.k ADD PUB gp16.ssd.nesdis.noaa.gov
> 
> dsserve.k ADD LOCAL/MSG AREA DIR='/data/scratch/msgglob09i*'  TYPE=IMAGE 
> “M8
> Latest MSG-1 band 9 10.8 um REALTIME

As noted above, the definition for LOCAL/MSG is incorrect for an output dataset.

Please let me know if the information above was helpful.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: PKJ-573885
Department: Support McIDAS
Priority: Normal
Status: Closed