20000624: McIDAS addenda on RH 6.2 (cont.)

>From: "James D. Marco" <address@hidden>
>Organization: Cornell
>Keywords: 200006231803.e5NI3JT07000 McIDAS-X addenda RedHat 6.2 Linux

Jdm,

re: running out of virtual memory
>Yup, the compiler system grabbed more than 100M when it got to the file,
>as I remember. (I ran 'top' during one run to monitor.) I puzzled over
>it too, hence the question about the new code containing the update. 
>Probably is a bad file.

Phew... (for me at least).

>I had to ftp to an older machine, unpack there in 
>a clean directory, create a new tar starting in the 'src' directory, ftp
>the file back to the new machine and unpack from the 'src' directory
>there...plus the download...lots of file and ftp work going on. Anyway, 
>it's OK now. 

I am assuming you did this because the RedHat 6.2 version of tar doesn't
understand relative file directories (e.g., ../src).  Please let me know
if this is not the case.

By the way, if relative file directory access/storage is no longer
available in RedHat 6.2 Linux, then I have an ugly problem on my hands.
The fact that I can not dictate that everyone's 'mcidas' HOME directory
be something specific would force me into a situation of telling people
to cd to a particular directory and then do the unpack of the tar file.
I guess the other alternative is to create a link to the src directory
in the update directory and then have files referenced as src/xxx.
If you could shed any more light on this tar snafu, I would be most
appreciative.

>If you don't hear any more on this, 'cuz it's MY problem, not a 
>distribution problem, write it off as a corrupt file and a hacked
>hacker hack. Sorry for the bother.  

No problem.  I am definitely glad that you let me know about the tar
change before I had to stumble on it myself.

>(I've seen the same problem on old Sun machines running Oasis 
>C++ and thought there might be a macro/pre-processor glitch in the
>code caused by some macro's nested too deeply...cpp eats memory and
>ccom can't run.)      

Luclily, statdisp.cp is not that complicated.

>Thanks for your help!

You are more than welcome.

Tom

>From address@hidden Sun Jun 25 06:27:57 2000
>Subject: Re: 20000624: McIDAS addenda on RH 6.2 (cont.)
Tom,
        In context below...

>By the way, if relative file directory access/storage is no longer
>available in RedHat 6.2 Linux, then I have an ugly problem on my hands.
>The fact that I can not dictate that everyone's 'mcidas' HOME directory
>be something specific would force me into a situation of telling people
>to cd to a particular directory and then do the unpack of the tar file.
>I guess the other alternative is to create a link to the src directory
>in the update directory and then have files referenced as src/xxx.
>If you could shed any more light on this tar snafu, I would be most
>appreciative.
I think I can test that and get the exact errors from here (home).  
Hang on . . . OK.
I generated an errlog:  "mcunpack >errlog 2>&1"     (bash syntax)

I included it below. An afterthought for any other readers: 
Don't try to include non-text (including tar format) files in mail messages. 

The error is delayed, typically due to cache mechanisms. Relavent error
is: "tar: <../fname>: Member name contains `..'

I checked the man pages during the update attempt, but didn't see
any setting to disable the name checking.  A new possibility that occurs
as I'm writing this...check the emacs-info on tar...hang on... 
Nope.  CURRENT docs are not a linux strength...Info dates back to 1994.

During the update attempt, I repeated the entire download/update sequence
twice. Soooo, this is probably not a network induced glitch. (I've
been assuming this, but wanted you to feel comfortable with the basis for 
that assumption.)

The error is reproducible.  I just ran mcunpack this morn and got the 
same error as in the previous two attempts. ("Tell me three times")   

Comment:
        It appears that GNU-tar has been modified to prevent parent 
directory access. (This is reasonable for users on a machine, but bad
for administrators. We need an administrative override flag...maybe there 
is one, but it's undocumented.)  
          
Fix:
        There are lots of fixes. I don't know how many other places you
do this, so I cannot determine the best fix. Quick analysis...
(Probably resembling my head...full of holes.)
        Yup. A link will work as you say.
                In the mcunpack script:
                "ln -s ../src src"
        Note: In some cases, this fails.  If the update directory is
        created as a link to an update version somewhere else on the
        disk (ex. update -> /archive/mcidas/update/ldm-mcidas.7.6.3), you 
        will link to the DISK reference (inode), not the directory
        hierarchy...the link may fail. 

        The weakest pre-condition (assumption), causing the failure, is:
                ".." always points to parent directory
        - In this case, the McIDAS "src" directory) -  
        This may not always be true. Both 'ln' and 'mount' break 
        this assumption regularly.      

        Less technical fix:
                Modify the script to copy the archive to the parent directory:
                        "cp <update_file.tar.Z> .."
                Change the current working directory to ".." in the script: 
                        "cd .."
                Call tar from there:
                        "tar -xZvf <update_file.tar.Z>"
                Clean it up:
                        "rm <update_file.tar.Z>"
                And exit back to the users shell:
                        "exit 0"

        Note: If the update directory is actually a link, the "cd .." is
a one-way change. This is why I mention the "exit 0".  You could do a
pushd/popd but this will fail in some shells. Also, it is prone to the 
same failure type, as above, sooo, doesn't portably solve the problem.  
Note: bash maintains the directory hierarchy for links, not the disk 
reference, sooo, it does not fail. sh, ksh, csh and tcsh will fail.   

        Best(??) fix: (I like this one, you will not.)
                Define a shell variable in the base installation that 
        points to the Mcidas version. 
                This is like the 'ldm' install, soooo, it's more complicated
        to set up. But the set up process uses shell variables this way,
        anyway:  McINST_ROOT=/home/mcidas; export McINST_ROOT
        Add (bash syntax):
         McINST_VERSION=$McINST_ROOT/mcidas7.6;export McINST_VERSION    
        (Might be a handy variable for other archival purposes, too. Disk
        space is cheap these days...)   

        Advantage:(obvious, but stated here for other readers.) 
                Once you define the source tree, you are free to work with 
        the absolute path available in the Shell Variables.  This is 
        flexible enough for users to create RCS/version trees, and still
        maintain version independent references within a single source 
        tree for required programatic definitions.      

        Downside:
                Lot's of tweaking to use the new variable in the McIDAS
        system.  But, you can implement this immediatly in the patches
        (update) and add in to the rest of the code as you go.
        
        The fix becomes:
                In the mcunpack script:
                        "cp <update_file.tar.Z> $McINST_VERSION/src"
                Change the current working directory to "$McINST_VERSION/src"
                        "cd $McINST_VERSION/src"
                Call tar from there:
                        "tar -xZvf <update_file.tar.Z>"
                Clean it up:
                        "rm <update_file.tar.Z>"
                You don't need to exit, now. Sooo, you can run any post-update
                stuff you may need. Just change back to the update directory.
                        "cd $McINST_VERSION/update"

        A quick state analysis indicates:

                (start_state)-->(start_update)-->(halt_state)

                The start_state and halt_state are both the same directory
                and, disregarding the actions of the update, nothing
                else has changed, thus a high degree of "correctness"
                is present for the mechanism.

Anyway, 
        Lots'a ways to do a fix. A Makefile will give better error controll
        for updates but is more to code and maintain. And so on... 
        Pick this apart for logical errors, do time/cost estimates, etc.  
        One last comment, the GNU-tar is the standard on a large number 
        of UNIX platforms. As the OS's update, the problem is not
        likeley to go away, sooo, I undestand your concern. 
                        
                        Hope this helps...
                                                jdm (jim) 


Contents of mcunpack errlog:
*** snip ****** snip ****** snip ****** snip ****** snip ****** snip ***
unpacking update files...
../src/ftops_.c
tar: ../src/ftops_.c: Member name contains `..'
../src/idu.pgm
tar: ../src/idu.pgm: Member name contains `..'
../src/decoder.cp
tar: ../src/decoder.cp: Member name contains `..'
../tcl/generic/tclPosixStr.c
tar: ../tcl/generic/tclPosixStr.c: Member name contains `..'
../netcdf/ncgen/ncgentab.c
tar: ../netcdf/ncgen/ncgentab.c: Member name contains `..'
../netcdf/ncgen/ncgenyy.c
tar: ../netcdf/ncgen/ncgenyy.c: Member name contains `..'
../src/M0getsplbyt.c
tar: ../src/M0getsplbyt.c: Member name contains `..'
../src/Graphics.c
tar: ../src/Graphics.c: Member name contains `..'
../src/m0uclock.c
tar: ../src/m0uclock.c: Member name contains `..'
../src/mapurx.for
tar: ../src/mapurx.for: Member name contains `..'
../src/mcasort.for
tar: ../src/mcasort.for: Member name contains `..'
../src/mcexptxt.for
tar: ../src/mcexptxt.for: Member name contains `..'
../src/mci_buff.c
tar: ../src/mci_buff.c: Member name contains `..'
../src/mci_updt.c
tar: ../src/mci_updt.c: Member name contains `..'
../src/mcidasx.sh
tar: ../src/mcidasx.sh: Member name contains `..'
../src/ptdisp.pgm
tar: ../src/ptdisp.pgm: Member name contains `..'
../data/GRIDTIME.MCB
tar: ../data/GRIDTIME.MCB: Member name contains `..'
../src/mcgui.k
tar: ../src/mcgui.k: Member name contains `..'
../src/cotv.pgm
tar: ../src/cotv.pgm: Member name contains `..'
../src/eu.pgm
tar: ../src/eu.pgm: Member name contains `..'
../src/la.pgm
tar: ../src/la.pgm: Member name contains `..'
../src/m0rabdec.for
tar: ../src/m0rabdec.for: Member name contains `..'
../src/maknav.pgm
tar: ../src/maknav.pgm: Member name contains `..'
../src/mapdef.for
tar: ../src/mapdef.for: Member name contains `..'
../src/mdksserv.fp
tar: ../src/mdksserv.fp: Member name contains `..'
../src/nldndisp.pgm
tar: ../src/nldndisp.pgm: Member name contains `..'
../src/strnvals.gui
tar: ../src/strnvals.gui: Member name contains `..'
../data/MDRDEC.CFG
tar: ../data/MDRDEC.CFG: Member name contains `..'
../data/SYSKEY.DOC
tar: ../data/SYSKEY.DOC: Member name contains `..'
../netcdf/configure
tar: ../netcdf/configure: Member name contains `..'
../src/redara.for
tar: ../src/redara.for: Member name contains `..'
../src/wrtara.for
tar: ../src/wrtara.for: Member name contains `..'
../src/GVAR.c
tar: ../src/GVAR.c: Member name contains `..'
../src/POES.c
tar: ../src/POES.c: Member name contains `..'
../src/SDIUtil.c
tar: ../src/SDIUtil.c: Member name contains `..'
../src/adump.pgm
tar: ../src/adump.pgm: Member name contains `..'
../src/agetserv.fp
tar: ../src/agetserv.fp: Member name contains `..'
../src/crc16.c
tar: ../src/crc16.c: Member name contains `..'
../src/crc16.h
tar: ../src/crc16.h: Member name contains `..'
../src/ddeservc.c
tar: ../src/ddeservc.c: Member name contains `..'
../src/imgprobe.pgm
tar: ../src/imgprobe.pgm: Member name contains `..'
../src/mcfc.sh
tar: ../src/mcfc.sh: Member name contains `..'
../src/mci_updt.c
tar: ../src/mci_updt.c: Member name contains `..'
../src/navapi.c
tar: ../src/navapi.c: Member name contains `..'
../src/navcalc.c
tar: ../src/navcalc.c: Member name contains `..'
../src/navdisp.c
tar: ../src/navdisp.c: Member name contains `..'
../src/new_ncftle.c
tar: ../src/new_ncftle.c: Member name contains `..'
../src/M0getsplbyt.c
tar: ../src/M0getsplbyt.c: Member name contains `..'
../src/poesadir.cp
tar: ../src/poesadir.cp: Member name contains `..'
../src/poesaget.cp
tar: ../src/poesaget.cp: Member name contains `..'
../src/grdimage.pgm
tar: ../src/grdimage.pgm: Member name contains `..'
../src/imgoper.pgm
tar: ../src/imgoper.pgm: Member name contains `..'
../src/mcinet.sh
tar: ../src/mcinet.sh: Member name contains `..'
../src/mcmkhelp.c
tar: ../src/mcmkhelp.c: Member name contains `..'
../src/mcsound.for
tar: ../src/mcsound.for: Member name contains `..'
../src/mcgput.for
tar: ../src/mcgput.for: Member name contains `..'
../src/giniadir.c
tar: ../src/giniadir.c: Member name contains `..'
../src/giniaget.c
tar: ../src/giniaget.c: Member name contains `..'
../src/giniutil.c
tar: ../src/giniutil.c: Member name contains `..'
../src/giniutil.h
tar: ../src/giniutil.h: Member name contains `..'
../src/nidsadir.c
tar: ../src/nidsadir.c: Member name contains `..'
../src/nidsaget.c
tar: ../src/nidsaget.c: Member name contains `..'
../src/nidsutil.c
tar: ../src/nidsutil.c: Member name contains `..'
../src/nidsutil.h
tar: ../src/nidsutil.h: Member name contains `..'
../src/nowradir.c
tar: ../src/nowradir.c: Member name contains `..'
../src/nowraget.c
tar: ../src/nowraget.c: Member name contains `..'
../src/nowrutil.c
tar: ../src/nowrutil.c: Member name contains `..'
../src/nowrutil.h
tar: ../src/nowrutil.h: Member name contains `..'
../src/servutil.c
tar: ../src/servutil.c: Member name contains `..'
../src/servutil.h
tar: ../src/servutil.h: Member name contains `..'
../src/mdx.pgm
tar: ../src/mdx.pgm: Member name contains `..'
../src/xsectg.pgm
tar: ../src/xsectg.pgm: Member name contains `..'
../src/upc.mac
tar: ../src/upc.mac: Member name contains `..'
../data/CORE.SAT
tar: ../data/CORE.SAT: Member name contains `..'
../data/DATALOC.BAT
tar: ../data/DATALOC.BAT: Member name contains `..'
../data/MASTERID.DAT
tar: ../data/MASTERID.DAT: Member name contains `..'
../src/fousp.mac
tar: ../src/fousp.mac: Member name contains `..'
../src/isenp.mac
tar: ../src/isenp.mac: Member name contains `..'
../src/nldnplt.mac
tar: ../src/nldnplt.mac: Member name contains `..'
../src/sfcplt.mac
tar: ../src/sfcplt.mac: Member name contains `..'
../src/spc.mac
tar: ../src/spc.mac: Member name contains `..'
../data/GINI.CFG
tar: ../data/GINI.CFG: Member name contains `..'
../data/GINIADDE.BAT
tar: ../data/GINIADDE.BAT: Member name contains `..'
../data/NOGRIB.CFG
tar: ../data/NOGRIB.CFG: Member name contains `..'
../src/grib.h
tar: ../src/grib.h: Member name contains `..'
../src/makefile
tar: ../src/makefile: Member name contains `..'
../src/mccomp.sh
tar: ../src/mccomp.sh: Member name contains `..'
../src/Mcgribdecoder.c
tar: ../src/Mcgribdecoder.c: Member name contains `..'
../src/sndhodo.pgm
tar: ../src/sndhodo.pgm: Member name contains `..'
../data/DSSERVE.BAT
tar: ../data/DSSERVE.BAT: Member name contains `..'
../data/GRIBDEC.CFG
tar: ../data/GRIBDEC.CFG: Member name contains `..'
../data/HRS.CFG
tar: ../data/HRS.CFG: Member name contains `..'
../data/POINT.PROG
tar: ../data/POINT.PROG: Member name contains `..'
../data/PROGRAM.DAT
tar: ../data/PROGRAM.DAT: Member name contains `..'
../data/PROGRAMS.PROG
tar: ../data/PROGRAMS.PROG: Member name contains `..'
../data/SATBAND
tar: ../data/SATBAND: Member name contains `..'
../data/gbtbpds001.a74v1
tar: ../data/gbtbpds001.a74v1: Member name contains `..'
../data/gbtbpds001.a98v1
tar: ../data/gbtbpds001.a98v1: Member name contains `..'
../src/mcmenu.c
tar: ../src/mcmenu.c: Member name contains `..'
../src/mdfix.pgm
tar: ../src/mdfix.pgm: Member name contains `..'
../src/upcguiprocs.tcl
tar: ../src/upcguiprocs.tcl: Member name contains `..'
../src/uwgrid.pgm
tar: ../src/uwgrid.pgm: Member name contains `..'
../src/CHANGES.760
tar: ../src/CHANGES.760: Member name contains `..'
../data/gbtbpds001.2v1
tar: ../data/gbtbpds001.2v1: Member name contains `..'
../data/gbtbpds001.2v2
tar: ../data/gbtbpds001.2v2: Member name contains `..'
../data/gbtbpds001.2v3
tar: ../data/gbtbpds001.2v3: Member name contains `..'
../data/gbtbpds001.av1
tar: ../data/gbtbpds001.av1: Member name contains `..'
../data/RTMODELS.CFG
tar: ../data/RTMODELS.CFG: Member name contains `..'
../data/UNIDATA.MNU
tar: ../data/UNIDATA.MNU: Member name contains `..'
../data/VERSION.TXT
tar: ../data/VERSION.TXT: Member name contains `..'
tar: Error exit delayed from previous errors
mcunpack done


James D. Marco, address@hidden, address@hidden
Programmer/Analyst, System/Network Administration, 
Computer Support, Et Al. 
Office:                 1020 Bradfield Hall, Cornell University 
Home:                   302 Mary Lane, Varna    (607)273-9132
Computer Lab:   1125 Bradfield          (607)255-5589