[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
20000317: NIDS and NOWrad setup (cont.)
- Subject: 20000317: NIDS and NOWrad setup (cont.)
- Date: Sat, 18 Mar 2000 11:35:50 -0700
>From: "Karli Lopez (McIDAS)" <address@hidden>
>Organization: University of Puerto Rico
>Keywords: 200002010754.AAA03412 NIDS NOWRAD ADDE NIDS.CFG
Karli,
re: 'WSI<tab>...' line in pqact.conf
>Oh ok, that's what i'd done (merge it into a single line) but I missed placing
>the first tab before the "FILE" entry.
>I had:
>----------------
>WSI<tab>...<newline>
>FILE<tab>...
>----------------
>instead of:
>----------------
>WSI<tab>...<newline>
><tab>FILE<tab>....
>----------------
>it is now "syntactically correct"!
Excellent.
>Now do I need the directories to be /data/.../directoryname/NIDS, or just /dat
> a/.../directoryname where "/data/directoryname" is the directory where the
>files are located (in out case it was set up orginally as "/unidata/ldm/data/m
> cidas/")
>my guess is that it's just the directory name and that NIDS is not
>necessary, but I want to be sure.
You are right. The location information in pqact.conf is where you
want to put the data. This is up to you. The DIRMASK entry in the
McIDAS ADDE configuration file for NIDS data,
~mcidas/workdata/NIDS.CFG, will need to be set to match whatever
directory location you choose.
The concept here is that the NIDS ADDE server needs to be able to find
where the NIDS data files are and how they are named. The regular
expression that is the value for the DIRMASK= parameter in NIDS.CFG
describes where the files are located. The regular expression for the
FILEMASK= parameter in NIDS.CFG describes how the files are named.
Perhaps a little "graphical" example of how this would look might be
useful.
Remember, these are only examples; you have to setup your
system according to the locations you choose for the data files:
pqact.conf entry says:
WSI
^NEX/(.*)/(.*)/([1-2][0-9])([0-9][0-9])([0-1][0-9])([0-3][0-9])([0-2][0-9])([0-6][0-9])
FILE data/nexrad/\1/\2/\2_\3\4\5\6_\7\8
In this example, data/nexrad/... refers to the ~ldm/data/nexrad... directory.
Remember that ~ldm/data is typically a link to some other directory. For
the purpose of illustration, let us assume that ~ldm/data is a link to
/data.
The files will be stored in directories that look like:
cd ~ldm/data/nexrad
pwd
/data/nexrad
ls
ABX CYS ENX FTG GRK OKX PUX
As you can see, we are getting NIDS data for several sites: ABS, CYS, ENX,
FTG, GRK, OKX, and PUX. These directory names are the NIDS station ID names.
cd FTG
ls
BREF1 BREF4 CREF16 LREF3 PRE3 PRET SRMV1 TOPS VEL2 VIL
BREF2 BRLR1 LREF1 PRE1 PRE3X PRETX SRMV2 VAD VEL3
BREF3 CREF LREF2 PRE1X PREA RESP STAT VEL1 VEL4
There is one directory for each of the NIDS product types: BREF1, BREF2, etc.
cd BREF1
ls
BREF1_20000318_1115 BREF1_20000318_1320 BREF1_20000318_1525
BREF1_20000318_1125 BREF1_20000318_1330 BREF1_20000318_1530
BREF1_20000318_1130 BREF1_20000318_1335 BREF1_20000318_1540
BREF1_20000318_1135 BREF1_20000318_1340 BREF1_20000318_1545
BREF1_20000318_1140 BREF1_20000318_1345 BREF1_20000318_1550
BREF1_20000318_1145 BREF1_20000318_1350 BREF1_20000318_1555
BREF1_20000318_1155 BREF1_20000318_1355 BREF1_20000318_1600
BREF1_20000318_1200 BREF1_20000318_1405 BREF1_20000318_1605
BREF1_20000318_1205 BREF1_20000318_1410 BREF1_20000318_1615
BREF1_20000318_1210 BREF1_20000318_1415 BREF1_20000318_1620
...
(these are the actual data files. They are named BREF_CCYYMMDD_HHMM
where:
CCYYMMDD_HHMM
^ ^ ^ ^ ^ ^__ minute of the hour
| | | | |____ hour of the day
| | | |_______ day of the month
| | |_________ month of the year
| |___________ year of the century
|_____________ century
BREF1 is the product type. In the way I setup the ADDE NIDS data set
(in ~mcidas/data/DSSERVE.BAT), it is also part of the data set name:
RTNIDS/BREF1 <- realtime NIDS Base Reflectivity at tilt 1 data
RTNIDS/BREF2 <- realtime NIDS Base Reflectivity at tilt 2 data
RTNIDS/BREF3 <- realtime NIDS Base Reflectivity at tilt 3 data
RTNIDS/BREF4 <- realtime NIDS Base Reflectivity at tilt 4 data
RTNIDS/LREF1 <- realtime NIDS Layer Reflectivity at level 1 data
RTNIDS/LREF2 <- realtime NIDS Layer Reflectivity at level 2 data
RTNIDS/LREF3 <- realtime NIDS Layer Reflectivity at level 3 data
RTNIDS/CREF <- realtime NIDS Composite Reflectivity data
and so on.
The NIDS.CFG file will contain entries that look like:
DIRMASK=/data/nexrad/\ID/\TYPE
FILEMASK=\TYPE_*
IDMASK=*
The "replaceable" token '\ID' represents the value of the ID= keyword
that will be passed by the IMGDISP command that will load images. The
\TYPE "replaceable" is the NIDS product type (e.g., BREF1, BREF2,
etc.). The "replaceable" concept is not part of regular expressions;
it is a concept that I added to make setting up the configuration file
easier.
A full regular expression for a NIDS data file then becomes the
concatenation of the DIRMASK and FILEMASK values:
/data/nexrad/\ID/\TYPE/\TYPE_*
A load request to the server that looks like:
IMGDISP RTNIDS/BREF1 ID=FTG
results in the regular expression being expanded to:
/data/nexrad/FTG/BREF1/BREF1_*
This will match all of the data files in /data/nexrad/FTG/BREF1 that
begin BREF1_.
Similarily, an image load that looks like:
IMGDISP RTINDS/CREF ID=PUB
will result in the regular expression:
/data/nexrad/PUB/CREF/CREF_*
This flexibility allows one to actually put all of the NIDS data files
into a single directory (like for a case study). For instance,
suppose that you had a directory named:
/var/data/sanjuan/case1
that had all NIDS products for the San Juan, Puerto Rico NEXRAD
(station ID JUA _I think_). A directory listing might look like:
cd /var/data/sanjuan/case1
ls
BREF1_20000212_1530 BREF1_20000212_1535 ...
BREF2_20000212_1530 BREF2_20000212_1535 ...
BREF3_20000212_1530 BREF3_20000212_1535 ...
BREF4_20000212_1530 BREF4_20000212_1535 ...
LREF1_20000212_1530 LREF1_20000212_1535 ...
LREF2_20000212_1530 LREF2_20000212_1535 ...
...
The DIRMASK= and FILEMASK= parameters in NIDS.CFG would look like:
DIRMASK=/var/data/sanjuan/case1/
FILEMASK=\TYPE_*
Notice that since there is only data for JUA in the directory that
I do not need to use the \ID replaceable.
Following on with the same kind of example, suppose I have all NIDS
products for FTP (Denver, CO) and PUB (Pueblo, CO) in the same
directory, /var/data/colorado. A directory listing might look like:
cd /var/data/colorado
ls
FTG-BREF1_20000212_1530 FTG-BREF1_20000212_1535 ...
FTG-BREF2_20000212_1530 FTG-BREF2_20000212_1535 ...
FTG-BREF3_20000212_1530 FTG-BREF3_20000212_1535 ...
FTG-BREF4_20000212_1530 FTG-BREF4_20000212_1535 ...
FTG-LREF1_20000212_1530 FTG-LREF1_20000212_1535 ...
FTG-LREF2_20000212_1530 FTG-LREF2_20000212_1535 ...
PUB-BREF1_20000212_1530 PUB-BREF1_20000212_1535 ...
PUB-BREF2_20000212_1530 PUB-BREF2_20000212_1535 ...
PUB-BREF3_20000212_1530 PUB-BREF3_20000212_1535 ...
PUB-BREF4_20000212_1530 PUB-BREF4_20000212_1535 ...
PUB-LREF1_20000212_1530 PUB-LREF1_20000212_1535 ...
PUB-LREF2_20000212_1530 PUB-LREF2_20000212_1535 ...
...
The DIRMASK= and FILEMASK= parameters would look like:
DIRMASK=/var/data/colorado/
FILEMASK=\ID-\TYPE_*
As you can see, this setup can be pretty flexible.
>So is McIDAS Area conversion turned off after i uncomment the original WSI
>lines from pqact.conf? I want to have both things active.
No. In order to turn off conversion to AREA files, you would have to comment
out all of the WSI lines that call for the invocation of the nids2area
decoder.
>many Thanks,
You are welcome.
By the way. I have been meaning to talk with you about your ADDE setup.
It appears that the executables 'compress' and 'uncompress' can not
be found by the ADDE routines on your system. A quick look at our
IRIX machines here at Unidata show:
uname -a
IRIX chevy 6.5 01200531 IP22
which compress uncompress
/usr/ucb/compress
/usr/ucb/uncompress
If compress and uncompress are not located in /usr/ucb on your system,
then the ADDE servers can not send/receive compressed data.
Compressed ADDE data transfers are controlled by an environment variable
named MCCOMPRESS that has to be set for each user that is going to
run McIDAS-X. If MCCOMPRESS is not defined, it is assumed to mean
that data is to be transferred in an uncompressed format. This may
be the reason that your IMGDISP RTIMAGES/GE... invocations were so
slow when your DATALOC for RTIMAGES was pointing back here to Unidata.
MCCOMPRESS must be set _before_ the McIDAS-X session is started.
Here is a sequence that will demonstrate how much transferring
compressed data helps:
<login as 'mcidas'>
unsetenv MCCOMPRESS (assumes you are using the C shell; adjust the
syntax if you are using the Bourne or Korn
shells)
cd workdata
mcidas
DATALOC ADD RTIMAGES ADDE.UNIDATA.UCAR.EDU
IMGDISP RTIMAGES/GE-IR STA=TJSU EU=IMAGE SF=YES REFRESH='EG;MAP H 1'
(try to get a feel for how long this took to complete)
EXIT
setenv MCCOMPRESS TRUE
mcidas
IMGDISP RTIMAGES/GE-IR STA=TJSU EU=IMAGE SF=YES REFRESH='EG;MAP H 1'
(compare how long this took with the uncompressed transfer)
DATALOC ADD RTIMAGES BREEZE.UPRM.EDU
IMGDISP RTIMAGES/GE-IR STA=TJSU EU=IMAGE SF=YES REFRESH='EG;MAP H 1'
(this invocation should fail until 'compress' and 'uncompress' and
can be found in the environment in which the remote ADDE server on
breeze runs. The remote ADDE server routine that looks for 'compress'
and 'uncompress' searches the following directories (in this order):
/bin
/usr/ucb
/usr/bin
/usr/bsd
If 'compress' and 'uncompress' are in different directories on your
system, then you should make a link from where they are to the /usr/ucb
directory (has to be done as 'root'). This should get things working.
I hope that this email was not too long to digest. I try to explain
things in detail since these email exchanges go into an inquiry tracking
system that all users can search and, hopefully, learn from.
Tom