[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[IDD #TTR-495389]: Slow access to local datasets through ADDE
This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
- Subject: [IDD #TTR-495389]: Slow access to local datasets through ADDE
- Date: Thu, 07 Feb 2008 16:55:54 -0700
Hi Samuel,
re:
> Do I still need to put the 'CURDAY' option into the dsserve.k command string?
> (e.g. dsserve.k ADD RTIMAGES/ANTARCTIC AREA TYPE=IMAGE
> DIRFILE=/data/ldm/mcidas/images/sat/ANTARCTIC/4km/IR/\\CURDAY/IR_*
> \"Antarctic IR Composite)
The \CURDAY "replaceable" gets replaced by the CCYYMMDD value for the current
day (UTC) in the
DIRFILE= regular expression that defines a datasets content. This replacement
occurs on the
server side when the ADDE request is received. If you want to only access data
from the current day, then
you would use \CURDAY. If you want to access data from all days, you would use
'*'. Remember,
the discussion of \CURDAY arose from my comment that you would likely want to
serve data
from the current day for the very numerous GINIEAST/GE1KVIS images. This way,
instead of having
to read and sort 'n*m' (number of days of data kept for a type of image TIMES
number of images per day)
files, the ADDE server would only have to read 'n' files. This becomes very
significant when one tries
to keep a number of days of GINIEAST/GINIWEST Visible images online since there
are 4 of each of
these each hour.
> If so, will it work this time?
It should, but... I just tried creating an ADDE dataset for the Antarctic IR
images using
the \CURDAY construct, and it didn't work. This means that the mod I made to
support \CURDAY
is not being used by the ADDE server for images in AREA format. Rats! I
apologize for this
oversight on my part!!
The \CURDAY construct does work for datasets of images of type GINI (images in
the GINIEAST,
GINIWEST, GINICOMP, and NEXRCOMP datasets) and images of type NEXR (images in
the RTNEXRAD
dataset), so I suggest that you use it in those first. I will work on the mods
needed to add
\CURDAY support to datasets of type AREA and make an addendum.
Here is an example I just ran on your machine that uses three different
"replaceables" to define
a dataset for NEXRAD Level III data:
<as 'mcidas' on your machine>
cd $MCDATA
dsserve.k ADD CURNIDS/N0R NEXR TYPE=IMAGE
DIRFILE=/data/ldm/mcidas/images/NIDS/\\ID/\\TYPE/\\TYPE_\\CURDAY_\* \"Test of
CURDAY for NIDS
The "replaceables" used are:
\ID -> NEXRAD Level III station ID (e.g., FTG, DIX, PBZ, etc.)
\TYPE -> NEXRAD Level III product type (e.g., N0R, N0S, etc.)
\CURDAY -> current data represented as CCYYMMDD
After making the CRUNIDS/N0R dataset definition on your machine, I am able to
see the just the
current day's NEXRAD Level III base reflectivity (N0R) images from the station
FTG:
-bash-2.05b$ imglist.k CURNIDS/N0R.ALL ID=FTG
Image file directory listing for:CURNIDS/N0R
Pos Satellite/ Date Time Center Band(s)
sensor Lat Lon
--- ------------- ------------ -------- ---- ---- ------------
1 RADAR 7 FEB 08038 00:06:00 FTG 1
2 RADAR 7 FEB 08038 00:16:00 FTG 1
3 RADAR 7 FEB 08038 00:25:00 FTG 1
4 RADAR 7 FEB 08038 00:35:00 FTG 1
5 RADAR 7 FEB 08038 00:45:00 FTG 1
6 RADAR 7 FEB 08038 00:55:00 FTG 1
7 RADAR 7 FEB 08038 01:04:00 FTG 1
8 RADAR 7 FEB 08038 01:14:00 FTG 1
9 RADAR 7 FEB 08038 01:24:00 FTG 1
10 RADAR 7 FEB 08038 01:33:00 FTG 1
11 RADAR 7 FEB 08038 01:43:00 FTG 1
12 RADAR 7 FEB 08038 01:53:00 FTG 1
13 RADAR 7 FEB 08038 02:22:00 FTG 1
14 RADAR 7 FEB 08038 03:30:00 FTG 1
15 RADAR 7 FEB 08038 04:39:00 FTG 1
16 RADAR 7 FEB 08038 06:37:00 FTG 1
17 RADAR 7 FEB 08038 07:25:00 FTG 1
18 RADAR 7 FEB 08038 07:35:00 FTG 1
19 RADAR 7 FEB 08038 10:50:00 FTG 1
20 RADAR 7 FEB 08038 11:48:00 FTG 1
21 RADAR 7 FEB 08038 11:58:00 FTG 1
22 RADAR 7 FEB 08038 12:08:00 FTG 1
23 RADAR 7 FEB 08038 12:18:00 FTG 1
24 RADAR 7 FEB 08038 12:27:00 FTG 1
25 RADAR 7 FEB 08038 12:37:00 FTG 1
26 RADAR 7 FEB 08038 12:47:00 FTG 1
27 RADAR 7 FEB 08038 12:57:00 FTG 1
28 RADAR 7 FEB 08038 13:06:00 FTG 1
29 RADAR 7 FEB 08038 13:16:00 FTG 1
30 RADAR 7 FEB 08038 13:26:00 FTG 1
31 RADAR 7 FEB 08038 13:36:00 FTG 1
32 RADAR 7 FEB 08038 15:34:00 FTG 1
33 RADAR 7 FEB 08038 17:50:00 FTG 1
34 RADAR 7 FEB 08038 18:00:00 FTG 1
35 RADAR 7 FEB 08038 18:10:00 FTG 1
36 RADAR 7 FEB 08038 18:19:00 FTG 1
37 RADAR 7 FEB 08038 18:29:00 FTG 1
38 RADAR 7 FEB 08038 18:39:00 FTG 1
39 RADAR 7 FEB 08038 18:49:00 FTG 1
40 RADAR 7 FEB 08038 18:58:00 FTG 1
41 RADAR 7 FEB 08038 19:08:00 FTG 1
42 RADAR 7 FEB 08038 19:18:00 FTG 1
43 RADAR 7 FEB 08038 19:28:00 FTG 1
44 RADAR 7 FEB 08038 19:37:00 FTG 1
45 RADAR 7 FEB 08038 19:47:00 FTG 1
46 RADAR 7 FEB 08038 19:57:00 FTG 1
47 RADAR 7 FEB 08038 20:07:00 FTG 1
48 RADAR 7 FEB 08038 20:16:00 FTG 1
49 RADAR 7 FEB 08038 20:26:00 FTG 1
50 RADAR 7 FEB 08038 20:36:00 FTG 1
51 RADAR 7 FEB 08038 20:46:00 FTG 1
imglist.k: done
To demonstrate that it really is accessing a subset of the files, note the
number of
days of images in the directory containing FTG N0R images:
-bash-2.05b$ ls /data/ldm/mcidas/images/NIDS/FTG/N0R
N0R_20080120_1640 N0R_20080125_0754 N0R_20080130_0121 N0R_20080204_0002
N0R_20080120_1650 N0R_20080125_0931 N0R_20080130_0131 N0R_20080204_0012
N0R_20080120_1709 N0R_20080125_1021 N0R_20080130_0141 N0R_20080204_0022
N0R_20080120_1719 N0R_20080125_1041 N0R_20080130_0150 N0R_20080204_0031
N0R_20080120_1729 N0R_20080125_1050 N0R_20080130_0200 N0R_20080204_0041
N0R_20080120_1738 N0R_20080125_1208 N0R_20080130_0210 N0R_20080204_0051
...
N0R_20080125_0350 N0R_20080130_0032 N0R_20080203_2254 N0R_20080207_2036
N0R_20080125_0410 N0R_20080130_0042 N0R_20080203_2304 N0R_20080207_2046
N0R_20080125_0518 N0R_20080130_0052 N0R_20080203_2333
N0R_20080125_0537 N0R_20080130_0102 N0R_20080203_2343
N0R_20080125_0547 N0R_20080130_0111 N0R_20080203_2352
\CURDAY, like all "replaceables" can be used anyplace in the DIRFILE= regular
expression
where part of the fully qualified pathname is the date in CCYYMMDD format.
This means
that you really don't have to organize images into daily directories... but
experience
has shown that it is useful to do so.
Again, I apologize for the \CURDAY construct not working (as I had thought that
it
did) for images of type AREA (e.g., UNIWISC images; the ones organized into the
RTIMAGES and CIMSS datasets). I will find and fix the problem and make an
addendum.
> Will scourBYnumber be able to identify and remove old directores? (e.g.
> 20080206 should be deleted in March)
No. You will need to switch to use of scourBYday. scourBYday can be found in
the same
directory as scourBYnumber on your machine: ~ldm/util. The directory specified
on the
scourBYday invocation command line will be the directory down to, but not
including
the various days. For instance, the invocation for scourBYday that keeps 5 days
of the Antarctic IR images would be:
scourBYday /data/ldm/mcidas/images/sat/ANTARCTIC/4km/IR 5
> P.S. I'm in the process of moving our NNEXRAD (NIDS) data to /usr/data (a
> different file system).
> This should free up some disk I/O.
Excellent. This should really help your I/O bottleneck as there are on the
order of 17,500
NNEXRAD (NIDS) files received every hour!
> We are still in the process of ordering a computer with more RAM, and CALU
> would like to utilize
> bandwidth to the fullest (even if it isn't always stored)
Very good.
> P.P.S. I've noticed Shu pulls down NEXRAD2 data, but it doesn't seem to be
> in the ADDE set, or on file.
> Am I missing something?
No, you are not missing anything. McIDAS has no ADDE server for NEXRAD Level 2
data. The
IDV, however, can use the NEXRAD Level 2 data directly, so you may still want
to ingest
the images and then make them available to machines where the IDV is used (NFS
mount, samba, etc.).
Cheers,
Tom
****************************************************************************
Unidata User Support UCAR Unidata Program
(303) 497-8642 P.O. Box 3000
address@hidden Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage http://www.unidata.ucar.edu
****************************************************************************
Ticket Details
===================
Ticket ID: TTR-495389
Department: Support McIDAS
Priority: Normal
Status: Closed
- Prev by Date: [IDD #TTR-495389]: Slow access to local datasets through ADDE
- Next by Date: [IDD #TTR-495389]: Slow access to local datasets through ADDE
- Previous by thread: [IDD #TTR-495389]: Slow access to local datasets through ADDE
- Next by thread: [IDD #TTR-495389]: Slow access to local datasets through ADDE
- Index(es):