20020916: UPRM as an IDD relay node for the Caribbean and Central America (cont.)

>From:  Luis Munoz <address@hidden>
>Organization:  UPRM
>Keywords:  200209170019.g8H0Jc113973 IDD ADDE MeteoForum

Hi Luis,

re: update to LDM 5.2 to enable reporting of realtime stats
>Thank you for the update.

I will be wanting to update UPRM to 5.2.1 in the coming days.  5.2.1 has
a performance enhancement that is needed by heavily loaded LDM servers
(not yours), so it is not absolutely necessary, but I want to keep certain
sites as up to date as possible.

re: realtime stats graphs
>This looks great!

Since I upgraded atmos to 5.2, I have been looking in on your ingestion
via these graphs on a regular basis.  I note that virtually all of the
problems I have seen have been related to network issues at FSU.

re: DNS server(s) do not have reverse name lookup for atmos
    % nslookup 136.145.165.204
    ** server can't find 204.165.145.136.in-addr.arpa: NXDOMAIN
>Ok.. weird that it wasn't there.  Don't know why.

>The person who set the campus DNS servers never configured reverse DNS.
>I'm not even sure that the person in charge now will be able to do that.
>What some departments have done (for example, the ECE department) is set
>up their own nameservers and because of the poor centralized management.
>This is something I tried to address a couple of times while I knew the
>person in charge, no luck at all.

This is bad news.  Without reverse name lookup, IDD site administrators
for any machine that is to feed atmos.uprm.edu will have to modify their
/etc/hosts file AND possibly modify /etc/nsswitch.conf.  This makes
the IDD setup more brittle as things will break if/when the IP address
or name of atmos.uprm.edu changes in the future.  Anything you can do
to get reverse name lookup working would be good.

re: I setup NTP on atmos
>Yes.  I had downloaded the NTP daemon, but I never got around to
>configuring/running it.

No problem.  I added the ntpdate entry to 'root's crontab file so that
the stats graphs would look correct, and so that there would be no
problem with it feeding from upstream hosts in the IDD.

re: I split the feed requests from pluto into three requests:
>>
>>    # 20020906 - UPC modification: split feeds from FSU
>>    request HDS             ".*"    pluto.met.fsu.edu
>>    request DDPLUS|IDS      ".*"    pluto1.met.fsu.edu
>>    request MCIDAS|NNEXRAD  ".*"    pluto2.met.fsu.edu

>I think I'm having problems with one or more of these feeds, this is the
>main purpose of this message.  I seem to be missing the nexrad and 1km
>visible feeds.

There are two different things at work here:

1) the NEXRAD feed _is_ through the IDD, so we need to find out if there
   is (still) a problem of some sort at your IDD feed site, FSU

2) the 1-km VIS imagery you have is being grabbed through McIDAS ADDE,
   not through the IDD

Given that the two products are coming to you in different ways, we
need to do different things to troubleshoot:

o for the NEXRAD feed, we need to contact the IDD site administrator
  at FSU

o for the 1 km VIS imagery, we need to look at the McIDAS client routing
  table setup (the table that says who to load the 1 km VIS imagery from).

>At first I thought maybe FSU was not delivering the
>information because either their server was down or some other network
>related problem  .I got to this conclution based on some error messages in
>the ldm log.  It tried to connect to FSU and the requests were timing out.
>In fact, they still are, so I don't know what the problem is.

I just logged onto atmos as 'ldm' and did pings and ldmpings to
pluto.met.fsu.edu:

ping pluto.met.fsu.edu
PING pluto.met.fsu.edu (128.186.98.4): 56 octets data
64 octets from 128.186.98.4: icmp_seq=0 ttl=56 time=59.3 ms
64 octets from 128.186.98.4: icmp_seq=1 ttl=56 time=57.4 ms
 ...

ldmping pluto.met.fsu.edu
Sep 17 16:44:34      State    Elapsed Port   Remote_Host           rpc_stat
Sep 17 16:44:34 RESPONDING   0.120233  388   pluto.met.fsu.edu  
Sep 17 16:44:59 RESPONDING   0.058993  388   pluto.met.fsu.edu  
Sep 17 16:45:25 RESPONDING   0.058980  388   pluto.met.fsu.edu  

This says that you can contact the machine and the LDM server on the
machine.

Next, I looked in the /etc/ldmd.conf file on atmos and see that the
request for the MCIDAS and NEXRAD feed are through pluto2:

request MCIDAS|NNEXRAD  ".*"    pluto2.met.fsu.edu

Pings and ldmpings to this alias for pluto also work.

Next, I did a notifyme to pluto2 for the NEXRAD feed and it works:

ldm@atmos:~/etc$ notifyme -vxl- -f NEXRAD -o 1800 -h pluto2.met.fsu.edu
Sep 17 16:47:08 notifyme[26106]: Starting Up: pluto2.met.fsu.edu: 
20020917161708.004 TS_ENDT {{NNEXRAD,  ".*"}}
        NOTIFYME(pluto2.met.fsu.edu) returns OK
Sep 17 16:47:08 notifyme[26106]: NOTIFYME(pluto2.met.fsu.edu): OK
Sep 17 16:47:08 notifyme[26106]: e7241feee263a2c636b6101f9d4e858e    10621 
20020917161736.929 NNEXRAD 258  SDUS54 KBMX 171617 /pN0RBMX
Sep 17 16:47:08 notifyme[26106]: 2cfd7a9f2605c11b7a63b1f34f58871c     1482 
20020917161737.008 NNEXRAD 261  SDUS33 KLOT 171614 /pN3RLOT
Sep 17 16:47:08 notifyme[26106]: c9a782fdbcd98b781df5c896c1be29af    12956 
20020917161737.405 NNEXRAD 262  SDUS72 KTAE 171614 /pN0ZEOX
Sep 17 16:47:08 notifyme[26106]: 8ca06aacc74d63d9f28c7a9fafb4ce48     2475 20020

In addition, I looked at the realtime stats page for the MCIDAS (UNIWISC)
feed on atmos, and see that you are apparently receiving imagery the
Unidata-Wisconsin imagery.

So, the question now is why you are not getting any NEXRAD data?

I checked to make sure that JUA is operational, and I see that it is.
Since FSU's ldm setup has been flakey for the past several days, I decided
to switch the feed for MCIDAS (which is the same as UNIWISC) and NEXRAD
to atm.geo.nsf.gov.  I waited a little while and see that you are now
getting NEXRAD products from JUA.

>The problem we've been having could probably be with FSU directly.  I
>think I had a problem like this once.  But the other time it used to be
>request denials when ldm issued the feedme command, this time it just
>times out.

I think that the problem is at FSU.

Next, we need to figure out why the loads of the 1 km VIS images is failing.
I logged on as 'mcidas' and see that you are running a series of scripts
out of cron, and the McIDAS invocations out of these scripts use the client
routing information specified in the 'mcidas' account.  I then CDed to the
~mcidas/workdata directory (as 'mcidas') and took a look at where you
are trying to load the 1 km VIS imagery:

cd ~mcidas/workdata
dataloc.k LIST

Group Name                    Server IP Address
--------------------         ----------------------------------------
CIMSS                        ATMOS.UPRM.EDU
GINICOMP                     SNOW.PLYMOUTH.EDU
GINIEAST                     CACIMBO.GGY.UGA.EDU
GINIWEST                     PAPAGAYO.UNL.EDU
RTGRIDS                      PAPAGAYO.UNL.EDU
RTIMAGES                     ATMOS.UPRM.EDU
RTNEXRAD                     ATMOS.UPRM.EDU
RTPTSRC                      ATMOS.UPRM.EDU
RTWXTEXT                     ATMOS.UPRM.EDU
TOPO                         ATMOS.UPRM.EDU
UPRM                         <LOCAL-DATA>

<LOCAL-DATA> indicates that data will be accessed from the local data directory.
DATALOC -- done

Since you are trying to get the 1 km VIS images from cacimbo.ggy.uga.edu,
I tried pinging it:

ping cacimbo.ggy.uga.edu

So, cacimbo is alive.  The next thing I tried to do was see if I could
contact the ADDE server on cacimbo:

dsinfo.k I GINIEAST

This just hung and eventually timed out.  So, it looks like the ADDE server
on cacimbo is not working for some reason.  Given that I switched you over
to using pscwx.plymouth.edu:

dataloc.k ADD GINIEAST pscwx.plymouth.edu

Group Name                    Server IP Address
--------------------         ----------------------------------------
GINIEAST                     PSCWX.PLYMOUTH.EDU

<LOCAL-DATA> indicates that data will be accessed from the local data directory.
DATALOC -- done

Now, when I do the DSINFO command, I get back results:

dsinfo.k I GINIEAST                      

        Dataset Names of Type: IMAGE in Group: GINIEAST

Name         NumPos   Content
------------ ------   --------------------------------------
GE1KVIS       9999    GINI 1 km VIS East CONUS
GE4K12        9999    GINI 4 km 12.0 um East CONUS
GE4K39        9999    GINI 4 km 3.9 um East CONUS
GE4KIR        9999    GINI 4 km 10.7 um East CONUS
GE8KWV        9999    GINI 8 km WV East CONUS
GNC24K12      9999    GINI 24 km 12.0 um Nhem-Composite
GNC24K39      9999    GINI 24 km 3.9 um Nhem-Composite
GNC24KIR      9999    GINI 24 km 10.7 um Nhem-Composite
GNC24KVIS     9999    GINI 24 km VIS Nhem-Composite
GNC24KWV      9999    GINI 24 km WV Nhem-Composite
GPN8KIR       9999    GINI 8 km 10.7 um Puerto Rico National
GPN8KVIS      9999    GINI 8 km VIS Puerto Rico National
GPN8KWV       9999    GINI 8 km WV Puerto Rico National
GPR1KVIS      9999    GINI 1 km VIS Puerto Rico Regional
GPR4K12       9999    GINI 4 km 12.0 um Puerto Rico Regional
GPR4K39       9999    GINI 4 km 3.9 um Puerto Rico Regional
GPR4KIR       9999    GINI 4 km 10.7 um Puerto Rico Regional
GPR8KWV       9999    GINI 8 km WV Puerto Rico Regional
GSN8K12       9999    GINI 8 km 12.0 um Super-National
GSN8K39       9999    GINI 8 km 3.9 um Super-National
GSN8KCTP      9999    GINI 8 km Cloud Top Pressure Product
GSN8KIR       9999    GINI 8 km 10.7 um Super-National
GSN8KLI       9999    GINI 8 km Lifted Index Product
GSN8KPW       9999    GINI 8 km Precipitable Water Product
GSN8KSFCT     9999    GINI 8 km Surface Temperature Product
GSN8KVIS      9999    GINI 8 km VIS Super-National
GSN8KWV       9999    GINI 8 km WV Super-National

DSINFO -- done

So, your script for loading the GINIEAST/GE1KPR images should start working
again.  I will have to check with UGA to find out what is going on with
cacimbo.

re: I am planning on running tests on feeding one or two sites in Brazil from
    atmos

>Understood.  Again, there's no problems with that. I hope you're able to
>relay the data with no problems.  We're glad we can help.

Wonderful!

Let's keep an eye on the realtime stats page for NNEXRAD for atmos and
see if more problems occur.

Tom
--
+-----------------------------------------------------------------------------+
* Tom Yoksas                                             UCAR Unidata Program *
* (303) 497-8642 (last resort)                                  P.O. Box 3000 *
* address@hidden                                   Boulder, CO 80307 *
* Unidata WWW Service                             http://www.unidata.ucar.edu/*
+>-----------------------------------------------------------------------------+