20020419: data problems at NWS Training Center (cont.)

>From: "Kevin Polston" <address@hidden>
>Organization: NOAA/NWS
>Keywords: 200204152110.g3FLALa10481 LDM

Kevin,

>I do not want to sound bitter but my problems getting data are really
>irritating. Not only is the data not timely, I am hardly getting any
>data!  For instance when I wrote this morning, everything was running
>smoothly and on time. However, this afternoon the visible satellite
>imagery was sporadic...getting images about once every half hour (not
>good when RISOP is going on) and as for my IR satellite images.....I had
>nothing from 17Z until 2025Z and as I write this it is 2345Z. That is
>horrible. Can network congestion be that bad?

Yes, network congestion can be that bad.

>I was better off just ftp'ing the data.

Perhaps, but that was causing us significant problems, and we can not
allow it to happen again.  We have over 150 sites that we have to worry
about...

>I have significantly cut back on acquiring any of the
>radar data (I just have 3 or 4 sites now as opposed to the many).  I cut
>back more than a month ago to just getting the EAST and WEST CONUS
>images instead of the all possible images. I thought that would help. 

If the problem is network congestion, then requesting less data should
help.  I jumped onto papagayo and see the connects/disconnects from
your machine throughout the log file.  Right at the moment, the
connection from papagayo to mkc-65-30-96-123.kc.rr.com seems reasonable
(from a traceroute from papagayo to mkc-65-30-96-123.kc.rr.com).  What
I can do is setup another machine to feed you data.  Given what you
want to receive, there are not many options.

One thing that has been a major hindrance to our helping you get better
data reception via the LDM is the inability to contact your machine
using any of the standard tools that we use to troubleshoot: ping,
ldmping, notifyme, and even remote logins to check out your setup.
Since your machine is hidden behind a firewall or through some other
access restriction mechanism (like TCP wrappers), we are essentially
handcuffed in what we can discern as to your problems.  Sending
configuration files back and forth helps up to a point, but when other
sites have problems as bad as yours, we typically are granted login
access to the 'ldm' account so we can verify that things are setup
correctly and then run network tests both from feed hosts and the local
machine.

I think that it is in your best interest to get someone there to help
you to allow us access to your machine.

>What is frustrating in addition to the lack of timeliness is the fact
>there is less data coming in now than I used to get and I have cut back
>quite a bit on what I am requesting!

I understand your frustration.  I hope you can understand mine given
that normal avenues of support have been cut off by the inability to
get to your machine.

>I don't mean to take this out on
>you since you have been a big help but I am really
>frustrated...especially with severe weather going on just west of us.
>And the thing that makes it really bad is that ldm used to be running
>just fine and I had very little complaints. 

Is the LDM not running again?  Or, is it running and the problem one of
not getting the data you want?

>I tell you what, the only thing I can think of that is different is I am
>using gribmaster to grab the ETA, AVN and MRF instead of ldm.

I don't know what gribmaster is.

>I did this
>because I thought it would help alleviate the dataflow problem with ldm.

If the problem is network bandwidth to your machine, then any use of
the network will worsen the problem.

>I wonder if that is interfering with getting the satellite data and the
>timeliness. I don't know. I will change it back and see what happens. 
>Grabbing text data should not be too big of a deal right (as far as
>slowing down the transfer of data).  If it is I can cut back on text
>data but I don't think I am getting very big files.

I don't know what you are really telling me when you comment on
"grabbing text data".  Did you turn back on your FTPs to motherlode?

>Heck, I feel maybe I might just shut everything down and try one image
>at a time to see if it can handle that. I've attahced a copy of my
>latest log file.  I see things in there that say "disconnected",
>"connection reset by peer", "skipped", "RECLASS", and "never
>completed".  None of these sound very good and aside from the obvious
>answers ...what do they mean?

All of these messages are telling you that the upstream LDM is unable
to feed the products you are requesting.  When the products on the
upstream site get to be 1 hour old and they still havn't been sent to
your machine.  your request is RECLASSed to a more current time.  This
means that numbers of products won't even try to be sent.

>Thanks for letting me vent....I know you've been a big help to me.  I'm
>just irritated because there is severe weather just west of here and
>nothing is running very well.

I really do understand your frustration.  I hope you understand mine.

>Does McIdas have this kind of problem?

If the network connection is marginal, nothing will be able to get data
in or out.  If the problem is that there is some sort of setup on your
end that is interfering with the LDM transfers, then there is nothing
we can do from here.  I say this since:

o other sites feeding from papagayo are not having problems
o you seem to be able to use FTP to get files when the LDM transfers
  fail

>Maybe I should install McIdas!  :-)  I was quite familiar with
>VDUC....how similiar is McIdas to the capabilities of VDUC?

I never knew what VDUC added to McIDAS, if anything.  As far as I can
tell, the Unidata version of McIDAS contains most everything
interesting added by others.  What McIDAS has never had is a nice GUI
interface to its capabilities.

Over the past several years, McIDAS has evolved into a system that
allows one to access datasets remotely.  Your system does not even have
to have the data locally.  You simply "point" at a remote host and use
the data from it.  A couple of sites that didn't have the capability to
use the LDM (no system administration support, etc.) switched to using
McIDAS ADDE and I hardly ever hear from them anymore.

>I'm just
>starting looking at trying to get another machine and then maybe trying
>to get McIdas set up. I know that is your specialty. What kind of
>machine would you recommend (best possible hardware and then I can work
>down if I have to. Stuff like memory, hard drive space, processor speed,
>video card, etc). 

Like anything, the more the better.  Given the price of PCs, I would
recommend one with a fast processor (like 1 Ghz or faster); lots of
memory (like 512 MB or more (1 GB is better :-)); a decent, fast hard
disk (at least a 40 GB disk, preferably SCSI); a large color monitor
(19" for aging eyes); keyboard, mouse, etc.  I use RedHat 7.1 Linux at
home, and like it well enough.  I have been exposed to FreeBSD on one
user's machine and find it to be much faster and more secure than
Linux.  In short, a reasonably well configured PC running Linux,
Solaris x86, or FreeBSD is all you need.  This is addition to a decent
network connection.

Like I said above, it is in your best interest to find out why we can't
reach your machine remotely.  Who knows, the same thing that is keeping
us out may be the thing that is causing the LDM to not work well.

Tom