20020419: data problems at NWS Training Center (cont.)

>From: "Kevin Polston" <address@hidden>
>Organization: NOAA/NWS
>Keywords: 200204152110.g3FLALa10481 LDM

Kevin,

>You know what is frustrating is how I have this "network congestion" (if
>that is what is the problem) but I didn't have this problem 2 weeks ago.

The network is a living thing.  Routes that are clear now may be
congested or even unreachable later today.

>Why is it that I can't even get a satellite image to show up but other
>data comes in?

The NOAAPORT satellite images are the largest products being sent in
the IDD.  The VIS images, for example, routinely are 12-14 MB in size
even when compressed.  After they are received and uncompressed, they
are 26 MB in size.  It is a lot easier to reliably send a small product
than a large one since the LDM breaks the products up into 16 KB chunks
and then sends each chunk to the downstream host(s).  If any of the
chunks can't be delivered, the transmission of the entire image fails.
Model data in grib format, on the other hand are relatively small since
each grib message stands alone as a single product.

>I am really struggling to understand how this has happened.

Perhaps the route to your machine during the day gets congested; perhaps
something on your own network is using up the bandwidth.  We have no
way of knowing -since we can't get to your machine!

>I even ran a "speed test" on the connection from a website and
>it showed I had pretty darn good speed.

If a single speed test indicated your continuous connectivity, this
would be a good measure.  It doesn't.

>So I am struggling to understand
>why this is happening. Even before when I did have some "network
>congestion" at least the images came in even if they were a little late.
>So that is a major source of my frustration in why nothing is coming in. 

It has to be that the congestion is bad enough to make transmitting
a single image is taking a seriously long time.  When this happens,
all products in the same datastream have to wait until the transmission
of the one either succeeds of times out.  If the wait goes over the
1-hour limit, then a RECLASS is done on the upstream machine and those
products that were waiting more than an hour are not sent.

>My comment about ftp'ing data was not meant to say I was going to do
>that (going back to doing automatic FTP's from motherlode) but just to
>say it was more reliable in the past than the ldm for me than in the
>past few days.

What may have been more reliable in the past may not be the same today.
Have you checked with your networking folks to see if they changed
something or if there is a problem with a local router?

>I also suspected you would comment about not being able
>to get into the machine here. I can understand your source of
>frustration there. I wish I knew why you can't get in. I will see if I
>can get someone to look into why you can't get in. 

Again, not being able to use the tools we have developed for supporting
LDM use handcuffs us.  A reasonable analogy would be to expect a
mechanic to fix a car without being able to see or touch it.

>In the meantime....if you can set up another site for me to connect to I
>would be willing to give that a try. I would still like to be able to go
>back to papagayo if the other site doesn't work as well. 

I have allowed your machine to request data from atm.geo.nsf.gov.  I
suggest that you edit your ~ldm/etc/ldmd.conf file and change the
request for NIMAGE to this other machine; keep your requests for the
other feeds going to papagayo.  We will then see how the new split of
feeds between different machines goes.

>Gribmaster is a program that downloads model data (grib files) from the
>OSO server. It is set to run up with cron so that it searches for new
>data every time it is run. If new data is found it retrieves that data.
>Not as effecient as ldm but I thought it might help make ldm run better
>if it wasn't trying to pull so much data.

The problem is not one that the LDM can't move a lot of data; it can.
I helped the National Met Service in Spain setup their own IDD network,
and they are moving all of their METEOSAT images around the country
with it.  And, the METEOSAT images are routinely 100 MB in size.  The
point is that if one's network is good, you can move a load of data
with the LDM, and move it in a timely manner.

>As for my comment as to
>grabbing "text data" I was referring to stuff like state forecasts, zone
>forecats, SPC discussions, etc (text data as opposed to model data or
>images).  

The textual data (the DDPLUS and IDS feeds) is the absolute smallest
volume of data being moved by the IDD.  If you were to limit your data
requests down to just DDPLUS and IDS and you still couldn't get
reliable delivery, I would say that either the network path to your
host is totally clogged, or you have some sort of problem with your LDM
setup.  We could check on the latter case if we could get to your
machine.

>Talking about VDUC now.....when I was at SELS they did some of their own
>work to it but it was very nice. The roam and zoom was better than what
>is available in nawips....I really liked the F keys where you could
>switch to another frame and the ability to toggle on or off colors or
>fields.

These are user interface related comments.  We do not provide the Fkey
menus developed by other groups (like VDUC), and the GUI interface I
have been working on is far from delivering complete functionality in a
point and click manner.

>Just so many things I liked about it moreso than the gempak
>stuff. I am going to try within the next couple of months to acquire a
>machine which I could load McIdas on to. Please explain a little more
>how this "pointing at a site to look at data" works. That is very
>intriguing!

The concept behind McIDAS ADDE is that data access is done through
datasets.  The data that the dataset is composed of are normal files,
but the beauty of ADDE is that the dataset does not have to reside on
your local machine.  Pointing at a dataset is the process of running a
McIDAS command that makes and entry in a table that tells McIDAS
routines to request the data from a particular host when the user tries
to access that data.  The thing we have been working on is the
establishment of a set of community servers that are open to all
community members.  If one server is not reachable, or access to it is
slow, the end user can simply bring up a GUI that allows him/her to
point at a different server for the same data.  The headache of
maintaining a local store of data files is then centralized to those
sites that can handle it (network bandwidth, disk storage, system
administration, etc.).  I use McIDAS ADDE every day from home over a
_terrible_ 24 Kbps dial-up connection.  Loading satellite imagery is
painful, but it works and is reliable.  Doing simple things like plots
and contours of data is reasonably fast.

>Are you saying I don't have to download any data at all for
>McIdas to work? I would like to understand how that works. 

Yes, I am saying just that.  I have absolutely no stores of data at my
house.  All of it resides on servers spread around the country/world.
Yes, that's right _world_.  Through a special arrangement, we got
office access to GMS and METEOSAT-5 data from the Australian Bureau of
Meteorology.  I can look at their site's holdings just as easily (but
not as fast) as I can look at the stores of data on a machine in my own
office.  On top of that, tests have shown that access to data through
ADDE is faster than reading the same data on an NFS mounted disk in
most cases.  The reason for this is that the "pipe" through which data
is read by ADDE is larger than the pipe provided by NFS.

Now, ADDE can also be used to copy the data from a remote server to
your local machine.  This way a site can put together case studies
of interesting data on the fly.  After downloading the data, one
would need to organize it into a locally served dataset (not hard)
and then it could be accessed quickly from then on.

On top of all of that, since you are an NWS (NOAA) employee, you would
be allowed access to the data available by ADDE on the SATEPS (I
believe SATEPS is the evolution of VDUC, but I am not sure) machines
back in DC.  They have literally all of the satellite data that there
is, including METEOSAT, INSAT, GMS, all of the US polar orbiters, GOES,
DMSP, etc.  They also have all of the data delivered in NOAAPORT
available in McIDAS ADDE accessible formats.

>Well Tom, I will talk to you tomorrow. I will see how things look in the
>morning and then maybe try the new site if you can get one set up.

The atm.geo.nsf.gov site is setup and ready to accept your feed
request.  Let me know if you have any problems switching your NIMAGE
feed to it.

Tom