20011203: realtime FTPs to motherlode.ucar.edu (cont.)

>From: "Kevin Polston" <address@hidden>
>Organization: NWSTC
>Keywords: 200111302138.fAULcON07166 motherlode IDD FTP

Kevin,

>I was unaware that there was such a tremendous log file being
>generated. I can cut back on several products I was
>receiving....

The problem is that every FTP that is initiated from your side
generates a log message.  It appears that you are using LOTS of FTPs to
see if new products are available: if yes, then the product is
transferred; if no, the FTP exits.  This is very inefficient at best.

>however the products I will continue to need would be the
>couple of satellite images (vis and IR) and profiler data.

From our perspective, it would be much better if you ran your own LDM
and ingested the data you want that way.  We can hook you up with a
university feed site (I think that the University of Nebraska-Lincoln
would be the best) that can send you the data you need, and you can
then run your own decoders, and do what you want with the data.

>Basically I
>am archiving the data along with other data sources I have that are not
>from "motherlode" such as model, text and radar.  I am trying to save
>as complete datasets as possible.  This information is then incoporated
>into classroom work and online courses that I use in teaching about
>different types of weather events. I have had this set up for several
>months now and didn't realize there was a problem.

The sheer volume of log information simply got to be a problem.  The
log files were growing so fast that they have to be rotated daily
instead of weekly.  Also, since motherlode is not owned by Unidata
exclusively, others decided to dump non-current log files.  This means
that we can not use logging information like we would like.

>I can cut back on
>several things I am receiving but I still would need to receive the
>satellite and profiler data...which is critical to everything I am
>doing.

We understand your desire to continue to get various types of data from
our server, but using FTP to do this is both inefficient and contrary
to what you were told were acceptable uses of our services when you
were given access back in April:

  From: Steve Chiswell <address@hidden>
  Date: Mon, 16 Apr 2001 11:12:44 -0600
  To: Kevin Polston <address@hidden>

  ...

  The idea of the files on motherlode is to provide a
  backup source for our IDD and ADDE service rather than realtime access.

  ...

>I apologize for putting a strain on your system....I was unaware
>of that.

I hope that you are willing to install and run an LDM on your system.
This way you can get the data you want as soon as it is available
without polling by FTP.  I can get the ball rolling on this as soon
as you agree.

Tom Yoksas