20020221: realtime FTPs to motherlode.ucar.edu (cont.)

>From: "Kevin Polston" <address@hidden>
>Organization: NOAA
>Keywords: 200202012111.g11LBCx14907 LDM installation

Kevin,

>Hey, another good news/bad news e-mail.....AGAIN!  :-)

OK, here goes...

>The good news is ...I grabbed what I needed to get and I got the
>satellite data to start coming in!  :-)  And it was looking timely
>too.  However.....it is now NOT coming in and I wonder if it is being
>sent out. The reason I say that is I had the notifyme -vl - -h
>"papagayo.unl.edu" command running and was watching all the data
>products come through. I have not seen a satellite product for some
>time. As I said before.....it was working.  I don't think it is
>anything I've done.

The machine injecting the data into the IDD was having problems yesterday,
so the problems you were seeing were most likely related to that.  Given
that, and given that you are now an official LDMer, you should signup
for the ldm-users email list (<address@hidden>) so you
will be kept up to date on datafeeds.  You can subscribe/unsubscribe
to any Unidata-maintained email list online at:

http://www.unidata.ucar.edu/mailinglist/mailing-list-form.html

You can subscribe to any email list you are subscribed to, but you
should wait for confirmation of your subscription before posting.

>When I checked the log file the only thing I am
>getting that suggests something is not quite right (but I don't know if
>it is related to satellite imagery) is this line:

>localhost uni0[1002]: FEEDME(uni0.unidata.ucar.edu):
>h_clnt_create(uni0.unidata.ucar.edu):
>Timed out while creating connection

You should not be setup to request anything from uni0.unidata.ucar.edu
(there is no such machine at the moment).  

There is (was?) a commented-out entry in your ldmd.conf file that
referenced uni0:

#request WMO ".*" uni0.unidata.ucar.edu

but commented out entries should never be seen.  Please make sure
that this entry was not uncommented along the way.  If it was,
then you need to comment it out (or, better yet, delete it) and
then stop and restart the LDM:

ldmadmin stop
<wait until all LDM processes like rpc.ldmd) exit>
ldmadmin start

>It seems as if it is a repeating line when I check the log file. So I
>don't know what I did but I am also wondering if it is the upstream
>site is not sending data. I just don't know.

Let's check your current copy of ldmd.conf before scratching our heads
too hard.

>Some more good news is the radr imagery is now very timely!   :-)  It
>seems whatever the problem was was fixed and now it appears all the
>radar data is in good shape.

Very good.

>Perhaps the satellite problem was linked
>to getting the radr data back on time?

Nope.

>I'm just guessing here though.
>The only thing I would like to see better is the surface obs more
>timely. By more timely I mean this.....I have a loop running which has
>surface obs overlaid on a radar plot. Well, the radar image almost
>always is near the top of the next hour before this past hours data
>appears on the image. The radar image is the dominant source as well.
>So I don't know why the obs appear late.

The surface obs should not be late.  If they continue to be late, we
need to work on splitting the feeds on your machine so they don't
get delayed by the transmission of the NIMAGE images, NEXRAD images,
or HRS data.

>So, to sum up....the satellite data was working.  :-)  Now I don't know
>if the images are being sent or not. If they are then I must have
>messed something up here. I did stop my ftp's to motherlode based on my
>success.....however....I hope I wasn't too premature since I haven't
>seen any satellite data since. The radar data is now back on time. That
>is good. Now if I could just get the surface obs to show up a little
>more timely then I would think most of the problems are solved
>
>***UPDATE**** - ok Tom, just as I am finishing writing this I see the
>satellite data has started coming in again.  :-)  It appears it is
>about an hour behind the current time but as I see the products
>scrolling by it looks as if it is getting back on time.  Ok...we'll see
>how it goes today. Hope you have a good time skiing! If there are any
>problems I'll e-mail you later this evening.

The hour lateness of the images is a reflection of the data flow catching
back up to realtime after a reboot/reconfiguration of the top level
relay for that data.

>>From address@hidden Thu Feb 21 19:34:49 2002
>
>Here is the situation as of this evening.....things are running pretty
>good. When I first checked it appeared the radar data had started to lag
>behind some again but just before I started this e-mail I checked again
>and it seems to be back on time. So do the surface obs...they appear to
>be timely now. So that is good.  

Very good.

>Of course, what would it be like if I sent an e-mail and didn't have
>some sort of problem.  :-)    Here is this evening's problem. I'm sure
>(well, at least I'm hoping) there is a way to work around this. Last
>night I thought I might have to make new satellite directories in order
>to match the data. Well, I had noticed that the directories that were
>created for me were this (in my pqact.conf file I set up the directory
>structure to look like this    /usr1/nawips/metdat/images/sat):
>
>/usr1/nawips/metdat/images/sat/_20020221/EAST-CONUS/4km/IR
>(substitute other regions in place of EAST-CONUS).
>
>My directory structure that I had before ldm was this:
>
>/usr1/nawips/metdat/images/sat/GOES-8/4km/IR

This is telling us that your pqact.conf entry is not quite what you
want.

>The reason I thought I needed to change my directory structure was so
>NMAP would be able to display it. However....I soon found it that
>apparently it would display just fine under the new directory
>structure.  The problem I have is this.....this evening I "stopped"
>getting satellite data. In actuality, the data was still coming in but
>was now under the _20020222 directory that was created. In addition,
>the permissions were once again set only for user:LDM so I had to
>change the permissions.

What is your 'umask' setting for the user running the LDM?  This would
be set in the ~ldm/.cshrc file, and should look like:

umask 002

>Once I did that everything is once again in
>good shape.  So......how do I get it set up so that I don't have to do
>this every day.

Make sure that 'umask' is set correctly in ~ldm/.cshrc.  If you need
to make a change to 'umask' in ~ldm/.cshrc, you need to make that
change active and then stop and restart the LDM:

<edit .cshrc>
source .cshrc
ldmadmin stop
<as always, wait until the LDM processes exit>
ldmadmin start

>Won't this action just keep creating more and more
>directories?

Your action may be setup to create a new directory for each day.

>I have seen on motherlode where you have a symbolic link
>set up to point to the "current" directory.

The setup on motherlode was done for a variety of reasons that are not
what a typical user would want.  Please don't try to use its directory
structure to model yours.

>Well, I don't mean to sound
>like a bonehead here but I am confused as to how to set that up here
>for this situation.  I mean I know how to create a symbolic link but I
>am at a loss as to how to set up the symbolic link so that the new
>directory which is going to be created every day will actually point to
>the current directory.

We do this on motherlode using a cron job.

>Am I making this too hard?

Yes.

>That seems to be the
>biggest thing I have right now that is a problem.

We need to adjust your pqact.conf action for filing the images
to match the setup you need to run NMAP2.  We also need to worry about
scouring those images so that your disk does not fill up.

>I also have one other issue which has just come up. It appears I am
>going to be moving my machine by next week (Thursday or Friday).  I
>will most likely have a different IP address. When I find out what it
>is can we use that in place of the one I am using now for papagayo?

Yes.

>It
>seems to me it would be pretty easy...just change the permission from
>the IP address I have now to the new one. I hope thats not a problem.

That is all that is needed.  This, of course, needs to be done on papagayo.
The other thing that _must_ work is the forward and reverse name lookup
for the machine/ip.  I believe we talked about this earlier, but here
it is again:

The machine feeding your machine must be able to do:

nslookup your_machine_name
nslookup your_machine_IP

The two answers must match, or the LDM will refuse to feed you.  So,
you should make sure that when the IP of your machine gets changed
your network folks add the new IP/name to the DNS server there.
There may be a failure to feed until your machine's new name/IP
information gets propogated to DNS servers on the net.


>I am going to work on this satellite directory structure for awhile. If
>I have anything else I will e-mail you. Hope you had a good time
>skiing.

Skiing was fantastic.  There was at least 8 inches of new snow with up
to a foot in some parts of the park.

Here is your current GINI decode/file action:

# NOAAPORT GINI Images
NIMAGE  ^sat/ch[0-9]/.*/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        PIPE    -close
        util/readpng -n -l logs/png.log 
/usr1/nawips/metdat/images/sat/_\2\3\4\5/\8/\9/\1/\1_\2\3\4\5_\6\7

The headers for the NOAAPORT GINI images looks like (small sampling):

sat/ch2/GOES-10/3.9/20020222 2200/NHEM-COMP/24km/ TIGF04 KNES 222200
sat/ch2/GOES-10/WV/20020222 2200/NHEM-COMP/24km/ TIGF05 KNES 222200
sat/ch2/GOES-10/IR/20020222 2200/NHEM-COMP/24km/ TIGF02 KNES 222200
sat/ch2/GOES-10/12.0/20020222 2200/NHEM-COMP/24km/ TIGF03 KNES 222200
sat/ch2/GOES-10/VIS/20020222 2200/SUPER-NATIONAL/8km/ TIGN01 KNES 222200
sat/ch2/GOES-10/VIS/20020222 2200/NHEM-COMP/24km/ TIGF01 KNES 222200

Your action for the first product in the list should file the image in

/usr1/nawips/metdat/images/sat/_20020222/NHEM-COMP/24km/3.9/3.9_20020222_0222

This looks like the naming you sent in above:

/usr1/nawips/metdat/images/sat/_20020221/EAST-CONUS/4km/IR

Now, the structure you seem to want is:

/usr1/nawips/metdat/images/sat/GOES-8/4km/IR/IR_YYMMDD_HHMM

So your pqact.conf action should be changed to:

# NOAAPORT GINI Images
NIMAGE  ^sat/ch[0-9]/(.*)/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/.*/(.*km)/
        PIPE    -close
        util/readpng -n -l logs/png.log 
/usr1/nawips/metdat/images/sat/\1/\9/\2/\2_\3\4\5\6_\7\8

The changes made were:

o capture the satellite as \1
o remove the capture of the type (e.g., NHEM-COMP, etc.) which was previously
  \8

The reason the second thing had to be done was because the '\' substitutions
only work for \1 - \9.

Tom