[LDM #AFT-567406]: GOES-17 data filing question

Hi Mike,

re:
> I have a little puzzler here, hopefully I'm missing something obvious.
> I've been receiving and filing the GOES-17 ABI/L1 data fine for a few
> months.  Within the last few days I started to have trouble with the
> FullDisk imagery filing for some reason, so I'm trying to troubleshoot.
> The CONUS and Mesoscales are coming in and filing fine.

Did you change how you are REQUESTing the SATELLITE feed products in the
last few days?

re:
> Here are some basics on the setup and troubleshooting, using CONUS and
> FullDisk for comparison.  Again the CONUS is coming in fine and the
> FullDisk is not.  The FullDisk is coming in very sporadically, like maybe a
> couple images a day per channel.
> 
> On machine vulcan.science.sjsu.edu:
> 
> ldmd.conf:
> request Satellite       "ABI-L1b-RadC"  idd.unidata.ucar.edu
> request Satellite       "ABI-L1b-RadF"  idd.unidata.ucar.edu

First comment:

If this really is the REQUEST line you are using, it shouldn't work
since the name of the feed is SATELLITE not Satellite.  To check this,
please send the output of:

<as 'ldm' on vulcan'
cd ~ldm
grep -i ^request etc/ldmd.conf

re:
> PQACT:
> SATELLITE
> /data/cspp-geo/.*/(OR_(ABI)-L1b-RadC-M.C(..)_(G..)_s(....)(...).*)
> PIPE    -close
> /usr/local/ldm/util/grbfile.sh
> /usr/local/ldm/var/data/images/sat2/\4/\2/CONUS \1 \2 \3 \5 \6 576
> /usr/local/ldm/var/logs/grb\4.log
> #
> SATELLITE
> /data/cspp-geo/.*/(OR_(ABI)-L1b-RadF-M.C(..)_(G..)_s(....)(...).*)
> PIPE    -close
> /usr/local/ldm/util/grbfile.sh
> /usr/local/ldm/var/data/images/sat2/\4/\2/FullDisk \1 \2 \3 \5 \6 192
> /usr/local/ldm/var/logs/grb\4.log

A quick look at the volume of SATELLITE data that vulcan is reporting
as being received shows that it is not getting everything wanted:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?SATELLITE+vulcan.science.sjsu.edu

re:
> NOTIFYME:
> This entry uses my REGEX and shows plenty of data available:  notifyme -vl-
> -h idd.unidata.ucar.edu -o 1800 -p
> "/data/cspp-geo/.*/(OR_(ABI)-L1b-RadF-M.C(..)_(G..)_s(....)(...).*)"

We know that all of the real-server backends of our idd.unidata.ucar.edu
cluster are getting all of the SATELLITE data.  This is most easily seen
by looking at the volume of SATELLITE from one or more of those real
server backend machines.  For example:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?SATELLITE+node2.unidata.ucar.edu

Even though you are not REQUESTing all of the SATELLITE feed, the volume
and peaks/valleys in volume is _heavily_ dominated by the FullDisk images
especially Channel 02 (0.64 um) images.

re:
> And the CONUS entry shows the same:
> notifyme -vl- -h idd.unidata.ucar.edu -o 1800 -p
> "/data/cspp-geo/.*/(OR_(ABI)-L1b-RadC-M.C(..)_(G..)_s(....)(...).*)"
> 
> But the CONUS imagery is coming in and filing fine, while the
> Fulldisk "RadF" is not.

Hmm...

re:
> My logs don't seem to show anything unusual, but this is where I need
> help.  What can I check next to determine the root cause of this issue?
> 
> Hmm...it looks like I see some latency issues on mainly Conduit and Ngrid,
> but not much on Satellite:
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?vulcan.science.sjsu.edu

I do see latency issues on vulcan for the SATELLITE feed:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?SATELLITE+vulcan.science.sjsu.edu

especially from 0-14 UTC on the 29th.

re:
> But not sure if that is related or not, and why would it only cause
> reception problems with FullDisk.

The FullDisk images in the SATELLITE feed are _the largest by far_ of any
products being distributed in the IDD, especially the Channel 02 images
which can be upwards of 440 MB in size.

re:
> I will dig into the latency and start by reducing the amount of data coming 
> in.
> Any other suggestions?

The first thing we want to see is all of the actual REQUEST lines in your
LDM configuration file, ~ldm/etc/ldmd.conf.

The next thing that we want to see is the outputs of:

/sbin/ifconfig -a    <- to get the name of your Ethernet interface

dmesg | grep <name of your Ethernet interface>

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: AFT-567406
Department: Support LDM
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.