[Support #GNA-666049]: Re: 20150720: re: [conduit] Reminder: CONDUIT data volume will increase dramatically Jul y 28 with addition of GFS 0.25° model output

Hi Pete,

re:
> Can I have you take a look at the modifications that I plan to make to
> be sure they look right? //

OK.  The best way to test the REQUEST patterns is to do a 'notifyme'
that has the pattern you want to test.  More on this at the end of this
note.

re:
> An example non-gfs conduit line (random NAM grid) from ldmadmin watch -f
> conduit looks like:
> 
> data/nccf/com/nam/prod/nam.20150723/nam.t18z.awip3d45.tm00.grib2
> !grib2/ncep/NAM_84/#000/201507231800F045/PRES/0 - GCBL! 000585
> 
> a random 0.5 deg GFS product looks like
> data/nccf/com/gfs/prod/gfs.2015040900/gfs.t00z.pgrb2.0p50.f126
> !grib2/ncep/GFS/#000/201504090000F126/HGHT/0
> 
> and a random status file looks like
> 
> .status.data/nccf/com/rap/prod/rap.20150723/rap.t19z.awp236pgrbf14.grib2
> 000297

Hmm.  I forgot about the status files when I came up with the examples
we put in the web page referenced by the announcement of the addition of
the 0.25 degree GFS data to CONDUIT.

re:
> The suggested fix to not get the 0.25 deg GFS data but to get everything
> else is
> 
> REQUEST CONDUIT "ncep/[^G]" upstream.host.name
> REQUEST CONDUIT "gfs.*[012]p[^2]" upstream.host.name
> 
> I don't think that will work right for me, since I need the status
> files, and they don't include the string 'ncep' in the line.

You could REQUEST the status files separately.

re:
> There are
> other things in conduit too that don't include the ncep string, like the
> ndfd from data2 (data2/ndfd/YEUZ98_KWBN_232102.grib2
> !grib2/nwstg/NWS_0/#000/201507232100F017/TMPK/0 - NONE! 000016

OK.  The first REQUEST above could include strings other than ncep.
For instance:

REQUEST CONDUIT "(ncep|ndfd)/[^G]" upstream.host.name

This assumes, of course that things like the NDFD do not have Product
IDS with string fragments that match 'ndfd/G'.

re:
> Plus I have a split feed, so there are multiple requests for each
> CONDUIT host.

Yes.

re:
> I'm thinking something like:
> 
> # Request all CONDUIT minus any gfs files - split feeds
> REQUEST CONDUIT "(^(gfs)).*[09]$" conduit.ncep.noaa.gov
> REQUEST CONDUIT "(^(gfs)).*[18]$" conduit.ncep.noaa.gov
> REQUEST CONDUIT "(^(gfs)).*[27]$" conduit.ncep.noaa.gov
> REQUEST CONDUIT "(^(gfs)).*[36]$" conduit.ncep.noaa.gov
> REQUEST CONDUIT "(^(gfs)).*[45]$" conduit.ncep.noaa.gov
> 
> # Request CONDUIT gfs NOT including 0p25 deg files - ending in 0 or 9
> REQUEST CONDUIT "gfs.*[012]p[^2].*[09]$" conduit.ncep.noaa.gov
> REQUEST CONDUIT "gfs.*[012]p[^2].*[18]$" conduit.ncep.noaa.gov
> REQUEST CONDUIT "gfs.*[012]p[^2].*[27]$" conduit.ncep.noaa.gov
> REQUEST CONDUIT "gfs.*[012]p[^2].*[36]$" conduit.ncep.noaa.gov
> REQUEST CONDUIT "gfs.*[012]p[^2].*[45]$" conduit.ncep.noaa.gov
> 
> 
> Does that look right to you?  Tried testing with notifyme but can't get
> it to recognize a pattern with a dollar sign in it..

Use single quotes in your 'notifyme' tests instead of double quotes.  For
instance:

% notifyme -vl- -f CONDUIT -p 'gfs.*[012]p[^2].*[09]$'

Jul 23 22:17:29 notifyme[2241] NOTE: Starting Up: localhost: 20150723221729.310 
TS_ENDT {{CONDUIT,  "gfs.*[012]p[^2].*[09]$"}}
Jul 23 22:17:29 notifyme[2241] NOTE: LDM-5 desired product-class: 
20150723221729.310 TS_ENDT {{CONDUIT,  "gfs.*[012]p[^2].*[09]$"}}
Jul 23 22:17:29 notifyme[2241] INFO: Resolving localhost to 127.0.0.1 took 
0.000329 seconds
Jul 23 22:17:29 notifyme[2241] NOTE: NOTIFYME(localhost): OK
^CJul 23 22:17:39 notifyme[2241] NOTE: exiting

We have two machines ingesting the new CONDUIT stream from NCEP:

gale.unidata.ucar.edu
lead.unidata.ucar.edu

Both of these machines should be configured to ALLOW machines with
'.edu' names to REQUEST data, so you can point your prospective REQUEST patterns
to either for testing.  For instance:

<as 'ldm' on idd.aos.wisc.edu>
notifyme -vl- -f CONDUIT -h gale.unidata.ucar.edu -p 'gfs.*[012]p[^2].*[09]$'

Please let me know if this doesn't work for you somehow.

re:
> Thanks!

No worries.

Just so you know: in order for us to be able to relay the data off of our
top level IDD relay cluster, idd.unidata.ucar.edu, we are bonding two
Gbps Ethernet interfaces together so that more than 1 Gbps can flow out
of each node.  We configured two machines first as a test, and the
other nodes in our cluster will be done tomorrow afternoon sometime.
In aggregate, idd.unidata.ucar.edu is now moving in excess of 4Gbs
all of the time!  That translates into 43 TB/day of data being moved!!

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: GNA-666049
Department: Support CONDUIT
Priority: Normal
Status: Closed