[IDD #NSA-980986]: Data Product Frequency

Hi,

re:
> I am working on some updates for monitoring and documentation of our LDM
> systems. Is there a comprehensive list anywhere of the expected frequency
> of updates for each LDM data product? 

We do not maintain such a list.  Whether or not some Unidata community member
has compiled such a list is something we can not say.  It may be best to send
a note to the address@hidden email list that asks if any of the
community members are doing/attempting to create this kind of a list.

re:
> I have searched the documentation
> and found this listing here,
> 
> https://www.unidata.ucar.edu/software/ldm/ldm-current/basics/feedtypes/
> 
> but it only lists the expected data volume for each feedtype, not the
> expected frequency of updates.

Since the page you referenced doesn't contain any data volume numbers,
I must assume that you are referring to pages that are linked from that
page.

Why don't we try to maintain such a list?

The contents of quite a few of the datastreams that we relay in the IDD
are not created or maintained by us, and since the contents of feeds that
are derived from NOAAPort ingest are changing all of the time, it would
be a losing battle to try and keep an up to date list.  What one can do,
however, is use the information provided in our real-time stats pages
to get an idea of how many products are being (by upstreams) and received
(by downstreams) as a function of time.

Our real-time stats pages can be accessed as follows:

Unidata HomePage
https://www.unidata.ucar.edu

  Data -> IDD Operational Status
  https://rtstats.unidata.ucar.edu/rtstats/

    Real-time Statistics -> Statistics by Host
    https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex

The top level IDD relay clusters that we maintain are composed of real
server backends that report these statistics individually.  Our primary
cluster, idd.unidata.ucar.edu, is composed of 10 real server backends 
who report under the following names:

node0.unidata,ucar.edu
node1.unidata.ucar.edu
 ...
node9.unidata.ucar.edu

Since all of these machines are configured identically, looking at the
real-time stats from one of the machines should be the same as looking
at any other/all other machines.  Randomly choose node 4 to look at:

node4.unidata.ucar.edu
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?node4.unidata.ucar.edu

In this page, you will find a number of useful links that may be of use:

Cumulative Volume Summary
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume?node4.unidata.ucar.edu

Data Volume Summary for node4.unidata.ucar.edu

Maximum hourly volume 127207.914 M bytes/hour
Average hourly volume  77932.558 M bytes/hour

Average products per hour     522487 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
SATELLITE             15623.692    [ 20.048%]    20107.408     6681.591
CONDUIT               14931.155    [ 19.159%]    59844.396    91693.068
NEXRAD2                9234.290    [ 11.849%]    11170.434   107095.977
NIMAGE                 8918.835    [ 11.444%]    12122.954     6158.341
NOTHER                 7281.188    [  9.343%]    10212.195    12069.273
NGRID                  7272.923    [  9.332%]    12556.205    67009.159
FNEXRAD                4628.434    [  5.939%]     4856.848     9390.614
HDS                    3705.001    [  4.754%]     9332.327    40848.955
NEXRAD3                2444.790    [  3.137%]     2927.778   102790.477
FNMOC                  2097.572    [  2.692%]     6264.742     7707.523
UNIWISC                 984.639    [  1.263%]     1142.834      918.523
GEM                     721.492    [  0.926%]     2767.542     4236.114
IDS|DDPLUS               85.339    [  0.110%]       96.538    65503.614
LIGHTNING                 3.034    [  0.004%]        6.059      350.455
GPS                       0.110    [  0.000%]        1.143        1.045
FSL2                      0.062    [  0.000%]        0.500       32.318

The 'products' link for each feed type shows a time series for number
of products sent for that feed type, and the 'volume' link for each
feed type shows a time series for that feed type.

The IDD feeds that derive from NOAAPort are:

HDS
IDS|DDPLUS
NEXRAD3d
NGRID
NOTHER

NIMAGE, which used to be populated by NOAAPort, was repurposed to
be a vehicle to distribute GOES-R/S imagery and level 2 products
that are distributed in NOAAPort as tiles (SCMI products) and individual
products (other Level 2 products) and some Level 2 products created by
Unidata community members (Texas Tech and CSU/CIRA).  The difference
in the individual products and tiles is that the NOAAPort broadcast
headers and footers have been stripped off, and the tiles have been
stitched together into full scenes, thus saving the need for each
site being fed these products from having to do the same work.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: NSA-980986
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.