[IDD #VRK-635010]: idd data

Hi Marck,

re: crafting an LDM REQUEST for specific grids from NGRID

> If it’s complicated, just leave the channel open don't worry about
> grid cover restrictions.

Right after sending you the email yesterday I believe that I crafted
a correct extended regular expression for the NGRID REQUEST.  I briefly
checked to see what you were receiving after implementing the addition
(remember, after any change to ~ldm/etc/ldmd.conf, one has to restart
the LDM), and it looked like you were/are getting the all grids in
NGRID that cover your area.

re:
> 950 MB in 75000 products a day is not that big anyways
> (http://www.unidata.ucar.edu/software/ldm/ldm-current/basics/feedtypes/ngrid.html).

The full volume of NGRID is quite a bit more than 950 MB/day.  You
can check the volume over the past couple of days for all feeds
available in the IDD from our real-time stats pages.  For instance,
here is how you would list the current volumes being reported by
one of toplevel IDD relay cluster's, idd.unidata.ucar.edu, real
server backends:

Unidata HomePage
http://www.unidata.ucar.edu

  Data -> IDD Operational Status
  http://www.unidata.ucar.edu/projects/idd/rtstats/

    Statistics by Host
    http://www.unidata.ucar.edu/cgi-bin/rtstats/siteindex

      Real-time Statistics for uni14.unidata.ucar.edu
      
http://www.unidata.ucar.edu/cgi-bin/rtstats/siteindex?uni14.unidata.ucar.edu

Click on the left link at the bottom:

Cumulative volume summary
http://www.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume?uni14.unidata.ucar.edu

This will result in a listing like:

Data Volume Summary for uni14.unidata.ucar.edu

Maximum hourly volume  20264.821 M bytes/hour
Average hourly volume  12228.592 M bytes/hour

Average products per hour     287493 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
NEXRAD2                3444.655    [ 28.169%]     4540.368    52493.044
CONDUIT                3239.972    [ 26.495%]     6094.883    73051.444
NEXRAD3                1315.536    [ 10.758%]     1665.192    76746.956
FSL2                   1310.731    [ 10.719%]     1712.212     1543.044
NGRID                  1120.736    [  9.165%]     2385.204    19996.556
FNMOC                  1042.083    [  8.522%]     6464.852     2775.778
HDS                     333.140    [  2.724%]      628.912    17392.356
NIMAGE                  143.169    [  1.171%]      269.929      193.067
GEM                      75.591    [  0.618%]      492.522      774.689
FNEXRAD                  72.893    [  0.596%]       92.229       71.578
IDS|DDPLUS               51.453    [  0.421%]       65.755    41723.578
EXP                      48.826    [  0.399%]      103.246      350.622
UNIWISC                  25.247    [  0.206%]       31.761       30.267
DIFAX                     3.441    [  0.028%]       13.310        4.733
LIGHTNING                 1.010    [  0.008%]        2.136      343.956
GPS                       0.108    [  0.001%]        1.342        1.044

You can see from this listing that the current average volume in the full
NGRID feed is 1120.731 MB/hr.  This translates into 26872.152 MB/day, and
this is ~26 GB/day, not 950 MB/day.  Quite a difference!  The numbers listed
in the page you referenced were grabbed in the early days of the NGRID
feed, and the volumes have been climbing steadily ever since.  To get
current values, one should always look at the Cumulative volume summary
for a machine that is getting all of a feed one is interested in, like
uni(14|16|17|18|19|20).unidata.ucar.edu.

By the way, the CONDUIT feed is the one that has the 0.5 degree Global GFS
model output.  The only problem is that these fields make up the lion's
share of the full CONDUIT volume, and CONDUIT contains about 77 GB/day
AND it is growing!!

Data, data, data... :-)

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: VRK-635010
Department: Support IDD
Priority: Normal
Status: Closed