[THREDDS #OYF-200327]: HRRR Data Question

Greetings!

The HRRR output we serve out though our demonstration TDS (thredds.ucar.edu) is
obtained directly from the NOAAPORT satellite broadcast. My understanding is 
that individual GRIB messages are sent out across the satellite broadcast at the
same time the model is running and output is being generated at NCEP. There will
be a delay associated with getting the output from NCEP servers uploaded into
the satellite broadcast, as well as the reception of the satellite broadcast on
our end (network transmission to the server hosting our TDS will also cause a
delay, but that one is relatively small).

As an example of timing, I logged the incoming messages we received for the 22
UTC run of the HRRR from yesterday (August 7th 2020). The server hosting our TDS
received the first forecast GRIB message from the HRRR at 22:58:50 UTC
(U-Component of Wind, 925 hPa, valid at 2020-08-07 22 UTC), and received the
final message at 23:22:23 UTC (Vertically-Integrated Liquid Water, valid at
2020-08-08 16 UTC). The TDS will wait until it believes all the GRIB messages 
for
a run have made been received from the broadcast, and will only expose the 
output
for access at that point. In this case, the 22 UTC run of the HRRR was 
accessible
at 23:30 UTC (so 8 minutes after we received the last GRIB message from the
satellite broadcast).

Because each GRIB message is independent (one horizontal slice valid at a single
time), you could start interrogating the output starting at 22:58 UTC if you had
your own satellite feed. However, because the TDS exposes collections of GRIB
messages as N-Dimensional variables, we have to wait until all of the messages
have arrived. If you did have your own satellite feed, you could specifically
watch the feed for any precip messages and act on those as soon as they hit 
disk.
Even then, you'd only get messages as fast the the model would output them, as
fast NCEP could get them into the NOAAPORT broadcast, and as fast as your
receiver received and decoded them. Note that this is the kind of setup the
National Weather Service forecast offices have in place.

Another approach would be to access the output directly from NCEP. As an 
example,
the GRIB messages for the HRRR can be found here:

https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/

NCEP collects all of the GRIB messages for a valid time and stores them as one
file. Again, using the 22 UTC run from yesterday, I saw that the collection of
messages valid at 22 UTC were made available at 22:45 UTC. The collection of 18
hour forecast (valid at 2020-08-08 16 UTC) were made available at 23:19 UTC.

To summarize the timings above for the 2020-08-07 22 UTC run of the HRRR:

TDS:
 * Output available at 23:30 UTC

NOAAPORT Satellite broadcast (with decoding/transmission delays):
 * First individual GRIB message available at 22:58:50 UTC
 * Last individual GRIB message available at 23:22:23 UTC

Direct from NCEP Servers:
 * 0 Hour forecast GRIB messages available at 22:45 UTC
 * 18 Hour forecast GRIB messages available at 23:19 UTC

One quick note about the GRIB collections we serve up on our TDS and the files
on the nomads server. The forecast GRIB messages we serve up as collections on
our TDS are contained within the files on the nomads server that follow the
pattern hrrr.t<RR>z.wrfprsf<FF>.grib2 (RR - runtime, FF - forecast hour).
However, not all of the GRIB messages generated by the HRRR are sent through the
satellite broadcast, so the files on the nomads server will contain more output
that what you see on our servers. 

I hope that gives you a better idea of dataflow used to drive our demonstration 
system, and how that relates to the process by which GRIB messages are produced
and transmitted from NCEP.

Cheers,

Sean

> Hello,
> 
> I was hoping to learn more information about the process by which datasets 
> become available on the THREDDS server. Specifically, we're using the 
> forecasted HRRR 1-hour precipitation products as a driver for a near 
> real-time flood model but we're finding the average lag between the model 
> initialization time and when it's publicly available to be on the order of 
> 1-2 hours. I realize there will always be some latency with such models but 
> do you know of any resources that could help reduce that lag?
> 
> Thank you in advance for your help - happy to provide additional details if 
> needed.
> 
> Best,


Ticket Details
===================
Ticket ID: OYF-200327
Department: Support THREDDS
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.