[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [conduit] Partial GFS files
- Subject: Re: [conduit] Partial GFS files
- Date: Wed, 17 Nov 2021 16:38:30 -0800
Hello Pete and Mike (and all),
Thank you Pete for your excellent assessment of the problem. Since I
run twice-daily WRF runs initialized and forced by GFS 0.25-degree
model forecasts for hours 0 to 180, I have had to develop a program
that uses wgrib2 to find out which GFS files are missing or incomplete
(which is actually much more likely than being completely missing). I
then run a wget command to retrieve these bad grids from either or both
of these servers:
server1 = "ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod";
server2 = "https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod";
I also need to do this for 1-degree GFS files to produce GFS forecast
graphics for our web page. It took a fair amount of effort to set
this up, but it has been necessary and now, after many iterations, it
is quite reliable. On a couple of rare occasions, however, certain
fields at certain pressure levels never made it into a specific GFS
forecast hour's file on either of those servers and I had to pivot.
On average, over the past 4 0Z/12Z cycles, my scripts needed to use
wget to retrieve about 12 files per run (of the 61 files for hour
0-180) due to CONDUIT files being incomplete.
I still recommend that CONDUIT carry the GFS 0.25-degree files, since
about 75% of the data are useful and CONDUIT does save us from
retrieving it all on our own.
If people are interested and have already built wgrib2, I suppose I
could make my PERL and CSH scripts available to the community. They
would need some editing to remove our local idiosyncracies.
Cheers,
David
--
David Ovens e-mail: address@hidden
Research Meteorologist phone: (206) 685-8108
Dept of Atm. Sciences plan: Real-time WRF forecasting for the
Box 351640 Pacific Northwest
University of Washington https://atmos.uw.edu/wrfrt
Seattle, WA 98195 Weather Graphics and Loops
https://atmos.uw.edu/~ovens/loops
On Wed, Nov 17, 2021 at 05:06:13PM +0000, Pete Pokrandt via conduit wrote:
> Mike (and all),
>
> This has been an ongoing issue for several months now. The root problem is
> that the queue sizes on the ingest machines at NCEP are not nearly large
> enough to handle the amount of data flowing through them. When the lags get
> large, products age off of their queues before the downstream sites have a
> chance to get them.
>
> We are a top level CONDUIT relay, so we are pulling data direct from NCEP and
> we have been plagued with these missing products for several months now.
> Unidata's servers are also generally missing the same products that we are .
>
> The Unidata folks are also aware of the issue, but since NCEP runs the
> servers, there's not much the Unidata folks can do other than let NCEP know
> that they need larger queues. It sounds like NCEP is planning to address
> this, but not until at least sometime in 2022.
>
> The short term solution (and Tom from Unidata can correct me if I'm way off
> base) is probably either
>
>
> 1. get rid of the 0.25 degree data from the CONDUIT stream until they can
> increase the queue sizes on the source machines
>
> or
>
> 1. Set up a data feed separate from CONDUIT where one site (Unidata?)
> downloads the data from the NCEP's server and injects it into a data feed
> separate from CONDUIT - sort of a parallel feed of the data that is flowing
> through CONDUIT.
>
> This has been frustrating for a while now.. The GFS data coming through
> CONDUIT is so unreliable that it is pretty useless as it is.
>
> Pete
>
>
> <http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>-----
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086 - address@hidden
>
> ________________________________
> From: conduit <address@hidden> on behalf of Michael Leuthold <address@hidden>
> Sent: Wednesday, November 17, 2021 9:28 AM
> To: address@hidden <address@hidden>
> Subject: [conduit] Partial GFS files
>
> Hello,
>
> Has anyone else had recent issues with missing GFS files? Over the
> past few weeks, I've had many that are short. The problem usually
> starts around forecast hour 100 and resolves around hour 200. Both .25
> and .5 degree have this issue. It's worse at 0 and 12Z.
>
> thanks.
>
> Mike
>
> -rw-rw-r-- 1 ldm ldm 523M Nov 17 04:16 2132100F081.grib2
> -rw-rw-r-- 1 ldm ldm 523M Nov 17 03:57 2132100F084.grib2
> -rw-rw-r-- 1 ldm ldm 522M Nov 17 04:17 2132100F087.grib2
> -rw-rw-r-- 1 ldm ldm 524M Nov 17 03:59 2132100F090.grib2
> -rw-rw-r-- 1 ldm ldm 521M Nov 17 04:00 2132100F093.grib2
> -rw-rw-r-- 1 ldm ldm 526M Nov 17 04:01 2132100F096.grib2
> -rw-rw-r-- 1 ldm ldm 127M Nov 17 04:18 2132100F099.grib2
> -rw-rw-r-- 1 ldm ldm 379M Nov 17 04:19 2132100F102.grib2
> -rw-rw-r-- 1 ldm ldm 521M Nov 17 04:03 2132100F105.grib2
> -rw-rw-r-- 1 ldm ldm 314M Nov 17 04:20 2132100F108.grib2
> -rw-rw-r-- 1 ldm ldm 519M Nov 17 04:04 2132100F111.grib2
> -rw-rw-r-- 1 ldm ldm 209M Nov 17 04:22 2132100F114.grib2
> -rw-rw-r-- 1 ldm ldm 524M Nov 17 04:06 2132100F117.grib2
> -rw-rw-r-- 1 ldm ldm 528M Nov 17 04:07 2132100F120.grib2
> -rw-rw-r-- 1 ldm ldm 181M Nov 17 04:24 2132100F123.grib2
> -rw-rw-r-- 1 ldm ldm 160M Nov 17 04:24 2132100F126.grib2
> -rw-rw-r-- 1 ldm ldm 524M Nov 17 04:08 2132100F129.grib2
> -rw-rw-r-- 1 ldm ldm 525M Nov 17 04:09 2132100F132.grib2
> -rw-rw-r-- 1 ldm ldm 317M Nov 17 04:25 2132100F135.grib2
> -rw-rw-r-- 1 ldm ldm 524M Nov 17 04:10 2132100F138.grib2
> -rw-rw-r-- 1 ldm ldm 474M Nov 17 04:27 2132100F141.grib2
> -rw-rw-r-- 1 ldm ldm 522M Nov 17 04:28 2132100F144.grib2
> -rw-rw-r-- 1 ldm ldm 220M Nov 17 04:29 2132100F147.grib2
> -rw-rw-r-- 1 ldm ldm 309M Nov 17 04:31 2132100F150.grib2
> -rw-rw-r-- 1 ldm ldm 519M Nov 17 04:14 2132100F153.grib2
> -rw-rw-r-- 1 ldm ldm 149M Nov 17 04:32 2132100F156.grib2
> -rw-rw-r-- 1 ldm ldm 519M Nov 17 04:15 2132100F159.grib2
> -rw-rw-r-- 1 ldm ldm 252M Nov 17 04:35 2132100F162.grib2
> -rw-rw-r-- 1 ldm ldm 515M Nov 17 04:16 2132100F165.grib2
> -rw-rw-r-- 1 ldm ldm 520M Nov 17 04:17 2132100F168.grib2
> -rw-rw-r-- 1 ldm ldm 498M Nov 17 04:38 2132100F171.grib2
> -rw-rw-r-- 1 ldm ldm 520M Nov 17 04:19 2132100F174.grib2
> -rw-rw-r-- 1 ldm ldm 517M Nov 17 04:19 2132100F177.grib2
> -rw-rw-r-- 1 ldm ldm 355M Nov 17 04:39 2132100F180.grib2
> -rw-rw-r-- 1 ldm ldm 427M Nov 17 04:40 2132100F183.grib2
> -rw-rw-r-- 1 ldm ldm 519M Nov 17 04:22 2132100F186.grib2
> -rw-rw-r-- 1 ldm ldm 515M Nov 17 04:23 2132100F189.grib2
>
> --
> Mike Leuthold
> Manager, Regional Weather Modeling Program
> University of Arizona, Hydrology and Atmospheric Sciences
> 520-282-1478
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> conduit mailing list
> address@hidden
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> conduit mailing list
> address@hidden
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
conduit mailing list
address@hidden
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/