Re: [conduit] GFS pgrb2.0p25 missing a lot of fields
Subject: Re: [conduit] GFS pgrb2.0p25 missing a lot of fields
Date: Mon, 6 Nov 2023 23:27:20 +0000
Msip_labels:
So, from NCEP's perspective, I'm wondering what to expect as far as fixing this problem. Can we expect to see the CONDUIT stream restored to the way it was before the RH8 upgrade, or should I start modifying codes to use the new field names? This change is
really screwing up my data processing.
Art
Arthur A. Person Assistant Research Professor, System Administrator Penn State Department of Meteorology and Atmospheric Science email:address@hidden,
phone:814-863-1563
From: conduit <address@hidden> on behalf of Mike Zuranski <address@hidden> Sent: Tuesday, October 31, 2023 11:43 PM To: Levi Jones - NOAA Affiliate <address@hidden> Cc: address@hidden <address@hidden> Subject: Re: [conduit] GFS pgrb2.0p25 missing a lot of fields
Hi Levi,
Is there any chance we'd be able to see the grib tables that were being used before, do a comparison?
Thanks,
-Mike
Mike Zuranski
Data Engineer II
Unidata Program Center
University Corporation for Atmospheric Research
On Tue, Oct 31, 2023 at 9:51 PM Jim Bresch <address@hidden> wrote:
A couple of comments...
The 'Used to be' column in Pete's email are GEMPAK parameters. The "is now" column are NCEP's grib abbreviations for the same fields. I can't comment on whether LDM/GEMPAK must have the GEMPAK label to function properly, but I'm seeing no issues with
my systems.
NCEP may be distributing fields that have no GEMPAK label. Therefore, it is probably best to switch to NCEP's grib abbreviations for everything.
Regarding PRMSL, it should not be used and in my view should no longer be distributed. Everyone should be using MSLET (available in the GFS since 2011). PRMSL is truncated and smoothed. You can Google for the details.
Jim
Jim Bresch / NCAR/MMM
On 10/31/23 7:58 AM, Levi Jones - NOAA Affiliate via conduit wrote:
Daryl,
Below is the Github where Unidata keeps the tables, they are managed separately from LDM and I am seeking their advice/help on what to do about the issues we are seeing with this feature of LDM.
On Tue, Oct 31, 2023 at 9:40 AM Herzmann, Daryl E [AGRON] <address@hidden> wrote:
Greetings Levi,
Thanks for your efforts on this. Could you/NCO consider placing these lookup tables on github / some website so that the community can see them and perhaps help out with their curation? Presently, I have been duplicating all my entries for the old and new
variable names as I am unsure if they are going to be switching back at some point or what is going on. It would be nice to see the canonical source and to know what all is potentially available.
daryl
________________________________________
From: conduit <address@hidden> on behalf of Levi Jones - NOAA Affiliate via conduit <address@hidden>
Sent: Tuesday, October 31, 2023 7:40 AM
To: Pete Pokrandt
Cc: address@hidden
Subject: Re: [conduit] GFS pgrb2.0p25 missing a lot of fields
Yea no worries,
So last week I did make changes to the tables that load the labels and there should have been improvements, but I am still aware of a handful of records not being labeled.
-Levi
On Tue, Oct 31, 2023 at 8:33 AM Pete Pokrandt <address@hidden<mailto:address@hidden>> wrote:
Actually now that I look again, the MSLP is coming through labeled as it should be - PRMSL - not sure how I missed that the first time.
Anyways looks like mostly just the different constituent mixing ratios are the main things that are unlabeled.
Levi - just saw your other email come through as I was composing this one. Thanks for keeping us up to date!
Did anything ever come of this? There are still a bunch of unlabeled fields, and the labels on some of the fields that are identified are different than they were before, as I pointed out below.
The missing PRMSL is still biting me. Any plans to look into this?
Ok, as promised - here is a list of the products from the 144h of the 0p25 GFS run from 12 UTC today that are coming through unlabeled in the ldm insert, assuming that the product number in the ldmadmin watch is the same as the record number in the grib2 file
- it should be.
I'm guessing a lot of these products (ice/rain/cloud/snow mixing ratio especially) were NOT being distributed back in September, 2022, so that may not be a good comparison.. It is odd though that PRMSL is one of them. Also ABSV (Absolute vorticity) - I'd have
thought that was in the data back in 2022 - although I do see something called AVOR in my listing from 2022.
Just for reference..
Pete
1:0:d=2023102412:PRMSL Pressure Reduced to MSL [Pa]:mean sea level:144 hour fcst:
2:997030:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:1 hybrid level:144 hour fcst:
3:1069957:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:1 hybrid level:144 hour fcst:
4:1382628:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:1 hybrid level:144 hour fcst:
5:1612356:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:1 hybrid level:144 hour fcst:
204:164612372:d=2023102412:ABSV Absolute Vorticity [1/s]:50 mb:144 hour fcst:
205:165775392:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:50 mb:144 hour fcst:
206:165775571:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:50 mb:144 hour fcst:
207:165775750:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:50 mb:144 hour fcst:
208:166776746:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:50 mb:144 hour fcst:
230:186417815:d=2023102412:ABSV Absolute Vorticity [1/s]:100 mb:144 hour fcst:
231:187243899:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:100 mb:144 hour fcst:
232:187244714:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:100 mb:144 hour fcst:
233:187278169:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:100 mb:144 hour fcst:
234:188391095:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:100 mb:144 hour fcst:
246:196980183:d=2023102412:ABSV Absolute Vorticity [1/s]:150 mb:144 hour fcst:
247:197845797:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:150 mb:144 hour fcst:
248:197871203:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:150 mb:144 hour fcst:
249:198015734:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:150 mb:144 hour fcst:
250:198930760:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:150 mb:144 hour fcst:
262:206955540:d=2023102412:ABSV Absolute Vorticity [1/s]:200 mb:144 hour fcst:
263:207870505:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:200 mb:144 hour fcst:
264:207926442:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:200 mb:144 hour fcst:
265:208235792:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:200 mb:144 hour fcst:
266:209108935:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:200 mb:144 hour fcst:
278:217697746:d=2023102412:ABSV Absolute Vorticity [1/s]:250 mb:144 hour fcst:
279:218650963:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:250 mb:144 hour fcst:
280:218652248:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:250 mb:144 hour fcst:
281:219055753:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:250 mb:144 hour fcst:
282:219958686:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:250 mb:144 hour fcst:
294:228798811:d=2023102412:ABSV Absolute Vorticity [1/s]:300 mb:144 hour fcst:
295:229788165:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:300 mb:144 hour fcst:
296:229790248:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:300 mb:144 hour fcst:
297:230230399:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:300 mb:144 hour fcst:
298:231134637:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:300 mb:144 hour fcst:
310:239847579:d=2023102412:ABSV Absolute Vorticity [1/s]:350 mb:144 hour fcst:
311:240811800:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:350 mb:144 hour fcst:
312:240814119:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:350 mb:144 hour fcst:
313:241317442:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:350 mb:144 hour fcst:
314:241729278:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:350 mb:144 hour fcst:
326:250416980:d=2023102412:ABSV Absolute Vorticity [1/s]:400 mb:144 hour fcst:
327:251362060:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:400 mb:144 hour fcst:
328:251366911:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:400 mb:144 hour fcst:
329:251856693:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:400 mb:144 hour fcst:
330:252266845:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:400 mb:144 hour fcst:
342:260815389:d=2023102412:ABSV Absolute Vorticity [1/s]:450 mb:144 hour fcst:
343:261735614:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:450 mb:144 hour fcst:
344:261753672:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:450 mb:144 hour fcst:
345:262255560:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:450 mb:144 hour fcst:
346:262413289:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:450 mb:144 hour fcst:
358:271826922:d=2023102412:ABSV Absolute Vorticity [1/s]:500 mb:144 hour fcst:
359:272711621:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:500 mb:144 hour fcst:
360:272772279:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:500 mb:144 hour fcst:
361:273261228:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:500 mb:144 hour fcst:
362:273450282:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:500 mb:144 hour fcst:
374:282898538:d=2023102412:ABSV Absolute Vorticity [1/s]:550 mb:144 hour fcst:
375:283760810:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:550 mb:144 hour fcst:
376:283875231:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:550 mb:144 hour fcst:
377:284353649:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:550 mb:144 hour fcst:
378:284403507:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:550 mb:144 hour fcst:
390:293745941:d=2023102412:ABSV Absolute Vorticity [1/s]:600 mb:144 hour fcst:
391:294596033:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:600 mb:144 hour fcst:
392:294733772:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:600 mb:144 hour fcst:
393:295130549:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:600 mb:144 hour fcst:
394:295213273:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:600 mb:144 hour fcst:
406:304658253:d=2023102412:ABSV Absolute Vorticity [1/s]:650 mb:144 hour fcst:
407:305510967:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:650 mb:144 hour fcst:
408:305646381:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:650 mb:144 hour fcst:
409:305997861:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:650 mb:144 hour fcst:
410:306098501:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:650 mb:144 hour fcst:
422:315852246:d=2023102412:ABSV Absolute Vorticity [1/s]:700 mb:144 hour fcst:
423:316715652:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:700 mb:144 hour fcst:
424:316874416:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:700 mb:144 hour fcst:
425:317231161:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:700 mb:144 hour fcst:
426:317350484:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:700 mb:144 hour fcst:
438:327189187:d=2023102412:ABSV Absolute Vorticity [1/s]:750 mb:144 hour fcst:
439:328066287:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:750 mb:144 hour fcst:
440:328248181:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:750 mb:144 hour fcst:
441:328576691:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:750 mb:144 hour fcst:
442:328719224:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:750 mb:144 hour fcst:
454:338724840:d=2023102412:ABSV Absolute Vorticity [1/s]:800 mb:144 hour fcst:
455:339624164:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:800 mb:144 hour fcst:
456:339872341:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:800 mb:144 hour fcst:
457:340196315:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:800 mb:144 hour fcst:
458:340368444:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:800 mb:144 hour fcst:
470:350778656:d=2023102412:ABSV Absolute Vorticity [1/s]:850 mb:144 hour fcst:
471:351709423:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:850 mb:144 hour fcst:
472:352084821:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:850 mb:144 hour fcst:
473:352412387:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:850 mb:144 hour fcst:
474:352654336:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:850 mb:144 hour fcst:
486:363027849:d=2023102412:ABSV Absolute Vorticity [1/s]:900 mb:144 hour fcst:
487:363971658:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:900 mb:144 hour fcst:
488:364339125:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:900 mb:144 hour fcst:
489:364644575:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:900 mb:144 hour fcst:
490:364922210:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:900 mb:144 hour fcst:
502:375213692:d=2023102412:ABSV Absolute Vorticity [1/s]:925 mb:144 hour fcst:
503:376156670:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:925 mb:144 hour fcst:
504:376475917:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:925 mb:144 hour fcst:
505:376785962:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:925 mb:144 hour fcst:
506:377063862:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:925 mb:144 hour fcst:
518:387171099:d=2023102412:ABSV Absolute Vorticity [1/s]:950 mb:144 hour fcst:
519:388109806:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:950 mb:144 hour fcst:
520:388324327:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:950 mb:144 hour fcst:
521:388602305:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:950 mb:144 hour fcst:
522:388869736:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:950 mb:144 hour fcst:
535:398856429:d=2023102412:ABSV Absolute Vorticity [1/s]:975 mb:144 hour fcst:
536:399789752:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:975 mb:144 hour fcst:
537:399926489:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:975 mb:144 hour fcst:
538:400126569:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:975 mb:144 hour fcst:
539:400373957:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:975 mb:144 hour fcst:
550:408742634:d=2023102412:ABSV Absolute Vorticity [1/s]:1000 mb:144 hour fcst:
551:409671199:d=2023102412:CLMR Cloud Mixing Ratio [kg/kg]:1000 mb:144 hour fcst:
552:409721118:d=2023102412:ICMR Ice Water Mixing Ratio [kg/kg]:1000 mb:144 hour fcst:
553:409840305:d=2023102412:RWMR Rain Mixing Ratio [kg/kg]:1000 mb:144 hour fcst:
554:410028889:d=2023102412:SNMR Snow Mixing Ratio [kg/kg]:1000 mb:144 hour fcst:
579:422705581:d=2023102412:PEVPR Potential Evaporation Rate [W/m^2]:surface:144 hour fcst:
584:426333488:d=2023102412:RH Relative Humidity [%]:2 m above ground:144 hour fcst:
591:431360739:d=2023102412:CPOFP Percent frozen precipitation [%]:surface:144 hour fcst:
599:435967795:d=2023102412:ACPCP Convective Precipitation [kg/m^2]:surface:0-6 day acc fcst:
608:437062442:d=2023102412:CRAIN Categorical Rain [-]:surface:138-144 hour ave fcst:
609:437147546:d=2023102412:LHTFL Latent Heat Net Flux [W/m^2]:surface:138-144 hour ave fcst:
611:438562857:d=2023102412:GFLUX Ground Heat Flux [W/m^2]:surface:138-144 hour ave fcst:
612:439090830:d=2023102412:UFLX Momentum Flux, U-Component [N/m^2]:surface:138-144 hour ave fcst:
613:439733862:d=2023102412:VFLX Momentum Flux, V-Component [N/m^2]:surface:138-144 hour ave fcst:
617:442075445:d=2023102412:V-GWD Meridional Flux of Gravity Wave Stress [N/m^2]:surface:138-144 hour ave fcst:
618:442361008:d=2023102412:VEG Vegetation [%]:surface:144 hour fcst:
628:447849943:d=2023102412:RH Relative Humidity [%]:entire atmosphere (considered as a single layer):144 hour fcst:
662:478897390:d=2023102412:PRES Pressure [Pa]:tropopause:144 hour fcst:
669:486166788:d=2023102412:PRES Pressure [Pa]:max wind:144 hour fcst:
716:530288348:d=2023102412:RH Relative Humidity [%]:0.44-0.72 sigma layer:144 hour fcst:
717:531006466:d=2023102412:TMP Temperature [K]:0.995 sigma level:144 hour fcst:
718:531878007:d=2023102412:POT Potential Temperature [K]:0.995 sigma level:144 hour fcst:
719:532738129:d=2023102412:RH Relative Humidity [%]:0.995 sigma level:144 hour fcst:
720:533573797:d=2023102412:UGRD U-Component of Wind [m/s]:0.995 sigma level:144 hour fcst:
721:534567172:d=2023102412:VGRD V-Component of Wind [m/s]:0.995 sigma level:144 hour fcst:
727:538028008:d=2023102412:PLPL Pressure of level from which parcel was lifted [Pa]:255-0 mb above ground:144 hour fcst:
730:539109833:d=2023102412:ALBDO Albedo [%]:surface:138-144 hour ave fcst:
I don't have a good ldmadmin watch from just before the update, but I have one from September of 2022. There seem to be significantly more unknown products now than then.
Here is the full list from the 144 hour forecast of the 12 UTC Sept 5, 2022 0p25 files - 10 grib records
Below is the list from the 144 hour forecast from 12 UTC today (October 24, 2023) 0p25 files - 137 grib records
These are out of order compared the original GRIB file because we use parallel requests to grab the conduit feed - they are not guaranteed to come in in the same order as the original GRIB file, but I should be able to sort by the product number at the very
end, and compare to a file ftp'd from an ncep server. I'll try to get to that and let everyone know which exact products are not getting labeled when inserted into the ldm queue.
I am with NCO Data Flow and I implemented the changes yesterday to fix the label issue. I verified there are significantly more fields being encoded properly than before, but it seems there are still several going out with generic labels. Now that being said,
when I saw this, I went back and looked at the pre-upgrade logs, there were also a significant amount of generic labels going out.
So my question to you all is with the changes I have made yesterday are we still seeing more generic labels than pre-upgrade?
I made the changes 10/23 just before the 12Z GFS came out.
Also keep in mind with these new grib tables, associated with the updated ldm version, there is the possibility labels could have changed slightly.
Any and all feedback is helpful at this point.
Thanks
--
Levi A. Jones
WCCIS NCO - Dataflow
ERT
NOAA Affiliate
C: 606-748-7384
Seems like the GFS pgrb2.0p25 (and probably 0p50 also) data are missing a lot of fields with the RH8 upgrade, e.g. PMSL. Anyone else seeing this?
Art
Arthur A. Person
Assistant Research Professor, System Administrator
Penn State Department of Meteorology and Atmospheric Science
email: address@hidden<mailto:address@hidden>, phone: 814-863-1563<callto:814-863-1563>
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
conduit mailing list address@hidden<mailto:address@hidden>
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
conduit mailing list
address@hidden
For list information or to unsubscribe, visit: https://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
conduit mailing list
address@hidden
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/