Greetings fellow LDMers!
As you may recall, a certain Service Change Notice garnered some attention a short time ago when some changes to the NOAAPort/SBN were announced. Stonie and I have been investigating the potential impacts & what this will mean for our community and we would like to share what we know.
TL;DR: We think we’ll be okay. We still have some questions ourselves, and we won’t have all the answers until they flip the switch, but we believe any impacts should be fairly limited for the most part. The best thing users can do is be ready to take any necessary action should it be needed that Monday morning.
What’s all this then?
SCN 24-54 (linked below) announces that as part of efforts to improve dataflow, the NWS will “migrate all grib, grib2, and bufr formatted data from the NMC channel to the NMC2 channel.” If you’re not familiar with NMC/NMC2 lingo, these are data channels for the SBN that while similar to LDM feedtypes do not exactly line up 1:1 with them. So what they’re doing is moving some data from one of these channels to another.
https://www.weather.gov/media/notification/pdf_2023_24/scn24-54_channel_realignment_to_sbn_aaa.pdfWhen LDM running NoaaportIngester receives SBN data, it does a series of checks to determine which LDM feedtype the product should be filed into. Again, it’s not a 1:1 relationship, and in places within the LDM code the logic gets a little fuzzy. So for those of us running LDMs, whether it’s at a NOAAPort receive site or requesting NOAAPort data relayed from another site, we could see this data get filed differently.
Dang it, Mike, I’m a Professor not an engineer! What does this mean for me?- The data itself is NOT changing. We believe the worst-case scenario here is we have to get used to new locations for data that may have moved around.
- For most products we anticipate no change to how they are received by the LDM. We believe the most likely adverse impact would be products formerly seen on HDS to now appear on NGRID, however we will not know the scale or selection of products until it happens.
- If you have REQUEST or PQACT entries that rely on feedtype-specific expressions (e.g. HDS instead of ANY) then there is a chance products that have been working could be missed, as they may now appear on a different feed.
What are the recommendations for LDM users?
It’s going to boil down to what YOUR configurations look like that will determine your best courses of action. In general though, here is our advice at this time:
- Take stock of your REQUEST and PQACT entries, especially those involving the HDS and NGRID feedtypes.
- Envision what your configurations would do if, for example, all HDS data moved into NGRID. Would EXEC actions fire twice? Would files be saved into improper directories?
- BE AWARE! Monday July 15th at or about 12Z the switch will be flipped. In the event something goes wrong (for them) they may revert. If you have concerns you may want to be around to monitor the rollout in case you need to take some action yourself.
What about the SCN’s recommendation to change everything over to just use ANY?Yeah, I was afraid you might bring that up. In general this actually isn’t the worst advice. It may seem counter-intuitive, in fact it goes against some guidance we’ve given users in the past. But in the spirit of ensuring you get everything you need this would be the safest bet.
However, there may be side-effects of doing this. Whether or not there are, or their potential impact, once again comes down to individual site configurations. Here are some potential concerns you may consider:
- I want to believe that if everything went over to use ANY instead of specific feedtypes that there would not be any overlap in Product IDs, but I’m not sure we can guarantee that. While I have begun efforts to catalog data on the SBN and IDD these tools are still some distance away from being able to answer those questions with confidence.
- In other words, say you save every data product into the same directory using the Product ID as its name (for the love of everything don’t do this! This is a thought experiment!!). If we ignore feedtypes entirely, would any data files be overwritten? We think the odds of this are pretty small.
- Some sites have seen that breaking ANY requests into multiple requests by feedtype results in significant performance improvements, in some cases such a configuration is required for the installation to work properly. If you know your site has such a concern, bear this in mind.
- Because we believe the potential impacts will be limited to the HDS and NGRID feedtypes, it is likely a safer option to replace only those REQUESTs with ANY instead of everything.
If you still have questions, feel free to reach out via our support channels. See below for more information.
A word about support:
As always, if you have any questions comments or concerns that need to be addressed in a timely manner, please do not hesitate to reach out to
address@hidden,
address@hidden,
address@hidden, or any other of our support inboxes listed here:
https://www.unidata.ucar.edu/support/index.html#request Admittedly it can be a little tough to choose at times, don’t worry about it too much as they will all still ping the right people. But requesting support through these inboxes really helps prevent things from falling through the cracks (plus the tracking makes us look good for NSF 😉). That being said here’s a guide:
- Support-idd for when you’re not getting data that you’re requesting from us
- Support-ldm when the LDM software just done broke
- Support-noaaport if you are a NOAAPort receive site and are having issues with that part of the pipeline
I will add that NSF Unidata staff are pretty loaded up these days, and it may take a little longer for us to respond to support inquiries. We always strive to address critical issues as soon as possible but it may not always be right away. This is especially true during holidays and weekends; we are not a 24/7 operation and do not always have the ability to address issues until the next business day.
At the same time, we have noticed that the occasional support ticket (new or a user response) doesn’t always alert us the way it should. We are prioritizing getting that addressed but please never feel like we are intentionally ignoring you. If at any time you feel your support inquiry has not been received, please send an email to
address@hidden as that inbox gets scrutinized a bit differently.
We ask that you try to write staff directly only as a last resort unless otherwise advised. It’s not that we don’t want to hear from you, but sometimes tapping on the glass startles the engineers. In all seriousness sometimes people go on PTO or other such things, and this can lead to delays in getting your issues addressed. Thank you for attending my NSF Unidata Support PSA. 😊
Monday July 15th at 12Z. I’ll be all coffee’d up and ready to go, and if stuff approaches ceiling fan levels I’ll do my best to keep you posted.
Best,
-Mike
Mike Zuranski
Data Engineer II
NSF Unidata Program Center
University Corporation for Atmospheric Research