[python #AHE-194717]: Siphon/THREDDS/NDFD Question

Greetings!

1. I wish it was simple, but NCEI's THREDDS server does not use collections for 
their GRIB holdings, and in fact they put each product in its own file, so it's 
not nearly as easy as it should be. Starting here:

https://www.ncei.noaa.gov/thredds/model/ndfd.html

You can either go for historical or current day's data, and with human-readable 
names or WMO header IDs. The problems are:
  a. For a script, you need to programmatically generate the URL to the 
directory of data you want like:

       
https://www.ncei.noaa.gov/thredds/catalog/model-ndfd-file-old/201910/20191017/catalog.xml

      OR

      walk in there with siphon with:

            base_cat = 
TDSCatalog('https://www.ncei.noaa.gov/thredds/model/ndfd.xml')
            sub_cat = base_cat.catalog_refs['NDFD - National Digital Forecast 
Database - Historical File Access'].follow()
            cat = 
sub_cat.catalog_refs['201812'].follow().catalog_refs['20181206'].follow()

       which is pretty verbose.

  b. The names of the files in either the WMO names or human-readable names are 
not really predictable, so you'll have to do some work in code to determine 
what the file you want is. For instance, look here:

     
https://www.ncei.noaa.gov/thredds/catalog/model-ndfd-file-old/201910/20191017/catalog.html

     My opinion: the WMO ones might be easiest to deal with, you'll just have 
to look at the spreadsheet you linked to figure out what the header you want 
is. I think ice accumulation starts with YZ.

Summary: It's going to be painful, but possible, to get it from NCEI.

2. We're saving to disk everything we're getting for the NDFD it appears, so I 
guess NOAAPORT just doesn't send out all of those fields. There are more from 
CONDUIT that we get:

https://thredds.ucar.edu/thredds/metadata/grib/NCEP/NDFD/NWS/CONUS/CONDUIT/Best?metadata=variableMap

but it's still not everything available. It's possible to request to add data 
to the CONDUIT feed, but that's beyond our control.

Ryan

> Ryan,
> 
> Thank you for the information. I have spent a few days digging through this 
> and seeing if I could get it closer to what I need. I had a couple clarifying 
> questions.
> 
> 1. Even if it only goes back to 2018 (despite saying 10 years), what do I 
> change "ds = cat.latest" to for a particular date/time? (Say Jan 1, 2020 at 
> 00 UTC)
> 
> 2. I have used the SOAP/XML to get "current" NDFD like the following:
> https://graphical.weather.gov/xml/SOAP_server/ndfdXMLclient.php?whichClient=NDFDgen&lat=35.23611&lon=-97.46488&product=time-series&begin=2020-05-31T00%3A00%3A00&end=2024-05-31T00%3A00%3A00&Unit=e&maxt=maxt&wspd=wspd&wgust=wgust&iceaccum=iceaccum
> 
> I only need those four variables ( 'Daily Maximum Temperature' ,'Wind Speed' 
> ,'Ice Accumulation' ,'Wind Speed Gust'). It looks like the variables in the 
> GRIB file example you sent has less than the full NDFD variables.
> 
> https://thredds.ucar.edu/thredds/metadata/grib/NCEP/NDFD/NWS/CONUS/NOAAPORT/Best?metadata=variableMap
> vs
> https://www.weather.gov/media/mdl/ndfd/NDFDelem_fullres_201906.xls
> 
> Is there a way to get the full list of variables?
> 
> Thank you again for all of your assistance.

Ticket Details
===================
Ticket ID: AHE-194717
Department: Support Python
Priority: Low
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.