[THREDDS #GUY-357996]: Grabbing RAP data with xarray from THREDDS

Hi,

If xarray is handling the GFS data from the URL, I'm guessing it's using 
opendap. You can usually tell this is being used on a THREDDS server by the 
presence of "dodsC" in the URL. This was the case for the NAM analysis in the 
MetPy Monday you linked. I can't find the GFS URL you're talking about, but if 
you send it I'd be happy to dig in to see what it's using.

Now...unfortunately it looks like NCEI reconfigured their THREDDS server. The 
URLs I sent you before definitely worked when I wrote the response, but they 
are indeed broken now. And if I go to the server, I don't see anything in the 
catalog that gets me to model output. I'm not sure why it changed, but since we 
don't manage that server we don't have any control over how it's set up.

It looks like in the meantime you'll have to access the data over HTTPS and 
download it directly.

Hope this helps,

Ryan

> Hi Ryan,
> 
> Thanks for the help, I guess I'm just confused at how xarray knows how to 
> handle the GFS data from the URL, but fails when trying to access RAP data.
> 
> Additionally, It looks like the link you included below does not exist (404) 
> 'https://www.ncei.noaa.gov/thredds/catalog/ruc130anl/201010/20101026/catalog.xml'?
> I even tried 
> 'https://www.ncei.noaa.gov/thredds/ncss/ruc130anl/201010/20101026/catalog.xml'
>  and it still failed.
> 
> Thanks,
> Alex Schueth
> 
> ________________________________
> From: Unidata THREDDS Support <address@hidden>
> Sent: Thursday, February 6, 2020 3:08 PM
> To: Schueth, Alex <address@hidden>
> Cc: address@hidden <address@hidden>
> Subject: [THREDDS #GUY-357996]: Grabbing RAP data with xarray from THREDDS
> 
> Greetings!
> 
> You're close to having it, but there are a couple more steps you need to do 
> to get the data--and a couple of options on how to proceed. The big 
> difference is that you're handing XArray URLs to data on the server, but the 
> URLs are not for services that xarray understands how to use.
> 
> Also, I would use siphon to make your life easier:
> 
> from siphon.catalog import TDSCatalog
> import xarray as xr
> from xarray.backends import NetCDF4DataStore
> 
> # Open catalog with Siphon
> cat = 
> TDSCatalog('https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncei.noaa.gov%2Fthredds%2Fcatalog%2Fruc130anl%2F201010%2F20101026%2Fcatalog.xml&amp;data=02%7C01%7Calex.schueth%40ttu.edu%7C3cb93115ed0344e15bad08d7ab48cc64%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637166201476005964&amp;sdata=zTc5HMr2gggq8D4LAv%2B0cas1NpVFTZ1gv%2Frcbsuc65c%3D&amp;reserved=0')
> ds = cat.datasets['ruc2anl_130_20101026_1200_000.grb2']
> ncss = ds.subset()
> 
> # See what variables are available
> #print(ncss.variables)
> 
> # Use NCSS to download a subset of variables, letting THREDDS convert GRIB to 
> netCDF
> query = ncss.query()
> query.variables('Temperature_isobaric', 'Geopotential_height_isobaric')
> nc = ncss.get_data(query)
> 
> dataset = xr.open_dataset(NetCDF4DataStore(nc))
> 
> I was also able to use siphon to download the grib file and open using 
> cfgrib, but it wasn't immediately clear how to get cfgrib to open without 
> error:
> 
> # Use cfgrib
> ds.download()
> dataset = xr.open_dataset(ds.name, engine='cfgrib', 
> backend_kwargs={'filter_by_keys':{'typeOfLevel': 'isobaricInhPa'}})
> 
> Hope this helps,
> 
> Ryan
> 
> > Hi,
> >
> > I am struggling grabbing data from the THREDDS server using xarray. I'm
> > not sure if it is a problem accessing the server or if it is a problem
> > with cf grib. I thought I would check here first to verify that the
> > THREDDS implementation is correct.
> >
> > If I try the following command (borrowing some of the syntax from
> >
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_9j7Y1-lk-o&amp;data=02%7C01%7Calex.schueth%40ttu.edu%7C3cb93115ed0344e15bad08d7ab48cc64%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637166201476015917&amp;sdata=MPdME1FR4GgAn930jqcR0HY07lvrnITp%2FPA7epXVk9M%3D&amp;reserved=0):
> >
> > ds = 
> > xr.open_dataset('https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncei.noaa.gov%2Fthredds%2Fncss%2Fruc130anl%2F&amp;data=02%7C01%7Calex.schueth%40ttu.edu%7C3cb93115ed0344e15bad08d7ab48cc64%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637166201476015917&amp;sdata=GkFu0kXI28mHHk3%2BRQbuJdhDNPRx3tJ4k696yuf0jO4%3D&amp;reserved=0
> > 201010/20101026/ruc2anl_130_20101026_1200_000.grb2',decode_times    
> > True,engine=)
> >
> > It says the file does not exist. Following the link, the path is
> > correct. If I remove the https://, it seems to work, prompting me to
> > use the cfgrib engine despite grabbing it from the netcdf subset server?
> >
> > Regardless, if I use the cfgrib engine:
> >
> > ds = 
> > xr.open_dataset('https://nam04.safelinks.protection.outlook.com/?url=www.ncei.noaa.gov%2Fthredds%2Fncss%2Fruc130anl%2F201010%2F&amp;data=02%7C01%7Calex.schueth%40ttu.edu%7C3cb93115ed0344e15bad08d7ab48cc64%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637166201476015917&amp;sdata=%2F646n%2BpaB8yy96tNE%2BHUuhriT4xJOfCLOCIL1DvBQHQ%3D&amp;reserved=0
> > 20101026/ruc2anl_130_20101026_1200_000.grb2',decode_times=True,engine    
> > 'cfgrib')
> >
> > Can't create file 
> > '/home/alex/www.ncei.noaa.gov/thredds/ncss/ruc130anl/201010/20101026/ruc2anl_130_20101026_1200_000.grb2.4cc40.idx'
> >
> > Followed by an error saying it is unable to open that .idx file. I looked
> > into the documentation for cfgrib and it appears that the backend can
> > be changed to not create an .idx file, however I end up getting this:
> >
> > ds = 
> > xr.open_dataset('https://nam04.safelinks.protection.outlook.com/?url=www.ncei.noaa.gov%2Fthredds%2Fncss%2Fruc130anl%2F201010%2F&amp;data=02%7C01%7Calex.schueth%40ttu.edu%7C3cb93115ed0344e15bad08d7ab48cc64%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637166201476015917&amp;sdata=%2F646n%2BpaB8yy96tNE%2BHUuhriT4xJOfCLOCIL1DvBQHQ%3D&amp;reserved=0
> > 20101026/ruc2anl_130_20101026_1200_000.grb2',decode_times=True,engine    
> > 'cfgrib',backend_kwargs={'indexpath':''})
> >
> > FileNotFoundError: [Errno 2] No such file or directory: 
> > '/home/alex/www.ncei.noaa.gov/thredds/ncss/ruc130anl/201010/20101026/ruc2anl_130_20101026_1200_000.grb2'
> >
> > I'm sure somewhere along the way I'm getting tripped up in the syntax, but
> > I'm not sure exactly where.
> 
> 
> Ticket Details
> ===================
> Ticket ID: GUY-357996
> Department: Support THREDDS
> Priority: Normal
> Status: Closed
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: GUY-357996
Department: Support THREDDS
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.