[LDM #TYP-174654]: Getting mini radar files from LDM

Greetings,

> Hi there-
> 
> First of all, this may not be for you, but I want to start with ldm support.

Reading ahead, I'm betting this is going to be more data-related than the LDM 
itself, mostly because you're getting usable data.

No worries though, happy to assist all the same!
 

> I am investigating using our LDM instead of the UNIDATA TDS for some of our 
> products.
> 
> I am now pulling down the National Composite (GINI) from LDM.

Would you be able to provide a link to the TDS product(s) as well as any 
pertinent pqact entries?  I'll want to recreate what you might be experiencing, 
and I'd need to know which exact data sets you're using.


> I use this to make weather products in Python, and while there are many 
> examples on how to display this radar data via the TDS, I can’t find any that 
> display radar data that is on the local disk.

If this is GINI format, or frankly any other weather data format, our MetPy 
Python library should be able to help.  Here's an example of plotting water 
vapor imagery in GINI format, I'd expect radar data to be much the same process 
if it's in the same format 
(https://unidata.github.io/MetPy/latest/examples/formats/GINI_Water_Vapor.html).
  Once I know what data you're working with I can try to plot that myself.  But 
that will read data from the local disk; I believe you just have to replace 
"get_test_data('WEST-CONUS_4km_WV_20151208_2200.gini', as_file_obj=False)" with 
the file path.


It sounds like you are referring to the feature of the TDS that shows a data 
preview in the browser before you retrieve the data, is that correct?  The LDM 
doesn't have that feature as the software is fundamentally different...

The TDS can be thought of as a data repository, where you can go to PULL the 
data you select and choose.  The LDM on the other hand is a subscription PUSH 
service; your client sends our server a subscription request, and anytime our 
server sees a product matching that request it pushes it to you.  The obvious 
benefit to this is you can act on the data immediately once it is available 
instead of a scripted schedule.  But because of that, and because this is 
server-side software without any GUI, a product preview isn't possible.


> When I open the data, the metadata looks different from what I get using 
> CdmRemote:
> 
> <xarray.Dataset>
> ...

Here too I'd need to know what data this is.  The good news though is you're 
able to read data that the LDM had saved for you.  If that wasn't working, 
xarray would have choked on this.  That's why I'm thinking this isn't LDM 
related, it looks like we're past that stage.


> I am overlaying this data on satellite data. Do you have any thoughts on how 
> to either manipulate the LDM to bring in a slightly different version of the 
> file? Or (more likely) an example on how to overlay LDM-acessed gini files 
> using Python?

So this is a dangerous question of sorts, only because as a former product 
developer I'm having too many thoughts come to mind. :)

Much of this question boils down to personal preference and what you are trying 
to achieve.  If you're trying to overlay radar data on satellite imagery, my 
first question is why GINI?  There are some good answers to that, but this is a 
less common route so I'm curious.  It could also be a factor if you or your 
site have bandwidth concerns and want to minimize the data flow.  But in the 
end you have a few options:

If you're using the composite radar data I think you are, that would be a good 
choice if you're either trying to retrieve as little data [bandwidth] as 
possible, it's probably the lightest weight out of all the options.  The same 
_might_ be true for satellite data, but that can be very situational.

This data is also available in grib2 format, a mosaic we create same as above 
but gridded.  This is an optimal choice for many as gridded data is often 
easier to work with than GINI, but it's situational/preferential.  I'm having 
trouble finding the details on this data now, I'll get back to you with more.

Another option could be MRMS data 
(https://www.unidata.ucar.edu/data/fnexrad.html#mrms_products), also in grib2 
format.  There are a variety of products both on that feed page, as well as on 
another NOAAPort feed.  Some are very nice, QC to remove ground clutter and 
such, and most if not all are rather hi-res at 1km.  This is a good choice if 
you want gridded data, QC'd data and/or high temporal resolution; the data is 
on a 2-minute cycle.

And lastly, depending on your needs you could request the individual site data 
and make your own mosaics instead.  My guess is this wouldn't be your first 
choice, but if you only wanted data over a relatively small area and wanted to 
minimize bandwidth, this could be a good way to do that.  This data is in NIDS 
format.


MetPy should be able to work with all of the above, and therefore you should 
have a good range of options to visualize with Python.  It's also important to 
remember that while Python is a great choice for data visualizations, it's not 
always the fastest.  Say you wanted to make a dozen of these products using the 
latest MRMS data.  You'd have to get all your processing done in under two 
minutes, otherwise your processing will start to snowball and your machine is 
going to have a bad time.  So it all pretty much comes down to the details.

 
> I apologize if this doesn’t make total sense. Please feel free to ask 
> questions.

Not at all, hopefully some of what I said was able to help.  I should be able 
to get you more answers on your issue once you can point me to the data you 
were / are trying to use.


> Have a good weekend
Thanks, you too!

-Mike

Ticket Details
===================
Ticket ID: TYP-174654
Department: Support IDD
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with NSF Unidata User Support are recorded in the 
Unidata inquiry tracking system and then made publicly available through the 
web.  If you do not want to have your interactions made available in this way, 
you must let us know in each email you send to us.