[IDV #RZP-160177]: unable to read netcdf file.

Hi Mark,

Here, you will find a [test2.nc] file that can be read into the
IDV. (All reference links can be found at the bottom.) Also, here is the
[DAP] link that the IDV can read.

When I need guidance from CF in the form of examples, I usually turn to
the [NODC] website, and specifically, in this case, the section on
[grids]. Using that resource as a guide, we can use a bit of [Python] to
create a data file according to these standards. I know you may be more
familiar with IDL, but Python is fairly straightforward to understand.

First create a netCDF file to store your data.

,----
| import netCDF4
| import numpy as np  
| 
| try:
|     ncfile.close()
| except:
|     pass
| ncfile = netCDF4.Dataset('/home/python/work/test2.nc',
|                          mode='w',
|                          format='NETCDF4_CLASSIC')
| ncfile
`----

Set some global attributes pertaining to standards and whatnot.

,----
| ncfile.Conventions = 'CF-1.6'
| ncfile.Metadata_Conventions = 'Unidata Dataset Discovery v1.0'
| ncfile.cdm_data_type = 'Grid'
| ncfile.featureType = "grid"
| ncfile.nodc_template_version = 'NODC_NetCDF_Grid_Template_v1.1'
`----

Now create the `lat', `lon', `z', `time' dimensions.

,----
| lat_dim = ncfile.createDimension('lat', ds['latitude'].size)
| lon_dim = ncfile.createDimension('lon', ds['longitude'].size)
| z_dim = ncfile.createDimension('z', ds['pressure'].size)
| time_dim = ncfile.createDimension('time', None)
`----

Create `lat' variable.

,----
| lat = ncfile.createVariable('lat', np.float32, ('lat',))
| lat.units = 'degrees_north'
| lat.long_name = 'latitude'
`----

Create `lon' variable.

,----
| lon = ncfile.createVariable('lon', np.float32, ('lon',))
| lon.units = 'degrees_east'
| lon.long_name = 'longitude'
`----

Create `z' variable

,----
| z = ncfile.createVariable('z', np.float32, ('z',))
| z.units = 'hPa'
| z.long_name = 'pressure'
`----

Create `time' variable.

,----
| time = ncfile.createVariable('time', np.float64, ('time',))
| time.units = 'hours since 2000-01-01'
| time.long_name = 'time'
`----

Create `temp' variable.

,----
| temp = ncfile.createVariable('temp',np.float64,('time','z', 'lat','lon'))
| temp.units = 'K' # degrees Kelvin
| temp.standard_name = 'air_temperature' # this is a CF standard name
`----

Now set your data. The `ds' variable contains your original data. The
code for this is not shown.

,----
| lat[:] = ds['latitude'].data
| lon[:] = ds['longitude'].data
| z[:] = ds['pressure'].data
| time[:] = np.array([0])
| temp[0,:] = ds['temperature'].data
`----

You are done. Now close the file.

,----
| ncfile.close();
`----

Now let's look at the result of what we have done with `ncdump'.

,----
| ncdump -h ../../../test2.nc
`----

,----
| netcdf test2 {
| dimensions:
|   lat = 181 ;
|   lon = 361 ;
|   z = 100 ;
|   time = UNLIMITED ; // (1 currently)
| variables:
|   float lat(lat) ;
|     lat:units = "degrees_north" ;
|     lat:long_name = "latitude" ;
|   float lon(lon) ;
|     lon:units = "degrees_east" ;
|     lon:long_name = "longitude" ;
|   float z(z) ;
|     z:units = "hPa" ;
|     z:long_name = "pressure" ;
|   double time(time) ;
|     time:units = "hours since 2000-01-01" ;
|     time:long_name = "time" ;
|   double temp(time, z, lat, lon) ;
|     temp:units = "K" ;
|     temp:standard_name = "air_temperature" ;
| 
| // global attributes:
|     :Conventions = "CF-1.6" ;
|     :Metadata_Conventions = "Unidata Dataset Discovery v1.0" ;
|     :cdm_data_type = "Grid" ;
|     :featureType = "grid" ;
|     :nodc_template_version = "NODC_NetCDF_Grid_Template_v1.1" ;
| }
`----

Compare the results of this `ncdump' with the file you originally
uploaded to us.  Do you see the differences?

,----
| ncdump -h ../../../test.nc
`----

,----
| netcdf test {
| dimensions:
|   longitude = 361 ;
|   latitude = 181 ;
|   pressure = 100 ;
| variables:
|   float longitude(longitude) ;
|     longitude:longitude = "degrees" ;
|   float latitude(latitude) ;
|     latitude:latitude = "degrees" ;
|   float pressure(pressure) ;
|     pressure:pressure = "hPa" ;
|   float temperature(pressure, latitude, longitude) ;
|     temperature:temperature = "K" ;
|     temperature:Projection = "cylindrical" ;
|   float probability(pressure, latitude, longitude) ;
|     probability:probability = "1" ;
| 
| // global attributes:
|     :Title = "test" ;
| }
`----

At this point, you should have enough information to tidy up your netCDF
file with IDL. Let us know if you have any additional questions.

Best,

Unidata IDV Support


[test2.nc]
https://motherlode.ucar.edu/repository/entry/show/RAMADDA/Unidata/Staff/Julien+Chastang/mark/test2.nc?entryid=3cb9100e-6a62-4c5c-af37-8cd9966fcce6

[DAP]
https://motherlode.ucar.edu/repository/opendap/3cb9100e-6a62-4c5c-af37-8cd9966fcce6/entry.das

[NODC] https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/

[grids] https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/grid.cdl

[Python] https://github.com/Unidata/unidata-python-workshop



> The data is from the AIRS instrument on AQUA. The data is basically 
> retrievals.
> There are 100 pressure levels and I am averaging 10 years of data for each 
> day of the
> year.  I sent you the data for one day.   There are two 3d fields: 
> probability and air temperature.
> Now the probability field is the probability
> that the air will decrease below some temperature.  This is a data product 
> that
> airlines would like to have if they want to avoid flying in those regions. The
> goal is to visualize some of these regions as “bubbles” of cold air in the
> upper troposphere.  The other field is just the averaged temperature.
> 
> 
> I am creating the files using IDL netcdf commands
> [first var is probability]
> [second var is air temperature]
> 
> s=size(first_var)
> dim1=ncdf_dimdef(id,'longitude',s(1))
> dim2=ncdf_dimdef(id,'latitude',s(2))
> dim3=ncdf_dimdef(id,'pressure',s(3))
> 
> varlon=ncdf_vardef(id,'longitude',dim1,/float)
> varlat=ncdf_vardef(id,'latitude',dim2,/float)
> varpres=ncdf_vardef(id,'pressure',dim3,/float)
> 
> 
> ncdf_attput,id,varlat,'latitude','degrees',/char
> ncdf_attput,id,varlon,'longitude','degrees',/char
> ncdf_attput,id,varpres,'pressure','hPa',/char
> 
> var1_data=ncdf_vardef(id,v1name,[dim1,dim2,dim3],/float)
> var2_data=ncdf_vardef(id,v2name,[dim1,dim2,dim3],/float)
> 
> ; create the attributes
> 
> ncdf_attput,id,var1_data,v1name,'K',/char
> ncdf_attput,id,var2_data,v2name,'1',/char
> ncdf_attput,id, /GLOBAL, 'Title', filename
> ncdf_attput,id,var1_data,'Projection','cylindrical'
> 
> ncdf_control,id,/endef
> 
> ; now write out data
> 
> ncdf_varput,id,varlat,latgrid
> ncdf_varput,id,varlon,longrid
> ncdf_varput,id,varpres,pgrid
> 
> ncdf_varput,id,var1_data,first_var
> ncdf_varput,id,var2_data,second_var
> 
> 
> 
> address@hidden
> Chief Scientist
> Science and Technology Corporation
> 10015 Old Columbia Road
> Suite E-250
> Columbia,  MD  21046
> phone: 410-309-0818 x 106
> cell: 240-481-7390
> http://markschoeberl.com
> 
> 
> 
> 
> > On Dec 10, 2015, at 3:44 PM, Unidata IDV Support <address@hidden> wrote:
> >
> > Hi Mark,
> >
> > Yes, it is not always as easy as we would like for data providers to adhere 
> > to
> > standards.
> >
> > Let's step back for a moment. What is the provenance of these data? Where 
> > are
> > they from?  How are you converting them into netCDF format?
> > These data are straightforward so we should hopefully be able to arrive at 
> > the
> > goal here quickly.
> >
> > Best,
> >
> > Unidata IDV Support
> >
> >> okay, actually the web site you pointed me at was helpful.  In your 
> >> documentation, I suggest that
> >> you point to this site.
> >>
> >> Just so I understand what you are saying.  My variable ‘latitude’  should 
> >> be named ‘grid_latitude’
> >> and ‘temperature’  should be ‘air_temperature’
> >>
> >> However, there is no variable named ‘probability’  … how do I deal with 
> >> that or other variables that
> >> I might invent that have no standard name.
> >>
> >> Thank you,
> >>
> >> Mark
> >>
> >>
> >> address@hidden
> >> Chief Scientist
> >> Science and Technology Corporation
> >> 10015 Old Columbia Road
> >> Suite E-250
> >> Columbia,  MD  21046
> >> phone: 410-309-0818 x 106
> >> cell: 240-481-7390
> >> http://markschoeberl.com
> >>
> >>
> >>
> >>
> >>> On Dec 10, 2015, at 1:24 PM, Unidata IDV Support <address@hidden> wrote:
> >>>
> >>> Mark,
> >>>
> >>> The problem with these data is that they do not follow CF (or any) 
> >>> conventions
> >>> so the IDV cannot open them.
> >>>
> >>> http://www.unidata.ucar.edu/software/netcdf/conventions.html
> >>>
> >>> I would also maybe suggest specifying the central longitude for the 
> >>> cylindrical
> >>> projection. (Panoply must be assuming a central longitude of 0, I guess.)
> >>>
> >>>
> >>> Best,
> >>>
> >>> Unidata IDV Support
> >>>
> >>>
> >>>> File has been uploaded.
> >>>> Please let me know what I need to do to the nc file to get this IDV to 
> >>>> work.
> >>>> Thanks
> >>>>
> >>>> - mark
> >>>>
> >>>> On Dec 10, 2015, at 10:40 AM, Unidata IDV Support <address@hidden> wrote:
> >>>>
> >>>> Mark,
> >>>>
> >>>> Please upload your data to
> >>>>
> >>>> http://motherlode.ucar.edu/repository/alias/idvupload
> >>>>
> >>>> and let us know when it has been uploaded.
> >>>>
> >>>> It is also possible that you simply need to select a "Data Source Type" 
> >>>> of "Grid
> >>>> files (netCDF/GRIB/OPeNDap/GEMPAK)" in the IDV Dashboard, "Data 
> >>>> Choosers" tab.
> >>>>
> >>>> Best,
> >>>>
> >>>> Unidata IDV Support
> >>>>
> >>>>
> >>>> I want to use the idv package to visualize my data.
> >>>>
> >>>> I created a netcdf file. It can be read and and visualized by PANOPLY.
> >>>> But Idv does not recognize it as a netcdf file - obviously it is looking
> >>>> for something I don't have in my file.
> >>>>
> >>>> I studied your documentation and even looked at an example file. But it
> >>>> beats me what I am doing wrong.
> >>>>
> >>>> Can you help?
> >>>>
> >>>> The file is too big to attach.
> >>>>
> >>>> Here is the PANOPLY screenshot. Can you help. Thank you. - Mark
> >>>>
> >>>> address@hidden
> >>>> Chief Scientist
> >>>> Science and Technology Corporation
> >>>> 10015 Old Columbia Road
> >>>> Suite E-250
> >>>> Columbia, MD 21046
> >>>> phone: 410-309-0818 x 106
> >>>> cell: 240-481-7390
> >>>> http://markschoeberl.com
> >>>>
> >>>>
> >>>>
> >>>> I want to use the idv package to visualize my data.
> >>>>
> >>>> I created a netcdf file. It can be read and and visualized by PANOPLY.
> >>>> But Idv does not recognize it as a netcdf file - obviously it is looking
> >>>> for something I don't have in my file.
> >>>>
> >>>> I studied your documentation and even looked at an example file. But it
> >>>> beats me what I am doing wrong.
> >>>>
> >>>> Can you help?
> >>>>
> >>>> The file is too big to attach.
> >>>>
> >>>> Here is the PANOPLY screenshot. Can you help. Thank you. - Mark
> >>>>
> >>>> address@hidden
> >>>> Chief Scientist
> >>>> Science and Technology Corporation
> >>>> 10015 Old Columbia Road
> >>>> Suite E-250
> >>>> Columbia, MD 21046
> >>>> phone: 410-309-0818 x 106
> >>>> cell: 240-481-7390
> >>>> http://markschoeberl.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Ticket Details
> >>>> ===================
> >>>> Ticket ID: RZP-160177
> >>>> Department: Support IDV
> >>>> Priority: Normal
> >>>> Status: Closed
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: RZP-160177
> >>> Department: Support IDV
> >>> Priority: Normal
> >>> Status: Closed
> >>>
> >>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: RZP-160177
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed
> >
> 
> 
> 


Ticket Details
===================
Ticket ID: RZP-160177
Department: Support IDV
Priority: Normal
Status: Closed