On 11/25/2013 1:53 PM, Roy Mendelssohn - NOAA Federal wrote:
Yes they are not understanding the CF DSG convention, but the point of a possible alternate way to do it is valid. The problem from our point of view is just that, there a lot of alternate ways to do it in netcdf4, and if we are going to write we would rather it to be the agreed upon way (right after we agree on 'groups' - sorry couldn't resist)
yep
And writing to the netcdf4 C library means writing to hdf5 library which means the user (the user in this case being someone who is installing an ERDDAP instance) has to install all of that, and down the road you can start having issue with library versions and locations, and we would have to figure out what is going on. While right now, it is all in the war file.
yep
But for now, the CF DSG defined in netcdf4 would be a start, even if the CF people don't agree to it (but perhaps the NASA folks will). I have purposefully tried to keep this discussion to a small group, because I saw the previous discussions having to do with CF and anything to do with netcdf4, and I can do without the flames, thank you very much. I would add that one way to take the comments I set along from the user is that they see a way that exists in netcdf4 that would make the representation of the data more useful to them. Now you can't just change standards for every person who has an idea of what would be more useful, but the same time usage (as with groups) may get way ahead of the standard, which would make the standard more and more irrelevant.
no way around it i think. CF is not going to figure this out. we can only do it, then submit as possible standard.ill just say that the CDM handles lots of formats and conventions that arent CF.
look at the bright side, you could call it "Roy's Conventions" and then reap the glory!
I think on server side its not so bad, and Unidata is willing to support questions and problems using it. However, you are right that CDM is not yet supporting full access to netcdf-4 features. Specific feature requests would help prioritize development there.-Roy On Nov 25, 2013, at 12:24 PM, John Caron <address@hidden> wrote:Hi Roy: On 11/25/2013 9:53 AM, Roy Mendelssohn - NOAA Federal wrote:Hi All: For some background, here are snippets from a user who is accessing the CalCOFI data (stations that are sampled by ship for a variety of parameters) through ERDDAP, and getting back netcdf files that follow the CF DSG specs.I have a question for you. I am playing with the SIO hydrographic bottle data, and did a test extract to examine the variables in a downloaded netcdf file (both nc and ncCF formats). What is puzzling me is that the dimensionality of the variable for measured oxygen, for example, is 1 rather than 4 (i.e. ) time, lat, long, depth. ncdump shows three coordinates, as specified in the header, implying that the dimension is 3, not 1:float o2ml_l(row) ; o2ml_l:actual_range = 0.f, 9.44f ; o2ml_l:coordinates = "time latitude longitude" ; o2ml_l:ioos_category = "Dissolved O2" ; o2ml_l:long_name = "Measured Oxygen" ; o2ml_l:units = "ml/L of seawater" ; not sure that i understand the problem. Is s/he just asking to understand the DSG convention? If so, its the usual confusion about dimensions vs coordinates.andNetCDF4 allows multiple unlimited dimensions (which in this case would be time and depth coordinates). The netCDF4 data are then read as a ragged array and gridding can be done easily because the dimensionality of the variables is known. Essentially the data become 4-dimensional semi-random point data rather than a regular array, and the plotting functions in NCL or other program can handle this.There are several things that come up in relation to this, that I thought you folks might provide some guidance: 1. We have been reluctant to provide hdf5 writes in ERDDAP because there is no pure Java way to do so. Many of the Java/C interfaces only work so-so, but even more the chance for error and support nightmares in making sure we can connect with the appropriate libraries is not something we really want to do. Is there anything on the horizon for a pure Java write interface - we do not need everything hdf5 can do, basically we need to be able to write netcdf4 files.CDM supports netcdf-4 writing using the netcdf C library: http://www.unidata.ucar.edu/software/thredds/v4.4/netcdf-java/reference/netcdf4Clibrary.html
IMO there will not be a pure java netcdf-4 writing library, at least not from Unidata and I doubt anyone else will be willing to spend the significant resources on it; THG has not and they would be the only other obvious group.2. hdf5 (even netcdf4) is a wide open format unless there are some standards on how things are to be written. The best that we can tell, the CF DSG are only defined in terms of netcdf3 (are we are incorrect on this?), and some of you had the rude awakening to what happens when you propose to try to do anything that keeps up with some changing technology. So even if there aren't formally approved standards for the type of write mentioned above, are there at least proposed and public available "standards" that we can both use and point users to?correct, CF is netcdf-3 only and its hard to get CF interested in netcdf4. My guess is that "we" will just do it, and later it will become part of CF. In a way thats better anyway, standards should arguably endorse proven technology not invent new ones. I would like to work on trying netcdf-4 to store point obs better, mulitple unlimited dimensions being one such possibility. If you want to experiment with that in ERDAP, I will do my best to collaborate. As usual, time is limited. We have 2 new hires coming on board in the next month and Im hoping that will give more time. Regards, JohnThanks -Roy ********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097 e-mail: address@hidden (Note new e-mail address) voice: (831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097 e-mail: address@hidden (Note new e-mail address) voice: (831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected" "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.