Re: netcdf4, java and standards

Yes they are not understanding the CF DSG convention, but the point of a
possible alternate way to do it is valid.  The problem from our point of
view is just that, there a lot of alternate ways to do it in netcdf4,
and if we are going to write we would rather it to be the agreed upon
way  (right after we agree on 'groups' - sorry couldn't resist)

And writing to the netcdf4 C library means writing to hdf5 library which
means the user  (the user in this case being someone who is installing
an ERDDAP instance) has to install all of that, and down the road you
can start having issue with library versions and locations, and we would
have to figure out what is going on.  While right now, it is all in the
war file.  


But for now, the CF DSG defined in netcdf4 would be a start, even if the
CF people don't agree to it (but perhaps the NASA folks will).  I have
purposefully tried to keep this discussion to a small group, because I
saw the previous  discussions having to do with CF and anything to do
with netcdf4, and I can do without the flames, thank you very much.  I
would add that one way to take the comments I set along from the user is
that they see a way that exists in netcdf4 that would make the
representation of the data more useful to them.  Now you can't just
change standards for every person who has an idea of what would be more
useful, but the same time usage (as with groups) may get way ahead of
the standard, which would make the standard more and more irrelevant.

-Roy



On Nov 25, 2013, at 12:24 PM, John Caron <address@hidden> wrote:

> Hi Roy:
> 
> 
> On 11/25/2013 9:53 AM, Roy Mendelssohn - NOAA Federal wrote:
>> Hi All:
>> 
>> For some background, here are snippets  from a user who is accessing
>> the CalCOFI data  (stations that are sampled by ship for a variety of
>> parameters) through ERDDAP, and getting back netcdf files that follow
>> the CF DSG specs.
>> 
>> 
>>>> I have a question for you. I am playing with the SIO hydrographic
>>>> bottle data, and did a test extract to examine the variables in a
>>>> downloaded netcdf file (both nc and ncCF formats).
>>>> 
>>>> What is puzzling me is that the dimensionality of the variable
>>>> for measured oxygen, for example, is 1 rather than 4 (i.e. )
>>>> time, lat, long, depth.
>>>> 
>>>> ncdump shows three coordinates, as specified in the header,
>>>> implying that the dimension is 3, not 1:
> 
>    float o2ml_l(row) ;
>       o2ml_l:actual_range = 0.f, 9.44f ;
>        o2ml_l:coordinates = "time latitude longitude" ;
>        o2ml_l:ioos_category = "Dissolved O2" ;
>        o2ml_l:long_name = "Measured Oxygen" ;
>        o2ml_l:units = "ml/L of seawater" ;
> 
> not sure that i understand the problem. Is s/he just asking to understand the 
> DSG convention?
> 
> If so, its the usual confusion about dimensions vs coordinates.
> 
>> 
>> and
>> 
>>> NetCDF4 allows multiple unlimited dimensions (which in this case
>>> would be time and depth coordinates). The netCDF4 data are then
>>> read as a ragged array and gridding can be done easily because the
>>> dimensionality of the variables is known. Essentially the data
>>> become 4-dimensional semi-random point data rather than a regular
>>> array, and the plotting functions in NCL or other program can
>>> handle this.
>> 
>> 
>> There are several things that come up in relation to this, that I
>> thought you folks might provide some guidance:
>> 
>> 1.  We have been reluctant to provide hdf5 writes in ERDDAP because
>> there is no pure Java way to do so.  Many of the Java/C interfaces
>> only work so-so, but even more the chance for error and support
>> nightmares in making sure we can connect with the appropriate
>> libraries is not something we really want to do.  Is there anything
>> on the horizon for a pure Java write interface  - we do not need
>> everything  hdf5 can do, basically we need to be able to write
>> netcdf4 files.
> 
> CDM supports netcdf-4 writing using the netcdf C library:
> 
> http://www.unidata.ucar.edu/software/thredds/v4.4/netcdf-java/reference/netcdf4Clibrary.html
> 
> I think on server side its not so bad, and Unidata is willing to support 
> questions and problems using it. However, you are right that CDM is not yet 
> supporting full access to netcdf-4 features. Specific feature requests would 
> help prioritize development there.
> 
> IMO there will not be a pure java netcdf-4 writing library, at least not from 
> Unidata and I doubt anyone else will be willing to spend the significant 
> resources on it; THG has not and they would be the only other obvious group.
> 
> 
>> 
>> 2.  hdf5 (even netcdf4) is a wide open format unless there are some
>> standards on how things are to be written.  The best  that we can
>> tell, the CF DSG  are only defined in terms of netcdf3  (are we are
>> incorrect on this?), and some of you had the rude awakening to what
>> happens when you propose to try to do anything that keeps up with
>> some changing technology.  So even if there aren't formally approved
>> standards for the type of write mentioned above, are there at least
>> proposed and public available "standards" that we can both use and
>> point users to?
> 
> correct, CF is netcdf-3 only and its hard to get CF interested in netcdf4. My 
> guess is that "we" will just do it, and later it will become part of CF. In a 
> way thats better anyway, standards should arguably endorse proven technology 
> not invent new ones.
> 
> I would like to work on trying netcdf-4 to store point obs better, mulitple 
> unlimited dimensions being one such possibility. If you want to experiment 
> with that in ERDAP, I will do my best to collaborate. As usual, time is 
> limited. We have 2 new hires coming on board in the next month and Im hoping 
> that will give more time.
> 
> Regards,
> John
> 
>> 
>> Thanks
>> 
>> -Roy
>> 
>> 
>> ********************** "The contents of this message do not reflect
>> any position of the U.S. Government or NOAA." **********************
>> Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS
>> Environmental Research Division Southwest Fisheries Science Center
>> 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097
>> 
>> e-mail: address@hidden (Note new e-mail address) voice:
>> (831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/
>> 
>> "Old age and treachery will overcome youth and skill." "From those
>> who have been given much, much will be expected" "the arc of the
>> moral universe is long, but it bends toward justice" -MLK Jr.
>> 

**********************
"The contents of this message do not reflect any position of the U.S. 
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.