Re: Streamable NetCDF volumes

Hi John,

Thanks for the information, I'm glad to hear that NetCDF will have a
streaming option in the long term (presuming your 5-year proposal is
approved).  As you have indicated this will make NetCDF a much better
fit for web services and on-the-fly sub-setting.

Good luck with your developments.

regards,
Chad 

On Aug 31, 2013, at 10:58 AM, John Caron <address@hidden> wrote:

> Hi Chad:
>
> Streaming netcdf is in our 5-year proposal to NSF as a bullet, under the
> strategic goal "Provide cyberinfrastructure leadership in data
> discovery, access, and use", where it says
>
>  Planned activities include:
>
>    - Creating a streaming protocol to allow use of netCDF in web
>      services, reducing the need to move large datasets to local
>      machines.
>
>    - Enhancing the C-based netCDF libraries with client support for new
>      remote access protocols such as DAP4, when it is available.
>
> The use in netCDF subsetting by coordinates is also mentioned as
> something we will support in the netCDF C-based libraries:
>
>    - Creating C-language netCDF APIs and web services that allow access
>      to local and remote data by geospatial coordinates.
>
>
>
> We expect to make progress on it in the next 5 years, but exactly
> when we will do what is unknown.
>
> Other answers are embedded:
>
>
> On 8/20/2013 12:28 PM, Chad Trabant wrote:
>>
>> Hello John,
>>
>> I work for the IRIS DMC where we manage a large archive of
>> seismological time series data.  We were recently brainstorming about
>> using NetCDF as a data delivery format.  We would want to send this
>> data to users via RESTful web services.   The issue that comes up is
>> the ability to create NetCDF in a streaming fashion, sending the data
>> to the user as it is extracted from our archive.  As you are probably
>> aware, by avoiding the creation of the entire volume prior to sending
>> to the user we 1) save resources and 2) avoid web service timeouts.
>>
>> A  bit of searching brought me to the following page and your contact
>> information:
>> http://www.unidata.ucar.edu/software/netcdf-java/stream/NcStream.html
>>
>> If you do not mind a couple of quick questions:
>>
>> 1) What is the status of support for the ncstream format?  In
>> particular, will this become a variant supported by standard NetCDF
>> libraries?
>
> yes
>
>>
>> 2) Is ncstream only a format for the wire or is this something that
>> you can save to files for later use?
>
> You can save to a file.
>
>>
>> 3) Are there alternate efforts to define a streamable NetCDF volume?
>
> not that i know of.
>
>>
>> We are not set on creating such an output format, at this point I'm
>> just trying to get a handle on what developments, if any, are headed
>> in this direction.
>>
>> Any information would be appreciated.
>>
>> regards, Chad Trabant IRIS DMC
>>
>
> Regards,
>
> John and Russ