[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[netCDF #JSG-595652]: RE: NetCDF streaming
- Subject: [netCDF #JSG-595652]: RE: NetCDF streaming
- Date: Wed, 08 Sep 2010 14:04:50 -0600
Hi Malcolm,
> A couple of questions for you based on your responses below. If I
> wanted to replace your existing ncstream transport with a different
> transport, would that be a lot of work?
I'm Cc:ing John Caron, the ncstream developer, to respond to that question.
> The second question concerns szip. I understand the issues with writing
> szip content, and why you don't ship software that does this. Can we do
> our own szip integration? How much trouble would that be? The szip
> folks told me that the product is free for use on our system, so I'd
> like to be able to swap out the zlib compression if szip works better.
Yes, I think that should be pretty easy. Remember to build HDF5 with
szip, and configure netCDF with the --with-szlib=... option to allow
reading szip-compressed files, but you'll have to do the modifications
to add the capability to write szip files, just following the pattern
for zlib.
--Russ
> -----Original Message-----
> From: Russ Rew [mailto:address@hidden]
> Sent: Monday, August 23, 2010 4:18 PM
> To: support-netcdf
> Cc: Horne, Randy (Contractor); address@hidden
> Subject: Re: NetCDF streaming
>
>
> Hi Malcolm,
>
> Unidata makes available a different technology for low latency data
> distribution, the LDM client-server software used for the Internet Data
> Distribution project:
>
> http://www.unidata.ucar.edu/software/ldm/
>
> The LDM distributes any kinds of data products over a network, using an
> event-oriented architecture that pushes the data products to downstream
> sites in a way that aims to minimize latency.
>
> We have also implemented an experimental version of an "ncstream"
> on-the-wire protocol for streaming data that conforms to the Unidata
> Common Data Model:
>
> http://www.unidata.ucar.edu/software/netcdf-java/stream/NcStream.html
>
> The goals of that implementation include "write optimized" data
> encoding, unlike netCDF which can be characterized as better for the
> reader than the writer.
>
> The Web service version of ncstream is "cdmremote":
>
> http://www.unidata.ucar.edu/software/netcdf-java/stream/CdmRemote.html
>
> As far as transport goes, ncstream is built on Google Protocol Buffers,
> so I think those are independent of the transport layer used:
>
> http://code.google.com/apis/protocolbuffers/
>
> We don't support writing with szip compression in our software because
> of licensing issues with the SZIP compression algorithm.
>
> I'm also Cc:ing John Caron, the creator of ncstream and cdmremote, in
> case he has anything to add or correct in what I've written above.
>
> --Russ
>
> > This is a multi-part message in MIME format.
> >
> > ------_=_NextPart_001_01CB42EC.124D694B
> > Content-Type: text/plain;
> > charset="us-ascii"
> > Content-Transfer-Encoding: quoted-printable
> >
> > Hello,
> >
> > =20
> >
> > I'm an engineer working on the GOES-R Ground Segment. I work with
> Randy
> > Horne, and I saw an earlier e-mail thread where you discussed the
> NetCDF
> > roadmap. I'm currently looking at mechanisms to transfer large
> amounts
> > of NetCDF format data between sites with low latency. Your research
> in
> > streaming may apply to this problem, but the information on the web
> site
> > is very sketchy. I'd like to know more about what you are doing, and
> > see if it would benefit my application. If so maybe we could be part
> of
> > a beta test.
> >
> > =20
> >
> > I'm very interested in what sort of transport you plan to provide.
> > We're currently looking at both TCP and UDP multicast based options.
> > We're also very interested in how this would combine with the
> integrated
> > szip compression feature. =20
> >
> > =20
> >
> > I look forward to hearing from you.
> >
> > =20
> >
> > Malcolm McRoberts
> >
> > Harris Corporation
> >
> > (321) 984 6109
> >
> > =20
> >
> >
> > ------_=_NextPart_001_01CB42EC.124D694B
> > Content-Type: text/html;
> > charset="us-ascii"
> > Content-Transfer-Encoding: quoted-printable
> >
> > <html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
> > xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
> > xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
> > xmlns:x=3D"urn:schemas-microsoft-com:office:excel" =
> > xmlns:p=3D"urn:schemas-microsoft-com:office:powerpoint" =
> > xmlns:a=3D"urn:schemas-microsoft-com:office:access" =
> > xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" =
> > xmlns:s=3D"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" =
> > xmlns:rs=3D"urn:schemas-microsoft-com:rowset"
> xmlns:z=3D"#RowsetSchema" =
> > xmlns:b=3D"urn:schemas-microsoft-com:office:publisher" =
> > xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadsheet" =
> > xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" =
> > xmlns:odc=3D"urn:schemas-microsoft-com:office:odc" =
> > xmlns:oa=3D"urn:schemas-microsoft-com:office:activation" =
> > xmlns:html=3D"http://www.w3.org/TR/REC-html40"; =
> > xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/"; =
> > xmlns:rtc=3D"http://microsoft.com/officenet/conferencing"; =
> > xmlns:D=3D"DAV:" xmlns:Repl=3D"http://schemas.microsoft.com/repl/"; =
> > xmlns:mt=3D"http://schemas.microsoft.com/sharepoint/soap/meetings/"; =
> > xmlns:x2=3D"http://schemas.microsoft.com/office/excel/2003/xml"; =
> > xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd"; =
> > xmlns:ois=3D"http://schemas.microsoft.com/sharepoint/soap/ois/"; =
> > xmlns:dir=3D"http://schemas.microsoft.com/sharepoint/soap/directory/";
> =
> > xmlns:ds=3D"http://www.w3.org/2000/09/xmldsig#"; =
> > xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint/dsp"; =
> > xmlns:udc=3D"http://schemas.microsoft.com/data/udc"; =
> > xmlns:xsd=3D"http://www.w3.org/2001/XMLSchema"; =
> >
> xmlns:sub=3D"http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/
> "=
> > xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#"; =
> > xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/"; =
> > xmlns:sps=3D"http://schemas.microsoft.com/sharepoint/soap/"; =
> > xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance"; =
> > xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/soap"; =
> > xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile"; =
> > xmlns:udcp2p=3D"http://schemas.microsoft.com/data/udc/parttopart"; =
> > xmlns:wf=3D"http://schemas.microsoft.com/sharepoint/soap/workflow/"; =
> > xmlns:dsss=3D"http://schemas.microsoft.com/office/2006/digsig-setup"; =
> > xmlns:dssi=3D"http://schemas.microsoft.com/office/2006/digsig"; =
> >
> xmlns:mdssi=3D"http://schemas.openxmlformats.org/package/2006/digital-si
> g=
> > nature" =
> >
> xmlns:mver=3D"http://schemas.openxmlformats.org/markup-compatibility/200
> 6=
> > " xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"; =
> >
> xmlns:mrels=3D"http://schemas.openxmlformats.org/package/2006/relationsh
> i=
> > ps" xmlns:spwp=3D"http://microsoft.com/sharepoint/webpartpages"; =
> >
> xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/2006/types
> "=
> > =
> >
> xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/2006/messa
> g=
> > es" =
> >
> xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/SlideLibrary
> /=
> > " =
> >
> xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortalServer/Pu
> b=
> > lishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" =
> > xmlns:st=3D"" xmlns=3D"http://www.w3.org/TR/REC-html40";>
> >
> > <head>
> > <meta http-equiv=3DContent-Type content=3D"text/html; =
> > charset=3Dus-ascii">
> > <meta name=3DGenerator content=3D"Microsoft Word 12 (filtered
> medium)">
> > <style>
> > <!--
> > /* Font Definitions */
> > @font-face
> > {font-family:Calibri;
> > panose-1:2 15 5 2 2 2 4 3 2 4;}
> > /* Style Definitions */
> > p.MsoNormal, li.MsoNormal, div.MsoNormal
> > {margin:0in;
> > margin-bottom:.0001pt;
> > font-size:11.0pt;
> > font-family:"Calibri","sans-serif";}
> > a:link, span.MsoHyperlink
> > {mso-style-priority:99;
> > color:blue;
> > text-decoration:underline;}
> > a:visited, span.MsoHyperlinkFollowed
> > {mso-style-priority:99;
> > color:purple;
> > text-decoration:underline;}
> > span.EmailStyle17
> > {mso-style-type:personal-compose;
> > font-family:"Calibri","sans-serif";
> > color:windowtext;}
> > .MsoChpDefault
> > {mso-style-type:export-only;}
> > @page Section1
> > {size:8.5in 11.0in;
> > margin:1.0in 1.0in 1.0in 1.0in;}
> > div.Section1
> > {page:Section1;}
> > -->
> > </style>
> > <!--[if gte mso 9]><xml>
> > <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
> > </xml><![endif]--><!--[if gte mso 9]><xml>
> > <o:shapelayout v:ext=3D"edit">
> > <o:idmap v:ext=3D"edit" data=3D"1" />
> > </o:shapelayout></xml><![endif]-->
> > </head>
> >
> > <body lang=3DEN-US link=3Dblue vlink=3Dpurple>
> >
> > <div class=3DSection1>
> >
> > <p class=3DMsoNormal>Hello,<o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p> </o:p></p>
> >
> > <p class=3DMsoNormal>I’m an engineer working on the GOES-R
> Ground
> > Segment. I work with Randy Horne, and I saw an earlier e-mail =
> > thread
> > where you discussed the NetCDF roadmap. I’m currently =
> > looking at mechanisms
> > to transfer large amounts of NetCDF format data between sites with low
> > latency. Your research in streaming may apply to this problem,
> but =
> > the
> > information on the web site is very sketchy. I’d like to =
> > know more
> > about what you are doing, and see if it would benefit my =
> > application. If
> > so maybe we could be part of a beta test.<o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p> </o:p></p>
> >
> > <p class=3DMsoNormal>I’m very interested in what sort of
> transport =
> > you plan
> > to provide. We’re currently looking at both TCP and UDP =
> > multicast
> > based options. We’re also very interested in how this
> would =
> > combine
> > with the integrated szip compression feature. <o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p> </o:p></p>
> >
> > <p class=3DMsoNormal>I look forward to hearing from
> you.<o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p> </o:p></p>
> >
> > <p class=3DMsoNormal>Malcolm McRoberts<o:p></o:p></p>
> >
> > <p class=3DMsoNormal>Harris Corporation<o:p></o:p></p>
> >
> > <p class=3DMsoNormal>(321) 984 6109<o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p> </o:p></p>
> >
> > </div>
> >
> > </body>
> >
> > </html>
> >
> > ------_=_NextPart_001_01CB42EC.124D694B--
>
>
Russ Rew UCAR Unidata Program
address@hidden http://www.unidata.ucar.edu
Ticket Details
===================
Ticket ID: JSG-595652
Department: Support netCDF
Priority: Normal
Status: Closed