[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[netCDF #JSG-595652]: RE: NetCDF streaming
This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
- Subject: [netCDF #JSG-595652]: RE: NetCDF streaming
- Date: Wed, 08 Sep 2010 14:04:50 -0600
Hi Malcolm, > A couple of questions for you based on your responses below. If I > wanted to replace your existing ncstream transport with a different > transport, would that be a lot of work? I'm Cc:ing John Caron, the ncstream developer, to respond to that question. > The second question concerns szip. I understand the issues with writing > szip content, and why you don't ship software that does this. Can we do > our own szip integration? How much trouble would that be? The szip > folks told me that the product is free for use on our system, so I'd > like to be able to swap out the zlib compression if szip works better. Yes, I think that should be pretty easy. Remember to build HDF5 with szip, and configure netCDF with the --with-szlib=... option to allow reading szip-compressed files, but you'll have to do the modifications to add the capability to write szip files, just following the pattern for zlib. --Russ > -----Original Message----- > From: Russ Rew [mailto:address@hidden] > Sent: Monday, August 23, 2010 4:18 PM > To: support-netcdf > Cc: Horne, Randy (Contractor); address@hidden > Subject: Re: NetCDF streaming > > > Hi Malcolm, > > Unidata makes available a different technology for low latency data > distribution, the LDM client-server software used for the Internet Data > Distribution project: > > http://www.unidata.ucar.edu/software/ldm/ > > The LDM distributes any kinds of data products over a network, using an > event-oriented architecture that pushes the data products to downstream > sites in a way that aims to minimize latency. > > We have also implemented an experimental version of an "ncstream" > on-the-wire protocol for streaming data that conforms to the Unidata > Common Data Model: > > http://www.unidata.ucar.edu/software/netcdf-java/stream/NcStream.html > > The goals of that implementation include "write optimized" data > encoding, unlike netCDF which can be characterized as better for the > reader than the writer. > > The Web service version of ncstream is "cdmremote": > > http://www.unidata.ucar.edu/software/netcdf-java/stream/CdmRemote.html > > As far as transport goes, ncstream is built on Google Protocol Buffers, > so I think those are independent of the transport layer used: > > http://code.google.com/apis/protocolbuffers/ > > We don't support writing with szip compression in our software because > of licensing issues with the SZIP compression algorithm. > > I'm also Cc:ing John Caron, the creator of ncstream and cdmremote, in > case he has anything to add or correct in what I've written above. > > --Russ > > > This is a multi-part message in MIME format. > > > > ------_=_NextPart_001_01CB42EC.124D694B > > Content-Type: text/plain; > > charset="us-ascii" > > Content-Transfer-Encoding: quoted-printable > > > > Hello, > > > > =20 > > > > I'm an engineer working on the GOES-R Ground Segment. I work with > Randy > > Horne, and I saw an earlier e-mail thread where you discussed the > NetCDF > > roadmap. I'm currently looking at mechanisms to transfer large > amounts > > of NetCDF format data between sites with low latency. Your research > in > > streaming may apply to this problem, but the information on the web > site > > is very sketchy. I'd like to know more about what you are doing, and > > see if it would benefit my application. If so maybe we could be part > of > > a beta test. > > > > =20 > > > > I'm very interested in what sort of transport you plan to provide. > > We're currently looking at both TCP and UDP multicast based options. > > We're also very interested in how this would combine with the > integrated > > szip compression feature. =20 > > > > =20 > > > > I look forward to hearing from you. > > > > =20 > > > > Malcolm McRoberts > > > > Harris Corporation > > > > (321) 984 6109 > > > > =20 > > > > > > ------_=_NextPart_001_01CB42EC.124D694B > > Content-Type: text/html; > > charset="us-ascii" > > Content-Transfer-Encoding: quoted-printable > > > > <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = > > xmlns:o=3D"urn:schemas-microsoft-com:office:office" = > > xmlns:w=3D"urn:schemas-microsoft-com:office:word" = > > xmlns:x=3D"urn:schemas-microsoft-com:office:excel" = > > xmlns:p=3D"urn:schemas-microsoft-com:office:powerpoint" = > > xmlns:a=3D"urn:schemas-microsoft-com:office:access" = > > xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" = > > xmlns:s=3D"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" = > > xmlns:rs=3D"urn:schemas-microsoft-com:rowset" > xmlns:z=3D"#RowsetSchema" = > > xmlns:b=3D"urn:schemas-microsoft-com:office:publisher" = > > xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadsheet" = > > xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" = > > xmlns:odc=3D"urn:schemas-microsoft-com:office:odc" = > > xmlns:oa=3D"urn:schemas-microsoft-com:office:activation" = > > xmlns:html=3D"http://www.w3.org/TR/REC-html40"; = > > xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/"; = > > xmlns:rtc=3D"http://microsoft.com/officenet/conferencing"; = > > xmlns:D=3D"DAV:" xmlns:Repl=3D"http://schemas.microsoft.com/repl/"; = > > xmlns:mt=3D"http://schemas.microsoft.com/sharepoint/soap/meetings/"; = > > xmlns:x2=3D"http://schemas.microsoft.com/office/excel/2003/xml"; = > > xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd"; = > > xmlns:ois=3D"http://schemas.microsoft.com/sharepoint/soap/ois/"; = > > xmlns:dir=3D"http://schemas.microsoft.com/sharepoint/soap/directory/"; > = > > xmlns:ds=3D"http://www.w3.org/2000/09/xmldsig#"; = > > xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint/dsp"; = > > xmlns:udc=3D"http://schemas.microsoft.com/data/udc"; = > > xmlns:xsd=3D"http://www.w3.org/2001/XMLSchema"; = > > > xmlns:sub=3D"http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/ > "= > > xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#"; = > > xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/"; = > > xmlns:sps=3D"http://schemas.microsoft.com/sharepoint/soap/"; = > > xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance"; = > > xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/soap"; = > > xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile"; = > > xmlns:udcp2p=3D"http://schemas.microsoft.com/data/udc/parttopart"; = > > xmlns:wf=3D"http://schemas.microsoft.com/sharepoint/soap/workflow/"; = > > xmlns:dsss=3D"http://schemas.microsoft.com/office/2006/digsig-setup"; = > > xmlns:dssi=3D"http://schemas.microsoft.com/office/2006/digsig"; = > > > xmlns:mdssi=3D"http://schemas.openxmlformats.org/package/2006/digital-si > g= > > nature" = > > > xmlns:mver=3D"http://schemas.openxmlformats.org/markup-compatibility/200 > 6= > > " xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"; = > > > xmlns:mrels=3D"http://schemas.openxmlformats.org/package/2006/relationsh > i= > > ps" xmlns:spwp=3D"http://microsoft.com/sharepoint/webpartpages"; = > > > xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/2006/types > "= > > = > > > xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/2006/messa > g= > > es" = > > > xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/SlideLibrary > /= > > " = > > > xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortalServer/Pu > b= > > lishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" = > > xmlns:st=3D"" xmlns=3D"http://www.w3.org/TR/REC-html40";> > > > > <head> > > <meta http-equiv=3DContent-Type content=3D"text/html; = > > charset=3Dus-ascii"> > > <meta name=3DGenerator content=3D"Microsoft Word 12 (filtered > medium)"> > > <style> > > <!-- > > /* Font Definitions */ > > @font-face > > {font-family:Calibri; > > panose-1:2 15 5 2 2 2 4 3 2 4;} > > /* Style Definitions */ > > p.MsoNormal, li.MsoNormal, div.MsoNormal > > {margin:0in; > > margin-bottom:.0001pt; > > font-size:11.0pt; > > font-family:"Calibri","sans-serif";} > > a:link, span.MsoHyperlink > > {mso-style-priority:99; > > color:blue; > > text-decoration:underline;} > > a:visited, span.MsoHyperlinkFollowed > > {mso-style-priority:99; > > color:purple; > > text-decoration:underline;} > > span.EmailStyle17 > > {mso-style-type:personal-compose; > > font-family:"Calibri","sans-serif"; > > color:windowtext;} > > .MsoChpDefault > > {mso-style-type:export-only;} > > @page Section1 > > {size:8.5in 11.0in; > > margin:1.0in 1.0in 1.0in 1.0in;} > > div.Section1 > > {page:Section1;} > > --> > > </style> > > <!--[if gte mso 9]><xml> > > <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> > > </xml><![endif]--><!--[if gte mso 9]><xml> > > <o:shapelayout v:ext=3D"edit"> > > <o:idmap v:ext=3D"edit" data=3D"1" /> > > </o:shapelayout></xml><![endif]--> > > </head> > > > > <body lang=3DEN-US link=3Dblue vlink=3Dpurple> > > > > <div class=3DSection1> > > > > <p class=3DMsoNormal>Hello,<o:p></o:p></p> > > > > <p class=3DMsoNormal><o:p> </o:p></p> > > > > <p class=3DMsoNormal>I’m an engineer working on the GOES-R > Ground > > Segment. I work with Randy Horne, and I saw an earlier e-mail = > > thread > > where you discussed the NetCDF roadmap. I’m currently = > > looking at mechanisms > > to transfer large amounts of NetCDF format data between sites with low > > latency. Your research in streaming may apply to this problem, > but = > > the > > information on the web site is very sketchy. I’d like to = > > know more > > about what you are doing, and see if it would benefit my = > > application. If > > so maybe we could be part of a beta test.<o:p></o:p></p> > > > > <p class=3DMsoNormal><o:p> </o:p></p> > > > > <p class=3DMsoNormal>I’m very interested in what sort of > transport = > > you plan > > to provide. We’re currently looking at both TCP and UDP = > > multicast > > based options. We’re also very interested in how this > would = > > combine > > with the integrated szip compression feature. <o:p></o:p></p> > > > > <p class=3DMsoNormal><o:p> </o:p></p> > > > > <p class=3DMsoNormal>I look forward to hearing from > you.<o:p></o:p></p> > > > > <p class=3DMsoNormal><o:p> </o:p></p> > > > > <p class=3DMsoNormal>Malcolm McRoberts<o:p></o:p></p> > > > > <p class=3DMsoNormal>Harris Corporation<o:p></o:p></p> > > > > <p class=3DMsoNormal>(321) 984 6109<o:p></o:p></p> > > > > <p class=3DMsoNormal><o:p> </o:p></p> > > > > </div> > > > > </body> > > > > </html> > > > > ------_=_NextPart_001_01CB42EC.124D694B-- > > Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: JSG-595652 Department: Support netCDF Priority: Normal Status: Closed
- Prev by Date: [netCDF #ZDR-128646]: Ncfile for cpp
- Next by Date: [Support #LRP-531622]: netCDF4 and variable length arrays in height or range
- Previous by thread: [netCDF #PSZ-497222]: Combination of nested NetCDF files
- Next by thread: [Support #LRP-531622]: netCDF4 and variable length arrays in height or range
- Index(es):