[THREDDS #XPY-504678]: RE: Problem processing test NetCDF 4 files

Greetings David,

I see Dennis answered your question about file sizes.

It's hard to say whether we will have the ability to deal with IMAGE 
featureTypes
in the netCDF Subset Service by next February. I think our current plan is to
serve out GOES-R data through ADDE at first. There would be a lot of work
to be done to get the IMAGE featureType implemented at both the netCDF-Java
and TDS level, a lot of which would depend on community feedback, which takes
quite a bit of time.

Sean

> Sean,
> 
> I know that the GOES-R launch date slipped a week or so from Nov. 4 due to 
> hurricane Matthew.  That means that the Advanced Baseline Imager (ABI) 
> instrument probably will not be operational until the first couple of weeks 
> of February 2017.  Do you think this feature will be available by then?
> 
> Thanks,
> David
> 
> -----Original Message-----
> From: Unidata THREDDS Support [mailto:address@hidden]
> Sent: Thursday, October 20, 2016 6:18 PM
> To: David Huie
> Cc: David Huie; address@hidden
> Subject: [THREDDS #XPY-504678]: RE: Problem processing test NetCDF 4 files
> 
> Hi David,
> 
> Our netCDF Subset Server currently does not support cdm_data_type = Image .
> I was hoping that netCDF-Java would be able to work with the file if it 
> thought it was a grid file (which, I'm still not sure why it doesn't), but it 
> looks like this isn't the case, either. In order to support the subsetting of 
> GOES-R products, we will need to extend the netCDF Subset Service to handle 
> the Image feature type. We are currently working on getting a new version of 
> the server out - any enhancements to the server will have to be done after we 
> are finished with this push. I would estimate this will take more than a few 
> months to get to. I've created a github issue to track this problem: 
> https://github.com/Unidata/thredds/issues/660
> 
> Cheers,
> 
> Sean
> 
> 
> On Tue, Oct 11, 2016 at 11:50 AM, David Huie < address@hidden> wrote:
> 
> > New Client Reply: RE: Problem processing test NetCDF 4 files
> >
> > Sean,
> >
> > We were able to overcome the error we were getting by putting those
> > lines you suggested in our catalog inside dataset element, but results
> > we are getting when we access the files using ncss are not correct.
> > Even when we try to do subsetting by using the coords found in file
> > manually (by using
> > NCDump) and time range, only we get is 5kb empty file. Both screenshot
> > and subsetted file are included in attachment.
> >
> > What are you suggesting to do next in order to read and subset the
> > NOAA files correctly?
> >
> > I'm not sure are you the right person to ask, but question that arise
> > here is, is NetCDF library capable of subsetting the data files from
> > GOES-R satellite? The files we are using are sample data downloaded from 
> > them.
> >
> > Thanks in advance.
> >
> > David
> >
> >
> >
> > 2016-10-10 16:31 GMT+02:00 David Huie <address@hidden
> > <mailto:address@hidden>>:
> > Vasil,
> >
> > Can you do what Sean suggests and then try using the subsetting
> > services and see if they work?
> >
> > David
> >
> > -----Original Message-----
> > From: Unidata THREDDS Support [mailto:address@hidden
> > <mailto:address@hidden>]
> > Sent: Thursday, October 06, 2016 4:52 PM
> > To: David Huie
> > Cc: address@hidden<mailto:address@hidden>
> > Subject: [THREDDS #XPY-504678]: RE: Problem processing test NetCDF 4
> > files
> >
> > Hi David,
> >
> > The issue is that the data type is set to "Image" inside of each of
> > the data files, and the netCDF Subset Service does not currently work
> > with all Image data types. However, if you add the following to the
> > catalog (inside the datasetScan element):
> >
> >     <netcdf
> > xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2
> > ">
> >       <attribute name="cdm_data_type" value="Grid"/>
> >     </netcdf>
> >
> > and you should be able to get the subset service to work on those files.
> >
> > Cheers,
> >
> > Sean
> >
> > > Sean,
> > >
> > > Here you go..
> > >
> > > David
> > >
> > > From: Vasil Savovski [mailto:address@hidden<mailto:vasil.
> > address@hidden>]
> > > Sent: Thursday, October 06, 2016 2:26 PM
> > > To: David Huie
> > > Subject: Re: [THREDDS #XPY-504678]: RE: Problem processing test
> > > NetCDF
> > > 4 files
> > >
> > > This is my local catalog. Explain that files are going to be
> > > constantly
> > added to the folder, thus we use datasetScan.
> > >
> > > these two parts are important:
> > >
> > > <service name="all" base="" serviceType="compound"> <service
> > > name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" /> <service
> > > name="dap4" serviceType="DAP4" base="/thredds/dap4/" /> <service
> > > name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> > > <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
> > > <!--service name="wms" serviceType="WMS" base="/thredds/wms/" /-->
> > > <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/"
> > > /> </service>
> > >
> > > ...
> > >
> > > <datasetScan name="Blue Sky Test Files" ID="blueSkyTestDatasetScan"
> > > path="blueSkyAll" location="content/dataset">
> > >
> > > <metadata inherited="true">
> > > <serviceName>all</serviceName>
> > > <dataType>Grid</dataType>
> > > </metadata>
> > >
> > > <filter>
> > > <include wildcard="OR_ABI-L1b-RadC-*.nc"/> <include
> > > wildcard="OR_ABI-L1b-RadF-*.nc"/> </filter> </datasetScan>
> > >
> > >
> > > 2016-10-06 20:22 GMT+02:00 Vasil Savovski
> > > <address@hidden<
> > mailto:address@hidden><mailto:address@hidden
> > <mailto:address@hidden>>>:
> > >
> > > ---------- Forwarded message ----------
> > > From: Unidata THREDDS Support <address@hidden<mailto:
> > address@hidden><mailto:address@hidden.
> > edu< mailto:address@hidden>>>
> > > Date: 2016-10-06 20:03 GMT+02:00
> > > Subject: [THREDDS #XPY-504678]: RE: Problem processing test NetCDF 4
> > files
> > > To:
> > > address@hidden<mailto:address@hidden><
> > mailto:address@hidden<mailto:dhuie@blueskydataservices.
> > com
> > >>
> > > Cc: address@hidden<mailto:address@hidden
> > ><mailto:address@hidden<mailto:address@hidden>
> > >>
> > >
> > >
> > > David,
> > >
> > > Could you send a copy of the configuration catalog you are using to
> > serve the data?
> > >
> > > Thanks!
> > >
> > > Sean
> > >
> > > > Ryan,
> > > >
> > > > We did that…here is a screenshot of what we see.  What next?
> > > >
> > > > David
> > > >
> > > > From: Vasil Savovski [mailto:address@hidden<mailto:vasil.
> > address@hidden><mailto:address@hidden<mailto:
> > address@hidden>>]
> > > > Sent: Thursday, October 06, 2016 12:11 PM
> > > > To: David Huie
> > > > Subject: Fwd: [THREDDS #XPY-504678]: RE: Problem processing test
> > NetCDF 4 files
> > > >
> > > >
> > > > ---------- Forwarded message ----------
> > > > From: Unidata THREDDS Support <address@hidden
> > <mailto:address@hidden><mailto:suppo
> > address@hidden<mailto:address@hidden
> > >><mailto:address@hidden<mailto:support-thredds@
> > unidata.ucar.edu><mailto:address@hidden<mailto:
> > address@hidden>>>>
> > > > Date: 2016-10-06 18:03 GMT+02:00
> > > > Subject: [THREDDS #XPY-504678]: RE: Problem processing test NetCDF
> > > > 4
> > files
> > > > To:
> > > > address@hidden<mailto:address@hidden
> > ><mailto:address@hidden<mailto:dhuie@
> > blueskydataservices.com>><mailto:address@hidden<mailto:
> > address@hidden><mailto:address@hidden<
> > mailto:address@hidden>>>
> > > > Cc: address@hidden<mailto:address@hidden
> > ><mailto:address@hidden<mailto:address@hidden
> > >><mailto:address@hidden<mailto:address@hidden
> > ><mailto:address@hidden<mailto:address@hidden>
> > >>>,
> > address@hidden<mailto:address@hidden.e
> > du
> > ><mailto:address@hidden<mailto:support-thredds@
> > unidata.ucar.edu>><mailto:address@hidden<mailto:
> > address@hidden><mailto:address@hidden.
> > edu< mailto:address@hidden>>>
> > > >
> > > >
> > > > David,
> > > >
> > > > My guess is that netCDF-java can't make sense of the coordinate
> > > > system
> > in the files, which is a prerequisite for the NetCDF Subset Service
> > (NCSS) being able to understand them.
> > > >
> > > > You can test this theory by opening the files under the "CoordSys"
> > > > tab
> > in ToolsUI and seeing
> > > >
> > > > Ryan
> > > >
> > > >
> > > > > Dear THREDDS support team,
> > > > >
> > > > > We sent this support request to you back in September and
> > > > > haven't
> > heard back from anyone so we are resending.  Any help you could
> > provide would be most appreciated.  Thank you.
> > > > >
> > > > > David Huie
> > > > >
> > > > > From: David Huie
> > > > > Sent: Friday, September 02, 2016 9:43 AM
> > > > > To: 'address@hidden<mailto:support-thredds@
> > unidata.ucar.edu><mailto:address@hidden<mailto:
> > address@hidden>><mailto:address@hidden
> > .edu <mailto:address@hidden><mailto:suppo
> > address@hidden<mailto:address@hidden>>>'
> > > > > Cc: address@hidden<mailto:address@hidden
> > ><mailto:address@hidden<mailto:address@hidden
> > >><mailto:address@hidden<mailto:address@hidden
> > ><mailto:address@hidden<mailto:address@hidden>
> > >>>
> > > > > Subject: Problem processing test NetCDF 4 files
> > > > >
> > > > > Dear THREDDS support team,
> > > > >
> > > > > We have been trying to set up NetCDF Subset Service of our test
> > THREDDS server, and although it does work with test files provided by
> > TDS installation, we have no luck processing the files from NOAA or SSEC.
> > > > >
> > > > > Test files from NOAA are downloaded from here:
> > http://www.goes-r.gov/products/data/ABI-L1b-ContFullDisk.html
> > > > >
> > > > > while SSEC files can be founded here: https://dl.dropboxusercontent .
> > com/u/53125944/Sample%20GOES%20files%20from%20SSEC%20--%
> > 20cspp-geo-grb-v0.3-referenceoutput.tar
> > > > >
> > > > > If you don't want to download the whole archive, and in order to
> > reproduce this locally, one of the test files we have trouble
> > processing can be downloaded from here: https://www.dropbox.com/s/
> > a84158cee2qvlv4/OR_ABI-L1b-RadF-M4C04_G16_s20161811455312_e20161811500
> > 122_
> > c20161811500177.nc?dl=0
> > > > >
> > > > > Error we are constantly getting:
> > > > >
> > > > > **Failed to find FeatureDatasetFactory for=
> > /usr/local/tds/apache-tomcat-8.5.4/content/thredds/public/
> > dataset/FD/OR_ABI-L1b-RadF-M4C04_G16_s20161811455312_e20161811500122_
> > c20161811500177.nc<http://c20161811500177.nc><http://c20161811500177.n
> > c>< http://c20161811500177.nc> datatype=IMAGE
> > > > >
> > > > > NCSS is enabled in threddsConfing.xml, we even tried compiling
> > netcdf library though we don't intend to write our own netcdf files.
> > > > >
> > > > > When opened with NetCDF ToolsUI program, test file supplied
> > > > > above
> > can be successfully verified, and it's indeed gridded dataset.
> > > > >
> > > > > Any help would be appreciated,
> > > > > Thanks in advance.
> > > > >
> > > > > Regards,
> > > > > David Huie
> > > > >
> > > > >
> > > > >
> > > > > [Blue Sky transparent logo -- small size] President Blue Sky
> > > > > Data Services, LLC
> > > > > 137 S. Courtenay Blvd., No. 615
> > > > > Merritt Island, FL  32952
> > > > >
> > > > > Tel:  321-271-1155<tel:321-271-1155><tel:321-271-1155<tel:
> > 321-271-1155>>
> > > > > E-mail:  address@hidden<mailto:dhuie@
> > blueskydataservices.com><mailto:address@hidden<mailto:
> > address@hidden>><mailto:address@hidden<
> > mailto:address@hidden><mailto:dhuie@blueskydataservices
> > .com <mailto:address@hidden>>><mailto:dhuie@
> > blueskydataservices.com<mailto:address@hidden><mailto:
> > address@hidden<mailto:address@hidden>><
> > mailto:address@hidden<mailto:dhuie@blueskydataservices.
> > com
> > ><mailto:address@hidden<mailto:dhuie@
> > blueskydataservices.com>>>>
> > > > > Website:  www.blueskydataservices.com<ht
> > tp://www.blueskydataservices.com><http://www.blueskydataservices.com><
> > http://www.blueskydataservices.com><http://www.blueskydataservices.com
> > />
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: XPY-504678
> > > > Department: Support THREDDS
> > > > Priority: Normal
> > > > Status: Open
> > > > ===================
> > > > NOTE: All email exchanges with Unidata User Support are recorded
> > > > in
> > the Unidata inquiry tracking system and then made publicly available
> > through the web.  If you do not want to have your interactions made
> > available in this way, you must let us know in each email you send to us.
> > > >
> > > >
> > > >
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: XPY-504678
> > > Department: Support THREDDS
> > > Priority: Normal
> > > Status: Open
> > > ===================
> > > NOTE: All email exchanges with Unidata User Support are recorded in
> > > the
> > Unidata inquiry tracking system and then made publicly available
> > through the web.  If you do not want to have your interactions made
> > available in this way, you must let us know in each email you send to us.
> > >
> > >
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: XPY-504678
> > Department: Support THREDDS
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in
> > the Unidata inquiry tracking system and then made publicly available
> > through the web.  If you do not want to have your interactions made
> > available in this way, you must let us know in each email you send to us.
> >
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: XPY-504678
> > Department: Support THREDDS
> > Priority: Normal
> > Status: Open
> > Link:  https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=
> > tickets&_a=viewticket&ticketid=27586
> >
> >
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: XPY-504678
> Department: Support THREDDS
> Priority: High
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 

Ticket Details
===================
Ticket ID: XPY-504678
Department: Support THREDDS
Priority: High
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.