[THREDDS #QCX-921252]: Wrong dates reported by thredds 4.3

Rostislav,

I've been thinking about your question, and I'm afraid I could not come up with 
a mechanism to add the variables you mention to a netcdf subset.  
Unfortunately, for reasons to do with CF imlementation requirements, this is 
not a feature we support (though we may implement this in a future release).  
Essentially, if you were serving up individual files, it would be no problem to 
wrap the files in a bit of NcML to add the dimension and variables.  However, 
as an aggregation, the best solution I can offer is to add the information as a 
global attribute, like this:

  <dataset name="Rostislav">
    <featureCollection name="Rostislavnc4data" featureType="FMRC" 
harvest="true" path="rostislav/fmrc">
      <metadata inherited="true">
        <serviceName>all</serviceName>
        <dataFormat>netCDF</dataFormat>
      </metadata>
      <collection spec="C:/Users/madry/work/RostislavKouznetsov/.*nc4" />
      <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
        <dimension name="dayhours" value="24"/>
        <attribute name="dayhours" value="24"/>
      </netcdf>
    </featureCollection>
  </dataset>

It may be worth noting that the information you would like to add is more akin 
to metadata than actual data.

Regards,
  Lansing Madry
  Unidata
  Boulder, Colorado

> Hello Lansing,
> 
> Thank you for ther reply.
> On the second point.
> If you download a file from
> http://silam.fmi.fi/thredds/fileServer/fires/files/reanal_v1_3_global_2000.nc4
> you will find there a dimension
> dayhours = 24 ;
> and two variables
> float dayhours(dayhours) ;
> float hourlyadj(dayhours) ;
> 
> They do not appear in files that come from netcdfsubset
> http://silam.fmi.fi/thredds/ncss/grid/fires/files/reanal_v1_3_global_2000.nc4?var=emis_PM_FRP&north=70&west=0&east=20&south=50&horizStride=1&time_start=2000-01-01T00%3A00%3A00Z&time_end=2000-12-31T00%3A00%3A00Z&timeStride=1&addLatLon=true&accept=netcdf
> 
> Is it possible to make the dimension "dayhours" and correspondinf
> variables to appear in netcdf subsets?
> 
> Thank you once more.
> 
> Best regards,
> 
> Rostislav
> 
> 
> 
> On Sat, Apr 20, 2013 at 01:09:30AM +0300, Unidata THREDDS Support wrote:
> >Rostislav,
> >
> >I was able to reproduce the failure you saw when asking for NetCDF4 versus 
> >NetCDF files.  I wasn't sure why that was happening, so I asked the 
> >developer who worked on the ncss service.  He thought it was probably a bug 
> >of some sort, so I opened a ticket to address the issue.  I don't know if it 
> >is an easy fix or not yet, but it is in the queue.  Thank you for bringing 
> >this to our attention!
> >
> >https://bugtracking.unidata.ucar.edu/browse/TDS-408
> >
> >On your second point, I'm not sure what you mean by the small, non-grid 
> >variable.  When I do an ncdump on the file, I see:
> >
> >  dimensions:
> >    lon = 3600;
> >    lat = 1800;
> >    time = UNLIMITED;   // (1 currently
> >  variables:
> >    float lon(lon=3600);
> >      :long_name = "longitude";
> >      :units = "degrees_east";
> >      :standard_name = "longitude";
> >
> >    float lat(lat=1800);
> >      :units = "degrees_north";
> >      :standard_name = "latitude";
> >      :long_name = "latitude";
> >
> >    double time(time=1);
> >      :units = "seconds since 1970-01-01 00:00:00";
> >      :calendar = "standard";
> >      :_ChunkSize = 1; // int
> >
> >    float emis_PM_FRP(time=1, lat=1800, lon=3600);
> >      :units = "kg/s";
> >      :_ChunkSize = 1, 1800, 3600; // int
> >
> >  // global attributes:
> >  :CDI = "Climate Data Interface version 1.4.1";
> >  :Conventions = "CF-1.0";
> >  :netcdf_generated = "2013-04-15 09:01 UTC";
> >  :producer = "Finnish Meteorological Institute";
> >  :data_produced = "2013-04-13";
> >  :dataset = "IS4FIRES wildland-fire emission reanalysis";
> >  :contact = "Mikhail Sofiev: address@hidden";
> >  :PM2P5_10fraction = 0.25999999046325684; // double
> >  :PM2P5fraction = 0.7400000095367432; // double
> >  :quantity = "PM emission rate";
> >  :CDO = "Climate Data Operators version 1.4.1 
> > (http://www.mpimet.mpg.de/cdo)";
> >
> >Which is the non-grid variable you mean?
> >
> >On the third point, I will have to see if there is a way to get the 
> >information you want served up in fewer steps, but I don't know if it is 
> >possible off-hand.
> >
> >Regards,
> >  Lansing
> >> Hello Lansing,
> >>
> >> Thank you for the detailed reply.
> >> I have set up the service according to your instructions.
> >> http://silam.fmi.fi/thredds/firesEmission.html
> >> Few more questions.
> >>
> >> 1. The subsetting of files does not work with nc4 output
> >> when accessing by file:
> >> http://silam.fmi.fi/thredds/ncss/grid/fires/files/reanal_v1_3_global_2000.nc4?var=emis_PM_FRP&time_start=2000-01-01T00%3A00%3A00Z&time_end=2000-01-02T00%3A00%3A00Z&accept=netcdf
> >>  works, but
> >> http://silam.fmi.fi/thredds/ncss/grid/fires/files/reanal_v1_3_global_2000.nc4?var=emis_PM_FRP&time_start=2000-01-01T00%3A00%3A00Z&time_end=2000-01-02T00%3A00%3A00Z&accept=netcdf4
> >>  (difference in the last argument)
> >> causes "I/O Exception handled : NetCDF: Invalid argument
> >> nc_def_var_chunking on variable time"
> >> Is it a bug or a feature?
> >>
> >> 2. I have a small non-grid variable in my file. It disappears when
> >> accessing the files via thredds (except for http).
> >> Can i make thredds to include it  into any netcdf produced
> >> from the datasets?
> >>
> >> 3. The data are obtained from reanalisys, so they have a single
> >> time dimension. To get to the metadata page for "Fire emission this year"
> >> i have to click "Fire emission this year" and then "Best timeseries".
> >> Is it possible to avoid the second step?
> >>
> >>
> >> Thank you once more!
> >>
> >> Best regards,
> >> Rostislav
> >>
> >>
> >> On Wed, Apr 17, 2013 at 11:49:39PM +0300, Unidata THREDDS Support wrote:
> >> >Hello Rostislav,
> >> >
> >> >I downloaded a few days worth of files from your TDS and experimented 
> >> >with setting up a local TDS to help with your questions.  To answer your 
> >> >first question, what you are observing is a feature, but I think you are 
> >> >misinterpreting the data.  With a datasetScan such as you have, the time 
> >> >reported is the last time the file was modified (i.e., touched on your 
> >> >server), not the time or date information contained in the file.  To 
> >> >answer your second question, I believe what you really want to do is to 
> >> >set up a featureCollection, instead of a datasetScan.  Here is a simple 
> >> >example that I used locally:
> >> >
> >> ><?xml version="1.0" encoding="UTF-8"?>
> >> ><catalog name="THREDDS Server Default Catalog : You must change this to 
> >> >fit your server!"
> >> >        
> >> > xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
> >> >        xmlns:xlink="http://www.w3.org/1999/xlink";>
> >> >
> >> >  <service name="all" base="" serviceType="compound">
> >> >    <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
> >> >    <service name="http" serviceType="HTTPServer" 
> >> > base="/thredds/fileServer/" />
> >> >    <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
> >> >    <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> >> >    <service name="ncss" serviceType="NetcdfSubset" 
> >> > base="/thredds/ncss/grid/" />
> >> >  </service>
> >> >
> >> >  <dataset name="Rostislav">
> >> >    <featureCollection name="Rostislavnc4data" featureType="FMRC" 
> >> > harvest="true" path="rostislav/fmrc">
> >> >      <metadata inherited="true">
> >> >        <serviceName>all</serviceName>
> >> >        <dataFormat>netCDF</dataFormat>
> >> >      </metadata>
> >> >      <collection spec="C:/Users/madry/work/RostislavKouznetsov/.*nc4" />
> >> >    </featureCollection>
> >> >  </dataset>
> >> ></catalog>
> >> >
> >> >This sets your data up to be served as a single logical unit via the 
> >> >NetCDF Subset Service.  After you have changed the names and paths to 
> >> >suit your system, you will also need to edit the threddsConfig.xml file 
> >> >to turn the NetcdfSubsetService on.  You do this by uncommenting the 
> >> >relevant block of xml and setting <allow> to true:
> >> >
> >> >  <!--
> >> >  The Netcdf Subset Service is off by default.
> >> >  -->
> >> >  <NetcdfSubsetService>
> >> >    <allow>true</allow>
> >> >    <scour>10 min</scour>
> >> >    <maxAge>-1 min</maxAge>
> >> >  </NetcdfSubsetService>
> >> >
> >> >Let me know if you need any further assistance.
> >> >
> >> >Regards,
> >> >  Lansing Madry
> >> >  Unidata
> >> >  Boulder, Colorado
> >> >
> >> >> Hello,
> >> >>
> >> >> I have put a collection of daily files into thredds, one field per day.
> >> >> http://silam.fmi.fi/thredds/catalog/fires3/catalog.html
> >> >>
> >> >> Unfortunately it reports wrong times for them.  e.g.
> >> >>
> >> >> http://silam.fmi.fi/thredds/catalog/fires3/catalog.html?dataset=fires3/reanal_v1_3_global_20130410.nc4
> >> >>
> >> >> shows "date 2013-04-15T09:01:19Z (modified)" instead of something like
> >> >> 2013-04-10T00:00:00Z.
> >> >>
> >> >> The dataset is described as:
> >> >> <datasetScan name="Fires3" path="fires3"
> >> >> location="/data/thredds/Fires/"  suffix=".nc4">
> >> >> <serviceName>all</serviceName>
> >> >> </datasetScan>
> >> >>
> >> >> Is it a bug or a feature?
> >> >>
> >> >> Can I somehow configure thredds to report these files as a single
> >> >> dataset?
> >> >>
> >> >> Thank you!
> >> >>
> >> >> Best regards,
> >> >> Rostislav
> >> >>
> >> >> --
> >> >> Rostislav Kouznetsov
> >> >>
> >> >> **************************************
> >> >> Air quality
> >> >> Finnish Meteorological Institute
> >> >> PL-503 FI-00101 Helsinki Finland
> >> >> Phone: +358 29 539 4630
> >> >> **************************************
> >> >>
> >> >>
> >> >
> >> >
> >> >Ticket Details
> >> >===================
> >> >Ticket ID: QCX-921252
> >> >Department: Support THREDDS
> >> >Priority: Normal
> >> >Status: Open
> >> >
> >>
> >> --
> >> Rostislav Kouznetsov
> >>
> >> **************************************
> >> Air quality
> >> Finnish Meteorological Institute
> >> PL-503 FI-00101 Helsinki Finland
> >> Phone: +358 29 539 4630
> >> **************************************
> >>
> >>
> >
> >
> >Ticket Details
> >===================
> >Ticket ID: QCX-921252
> >Department: Support THREDDS
> >Priority: Normal
> >Status: Open
> >
> 
> --
> Rostislav Kouznetsov
> 
> **************************************
> Air quality
> Finnish Meteorological Institute
> PL-503 FI-00101 Helsinki Finland
> Phone: +358 29 539 4630
> **************************************
> 
> 


Ticket Details
===================
Ticket ID: QCX-921252
Department: Support THREDDS
Priority: High
Status: Open