[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: MODIS HDF-EOS Data in the NetCDF-Java CDM and THREDDS

  • Subject: Re: MODIS HDF-EOS Data in the NetCDF-Java CDM and THREDDS
  • Date: Tue, 26 Mar 2013 13:43:23 -0600
hope so.
note that the dateFormatMark="MOD13Q1.A#yyyyDDD" is just counting chars, then applying the java SimpleDateFormat, so the date in the filename just has to start at character 9, it doesnt have to match MOD13Q1.A 

On 3/26/2013 1:38 PM, Blodgett, David wrote:

Ahhh, so the "/" in the variable name will work to represent groups. I
didn't want to believe that would work, glad it will.

Thanks!

- Dave


On Tue, Mar 26, 2013 at 2:36 PM, John Caron <address@hidden
<mailto:address@hidden>> wrote:

    update stardate 2349873.432874

    ok, enhance actually already works, but i ran into another problem
    with joinExisting.

    So I retreated to joinNew using the kludge of extracting the date
    from the filename.

    this only works if the filenames are uniform:

    <?xml version='1.0' encoding='UTF-8'?>
    <netcdf xmlns='http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2'>
       <aggregation dimName='time' type='joinNew'>
         <variableAgg
    name='MODIS_Grid_16DAY_250m_500m_VI/Data_Fields/250m_16_days_NDVI' />
         <scan location='G:/work/blodgett/' suffix='.hdf'
    subdirs='false' dateFormatMark="MOD13Q1.A#yyyyDDD"/>
       </aggregation>
    </netcdf>

    you also have to list the variables you want aggregated explicitly;
    i only did one, you can add the others:

         <variableAgg
    name='MODIS_Grid_16DAY_250m_500m_VI/Data_Fields/250m_16_days_NDVI' />

    this form should work with the 4.3.16 release.

    let me know if this is adequate for you for now, and any problems.

    john

    ps: use this as reference:

    
http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/AnnotatedSchema4.html#aggregation



    On 3/26/2013 1:29 PM, Blodgett, David wrote:
     > ... enhance ... enhance ... go back ... enhance ...
     >
     > Love it!
     >
     > Thanks John, so no easy path forward on aggregations yet, but the
    master
     > plan includes it?
     >
     > I'm happy with that.
     >
     >
     > On Tue, Mar 26, 2013 at 1:59 PM, John Caron
    <address@hidden <mailto:address@hidden>
     > <mailto:address@hidden <mailto:address@hidden>>>
    wrote:
     >
     >     Hi David:
     >
     >     I had a look at this again. I changed HdfEosModisConvention
    to use a
     >     1D instead of a scalar time coordinate.
     >
     >     However, joinExisting isnt working because the dataset needs
    to be
     >     "enhanced" (with HdfEosModisConvention) first, before the time
     >     coordinate is available. Currently joinExisting aggregation
    doesnt
     >     do that.
     >
     >     I can probably hack that in. Really I want to get the GRID
    feature
     >     collection working, but thats going to be a few months.
    Aggregation
     >     is old and crufty.
     >
     >     Anyway, Ill let you know if i can get "enhanced" working.
     >
     >
     >     John
     >
     >
     >
     >     On 3/20/2013 7:50 AM, Blodgett, David wrote:
     >
     >         Morning John,
     >
     >         I pulled down the latest thredds war and have been
    testing a few
     >         things
     >         out. Did you get temporal aggregation working on the
    structured
     >         data at
     >         all? Any hints on how to structure the ncml for that?
     >
     >         Looks like the spatial stuff is working. Fantastic.
     >
     >         Thanks!
     >
     >         - Dave
     >
     >
     >
     >         On Mon, Mar 4, 2013 at 9:07 AM, Blodgett, David
     >         <address@hidden <mailto:address@hidden>
    <mailto:address@hidden <mailto:address@hidden>>
     >         <mailto:address@hidden <mailto:address@hidden>
    <mailto:address@hidden <mailto:address@hidden>>>> wrote:
     >
     >              OK, we'll keep an eye out for the release, thanks so
    much John!
     >
     >              No worries on the aggregation, the tiles are big enough
     >         that I think
     >              we can deal without aggregation.
     >
     >
     >              On Mon, Mar 4, 2013 at 8:43 AM, John Caron
     >         <address@hidden <mailto:address@hidden>
    <mailto:address@hidden <mailto:address@hidden>>
     >              <mailto:address@hidden
    <mailto:address@hidden>
     >         <mailto:address@hidden
    <mailto:address@hidden>>__>> wrote:
     >
     >                  On 3/1/2013 12:03 PM, Blodgett, David wrote:
     >
     >                      John,
     >
     >                      Really happy you added this stuff in, what's
    your
     >             plan for
     >                      releasing it?
     >
     >
     >                  going to try to release 4.3.16 in next week or two.
     >
     >
     >
     >                      You think there's any chance that tiled
    aggregation
     >             will work?
     >
     >
     >                  im reluctant to recommend using whats there
    currently.
     >         But doing
     >                  it right is a challenge and wont be done soon.
     >
     >
     >
     >
     >                      - Dave
     >
     >
     >                      On Tue, Feb 26, 2013 at 8:30 AM,
    address@hidden <mailto:address@hidden>
     >             <mailto:address@hidden <mailto:address@hidden>>
     >                      <mailto:address@hidden
    <mailto:address@hidden> <mailto:address@hidden
    <mailto:address@hidden>>>
     >             <address@hidden <mailto:address@hidden>
    <mailto:address@hidden <mailto:address@hidden>>
     >
     >                      <mailto:address@hidden
    <mailto:address@hidden> <mailto:address@hidden
    <mailto:address@hidden>>>>
     >             wrote:
     >
     >                          Yes the tiling scheme is consistent for
    all the
     >             MODIS tiles.
     >
     >                          Thanks,
     >
     >                          Jason Werpy
     >                          Enterprise Architect
     >                          Information Dynamics
     >                          USGS/EROS
     >                          Sioux Falls, SD 57198
     >                          phone: 605-594-2723 <tel:605-594-2723>
     >                          cell: 605-690-3576 <tel:605-690-3576>
     >                          fax: 605-594-2530 <tel:605-594-2530>
     > address@hidden <mailto:address@hidden> <mailto:address@hidden
    <mailto:address@hidden>>
     >             <mailto:address@hidden <mailto:address@hidden>
    <mailto:address@hidden <mailto:address@hidden>>>
     >
     >
     >                          NOTICE: This email may contain confidential,
     >             proprietary,
     >                          or competition
     >                          sensitive bid or proposal procurement
     >                          information. Unauthorized
     >                          disclosure of this information may carry
     >                          criminal penalties as set forth
     >                          in the Procurement Integrity Act, 41
    U.S.C. 423, as
     >                          amended. Further,
     >                          the unauthorized disclosure of certain
     >                          commercial information by civil
     >                          servants may result in fines or imprisonment
     >             under the
     >                          Trade Secrets Act
     >                          (18 U.S.C. 1905). If you have received this
     >             information in
     >                          error, please
     >                          delete it, including all copies, and
    notify the
     >             sender of
     >                          the error
     >                          immediately.
     >
     >
     >                          On Feb 26, 2013, at 8:26 AM, "Blodgett,
    David"
     >                          <address@hidden
    <mailto:address@hidden> <mailto:address@hidden
    <mailto:address@hidden>>
     >             <mailto:address@hidden
    <mailto:address@hidden> <mailto:address@hidden
    <mailto:address@hidden>>>> wrote:
     >
     >                              In this case, the data is 16 day
    from about
     >                 2000 to
     >                              present. So, on the order of 275 time
     >                 steps. There are
     >                              daily products and annual products
    though,
     >                 in wich case,
     >                              we are looking 12 to 4000+ time steps.
     >
     >                              Jason, is the tile scheme you sent over
     >                 used for all the
     >                              different resolutions? Looking at a
    few of
     >                 them it looks
     >                              like that is the case.
     >
     >                              - Dave
     >
     >
     >
     >                              On Tue, Feb 26, 2013 at 7:48 AM,
    John Caron
     >                              <address@hidden
    <mailto:address@hidden>
     >                 <mailto:address@hidden
    <mailto:address@hidden>>
     >                 <mailto:address@hidden
    <mailto:address@hidden>
     >                 <mailto:address@hidden
    <mailto:address@hidden>>__>>
     >
     >                              wrote:
     >
     >                                  Heres an image of this file.
     >
     >                                  questions about aggregations:
    how many
     >                 time steps
     >                                  will be in each aggregation ?
    how many
     >                 different
     >                                  aggregations (tiles?) will you
    serve?
     >
     >
     >
     >                                  On 2/24/2013 6:55 PM, David
    Blodgett wrote:
     >
     >                                      Sweet, thanks John. There
    was data
     >                 in the tile,
     >                                      but was hard to find...
    BORING, I
     >                 know. Sorry
     >                                      about that.
     >
     >                                      Here's a tile over the great
    lakes
     >                 to give you a
     >                                      coast to compare locations to.
     >
     >
    
http://e4ftl01.cr.usgs.gov/__MOLT/MOD13Q1.005/2000.03.05/__BROWSE.MOD13A1.A2000065.__h11v04.005.2008238024700.1.jpg
     >
    
<http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.03.05/BROWSE.MOD13A1.A2000065.h11v04.005.2008238024700.1.jpg>
     >
    
http://e4ftl01.cr.usgs.gov/__MOLT/MOD13Q1.005/2000.03.05/__MOD13Q1.A2000065.h11v04.005.__2008238031620.hdf
     >
    
<http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.03.05/MOD13Q1.A2000065.h11v04.005.2008238031620.hdf>
     >
    
http://e4ftl01.cr.usgs.gov/__MOLT/MOD13Q1.005/2000.03.05/__MOD13Q1.A2000065.h11v04.005.__2008238031620.hdf.xml
     >
    
<http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.03.05/MOD13Q1.A2000065.h11v04.005.2008238031620.hdf.xml>
     >
     >                                      - Dave
     >
     >                                      On Feb 24, 2013, at 6:33 PM,
    John
     >                 Caron wrote:
     >
     >                                          Hi:
     >
     >                                          Im hacking away at this, but
     >                 the datafiles i
     >                                          downloaded _seem_ to have
     >                 nothing but missing
     >                                          values. Or im doing
    something
     >                 wrong. Can you
     >                                          send me a file that you know
     >                 has real data in
     >                                          it. thanks
     >
     >                                          John
     >
     >                                          On 2/24/2013 9:22 AM, David
     >                 Blodgett wrote:
     >
     >                                              Hi John,
     >
     >                                              Actually, I was assuming
     >                 that the
     >                                              sinusoidal
    projection was
     >                 in there, doh!
     >                                              That was based on
     >                 gdal_translate creating
     >                                              a geotiff with this
     >                 projection info:
     >
     >                                              PROJCS["unnamed",
     >                                                   GEOGCS["Unknown
    datum
     >                 based upon the
     >                                              custom spheroid",
     >
     >                   DATUM["Not_specified_based_on___custom_spheroid",
     >
    SPHEROID["Custom
     >
      spheroid",6371007.181,0]],
     >
    PRIMEM["Greenwich",0],
     >
     >                   UNIT["degree",0.__0174532925199433]],
     >
    PROJECTION["Sinusoidal"],
     >
     >                   PARAMETER["longitude_of___center",0],
     >
     >                   PARAMETER["false_easting",0],
     >
    PARAMETER["false_northing",0],
     >                                                   UNIT["metre",1,
     >
    AUTHORITY["EPSG","9001"]]]
     >
     >                                              There's a georeferencing
     >                 section here
     >
     >                 (http://www.gdal.org/frmt___hdf4.html
     >                 <http://www.gdal.org/frmt_hdf4.html>) that
     >                                              has a bit more info
    about
     >                 gdal's
     >                                              implementation. Might be
     >                 something
     >                                              helpful there? Looks
    like
     >                 the ODL text
     >                                              you point to is
    where its
     >                 coming from.
     >
     >                                              4 and 5 are
    questions for
     >                 Jason and maybe
     >                                              Jordan. Guys?
     >
     >                                              - Dave
     >
     >
     >
     >
     >
     >                                              On Feb 23, 2013, at 1:22
     >                 PM, John Caron
     >                                              wrote:
     >
     >
     >                                                  On 2/22/2013
    2:07 PM,
     >                 David Blodgett
     >                                                  wrote:
     >
     >                                                      Welcome back
    John.
     >                 Hope you had a
     >                                                      good trip!
     >
     >                                                  thanks!
     >
     >                                                      I actually just
     >                 spent a bunch of
     >                                                      time with
    this the
     >                 other day and have
     >                                                      been meaning to
     >                 summarize what
     >                                                      I'm finding.
     >
     >                                                      The first
    issue we
     >                 are running
     >                                                      into is your #1
     >                 below. In this
     >                                                      case there
     >                                                      is an
    external xml
     >                 file that
     >                                                      contains a
    lot of
     >                 metadata, including
     >                                                      information
    about
     >                 the projection
     >
      (sinusoidal), which
     >                 is currently
     >                                                      unsupported by
     >                 NetCDF-Java?
     >
     >                                                  1) where do you get
     >                 this xml file?
     >
     >                                                  2) im looking at
     >
     >
     >                 MOD13Q1.A2000065.h00v08.005.__2008238080422.hdf
     >
     >                 MOD13Q1.A2000065.h00v08.005.__2008238080422.hdf.xml
     >
     >                                                  in your union
     >                 subdirectory. The xml
     >                                                  file does not
    seem to
     >                 have projection
     >                                                  info in it.
     >
     >                                                  3) the internal
    ODL text
     >                                                  (StructMetadata) has
     >
     >                                                  ...
     >
     >
    UpperLeftPointMtrs=(-20015109.__354000,1111950.519667)
     >
     >                 LowerRightMtrs=(-18903158.__834333,-0.000000)
     >
      Projection=GCTP_SNSOID
     >
     >                 ProjParams=(6371007.181000,0,__0,0,0,0,0,0,0,0,0,0,0)
     >
     >                 SphereCode=-1
     >
     >                                                  ...
     >
     >                                                  which may be
    enough to
     >                 calculate the
     >                                                  lat/lon values.
    Since
     >                 this is the
     >                                                  best choice, lets
     >                 investigate it
     >                                                  before we go
    down any
     >                 other paths.
     >
     >                                                  4) Can anyone track
     >                 down the code in
     >                                                  the hyrax
    handler that
     >                 does this? Or
     >                                                  does the hyrax
    handler
     >                 actually link
     >                                                  in the hds-eos
    libraries ??
     >
     >                                                  5) how confident are
     >                 you that all the
     >                                                  files that you
    want to
     >                 aggregate have
     >                                                  the same lat/lon
     >                 geolocation ?
     >
     >
     >                                                  John
     >
     >
     >
     >
     >
     >
     >
     >