[IDV #XYY-917880]: ERA_Interim Data

> Yuan,
> 
> I contacted the people where I got the data from, and this is their
> response:
> 
> Hi Stefan,


Stefan,
   It is very clear that the ensemble information is in the dataset, here is 
what you can get from the wgrib:

1:0:d=10091218:PV:kpds5=60:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
2:262252:d=10091218:Z:kpds5=129:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
3:524504:d=10091218:T:kpds5=130:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
4:786756:d=10091218:Q:kpds5=133:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
5:1049008:d=10091218:W:kpds5=135:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
6:1311260:d=10091218:VO:kpds5=138:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
7:1573512:d=10091218:D:kpds5=155:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
8:1835764:d=10091218:R:kpds5=157:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
9:2098016:d=10091218:O3:kpds5=203:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
10:2360268:d=10091218:CLWC:kpds5=246:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
11:2556984:d=10091218:CIWC:kpds5=247:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:anl:type=analysis:ens 14/2/4/1:weighted mean no bias:NAve=0
12:2753700:d=10091218:CC:kpds5=248:kpds6=100:kpds7=1:TR=0:P1=0:P2=0:TimeU=1:1
mb:


and they can check  the PDS section beyond octet 40. 


Yuan
> 
> Since these are grib 1 files there is no way to encode
> an "ensemble dimension set to 1" in the grib 1 standard
> definition of the product definition section (pds). Perhaps
> idv is finding/decoding a set of bytes that are ECMWF-defined
> beyond the end of the pds which has no real bearing on the
> actual grib 1 standard (not uncommon), and is interpreting these in an
> incorrect manner. Did unidata say specifically in which section
> idv was finding these bytes referring to an ensemble, and if
> so, specifically which bytes. This is the first time we have encountered
> such a user report, out of hundreds of users and many terabytes
> of grib 1 era-interim data.
> 
> Best regards,
> 
> Dave Stepaniak
> 
> Let me know if this helps.
> 
> V/R,
> 
> Stefan
> 
> Stefan Cecelski
> Ph.D. Candidate and Graduate Research Assistant
> AOSC Department: University of Maryland College Park
> 2343 Computer and Space Sciences Building(CSS #224)
> 703.801.1370
> address@hidden
> 
> 
> 
> 
> 
> On Aug 4, 2011, at 4:46 PM, Unidata IDV Support wrote:
> 
> >> Yuan,
> >>
> >> This must be a mistake on the data set provider's part. The ERA-
> >> Interim data set is an extension of the ERA-40 reanalysis, which is a
> >> determinisitic forecast(with NO ensembles). I will email them about
> >> this issue. In the meantime, can I just edit the metadata to set
> >> ens=0? Will the aggregate data by time work if I do it with all
> >> files?
> >>
> >> V/R,
> >>
> >> Stefan
> >
> > Stefan,
> >   There is no way to remove this additional dimension with ncml.
> > Also, this ensemble dimension is not well defined according to the
> > CDM standard. I created a ncml for your reference, but this ncml
> > file only modified the the single grib file, you can not do the
> > aggregation with it. Loading this ncml file to the IDV, the display
> > is a bit normal than the original file, and you can actually load
> > the second ncml file.
> >
> >
> >
> > Yuan
> >>
> >> Stefan Cecelski
> >> Ph.D. Candidate and Graduate Research Assistant
> >> AOSC Department: University of Maryland College Park
> >> 2343 Computer and Space Sciences Building(CSS #224)
> >> 703.801.1370
> >> address@hidden
> >>
> >>
> >>
> >>
> >>
> >> On Aug 3, 2011, at 2:02 PM, Unidata IDV Support wrote:
> >>
> >>>> Yuan,
> >>>>
> >>>> Funny thing is; this is a dertministic forecast. I don't see
> >>>> anywhere
> >>>> in the .ctl file that 'tags' as an ensemble or anywhere in the grib
> >>>> data that designates the data from an ensemble. The data was
> >>>> directly
> >>>> downloaded from here:
> >>>>
> >>>> http://dss.ucar.edu/datasets/ds627.0/
> >>>>
> >>>> Quite odd.
> >>>>
> >>>
> >>> Well, you may get the wrong dataset. ;)
> >>>
> >>> If you load one dataset into the IDV, and the bring up the
> >>> properties window of this dataset (Data Sources: double click on the
> >>> dataset, or right click and select the Properties), you will see
> >>> this ensemble dimension.
> >>>
> >>>
> >>>
> >>> Yuan
> >>>> V/R,
> >>>>
> >>>> Stefan
> >>>>
> >>>> Stefan Cecelski
> >>>> Ph.D. Candidate and Graduate Research Assistant
> >>>> AOSC Department: University of Maryland College Park
> >>>> 2343 Computer and Space Sciences Building(CSS #224)
> >>>> 703.801.1370
> >>>> address@hidden
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Aug 3, 2011, at 1:46 PM, Unidata IDV Support wrote:
> >>>>
> >>>>>> Yuan,
> >>>>>>
> >>>>>> Sorry, another one is uploading now.
> >>>>>>
> >>>>>> V/R,
> >>>>>>
> >>>>>> Stefan
> >>>>>
> >>>>> The problem is the ensemble dimension, it is not recognized in the
> >>>>> IDV, can you remove this dimension?
> >>>>>
> >>>>>
> >>>>> Yuan
> >>>>>>
> >>>>>> Stefan Cecelski
> >>>>>> Ph.D. Candidate and Graduate Research Assistant
> >>>>>> AOSC Department: University of Maryland College Park
> >>>>>> 2343 Computer and Space Sciences Building(CSS #224)
> >>>>>> 703.801.1370
> >>>>>> address@hidden
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Aug 3, 2011, at 1:16 PM, Unidata IDV Support wrote:
> >>>>>>
> >>>>>>>> Yuan,
> >>>>>>>>
> >>>>>>>> I have uploaded a grib file(scalar variables), along with
> >>>>>>>> the .ctl
> >>>>>>>> and .idx files. The ERA data is split into three files for each
> >>>>>>>> time:
> >>>>>>>> scalar 3D fields, U-V 3D fields and surface fields. As I have
> >>>>>>>> said
> >>>>>>>> before, I tried using the Aggregate by Time and opening
> >>>>>>>> multiple .grib
> >>>>>>>> files just in IDV without using the .ctl/.idx file. This was
> >>>>>>>> unsuccessful. Thanks for your help!
> >>>>>>>>
> >>>>>>>> V/R,
> >>>>>>>>
> >>>>>>>> Stefan
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Stefan,
> >>>>>>> You only uploaded one grib file, could you add a few more?  For
> >>>>>>> the grib file, you don't need .ctl and .idx file in the IDV.
> >>>>>>>
> >>>>>>>
> >>>>>>> Yuan
> >>>>>>>> Stefan Cecelski
> >>>>>>>> Ph.D. Candidate and Graduate Research Assistant
> >>>>>>>> AOSC Department: University of Maryland College Park
> >>>>>>>> 2343 Computer and Space Sciences Building(CSS #224)
> >>>>>>>> 703.801.1370
> >>>>>>>> address@hidden
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Jul 28, 2011, at 12:09 PM, Unidata IDV Support wrote:
> >>>>>>>>
> >>>>>>>>>> Yuan,
> >>>>>>>>>>
> >>>>>>>>>> I tried using Aggregate Grids By Time, then selecting
> >>>>>>>>>> multiple
> >>>>>>>>>> time
> >>>>>>>>>> files, and it would only load one time. For example, I have
> >>>>>>>>>> files
> >>>>>>>>>> every 6 hours for the ERA-Interim data set. I would load up 3
> >>>>>>>>>> days
> >>>>>>>>>> worth of GRIB files, and one middle time would be produced as
> >>>>>>>>>> the
> >>>>>>>>>> only
> >>>>>>>>>> time step.
> >>>>>>>>>>
> >>>>>>>>>> V/R,
> >>>>>>>>>>
> >>>>>>>>>> Stefan
> >>>>>>>>>>
> >>>>>>>>> Hi Stefan,
> >>>>>>>>>
> >>>>>>>>> Could you upload a few sample files together with the ctl
> >>>>>>>>> files?
> >>>>>>>>>
> >>>>>>>>> http://motherlode.ucar.edu/repository/alias/idvupload
> >>>>>>>>>
> >>>>>>>>> Yuan
> >>>>>>>>>
> >>>>>>>>> Ticket Details
> >>>>>>>>> ===================
> >>>>>>>>> Ticket ID: XYY-917880
> >>>>>>>>> Department: Support IDV
> >>>>>>>>> Priority: Normal
> >>>>>>>>> Status: Open
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Ticket Details
> >>>>>>> ===================
> >>>>>>> Ticket ID: XYY-917880
> >>>>>>> Department: Support IDV
> >>>>>>> Priority: Normal
> >>>>>>> Status: Open
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> Ticket Details
> >>>>> ===================
> >>>>> Ticket ID: XYY-917880
> >>>>> Department: Support IDV
> >>>>> Priority: Normal
> >>>>> Status: Open
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: XYY-917880
> >>> Department: Support IDV
> >>> Priority: Normal
> >>> Status: Open
> >>>
> >>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: XYY-917880
> > Department: Support IDV
> > Priority: Normal
> > Status: Closed<pl0.ncml>
> 
> 
> 


Ticket Details
===================
Ticket ID: XYY-917880
Department: Support IDV
Priority: Normal
Status: Closed