[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Few questions...



Rob Weingruber wrote:

   Alrighty....
ftp to ftp.rap.ucar.edu <ftp://ftp.rap.ucar.edu> and login as anonymous (using your email as the password). 

cd pub/irap/weingrub

and the file is called caron.jar

Unjar.  (Lemme know if something aint working ;-).

The 2006070611 directory has 11:00Z gen time data, with final analysis data
for 8:00, 9:00 and 10:00. The 11:00 data in that directory is the 0 hour forecast, 
the 12:00 data is the 1 hour forecast, etc.


im trying to figure out if that info is in the file itself:

in

2006070611/wrfout_d01_2006-07-06_080000.DPG_F.nc

i see global attribute

:START_DATE = "2006-07-06_07:00:00";
:SIMULATION_START_DATE = "2006-07-03_12:00:00";

and variable data

char Times(Time=1, DateStrLen=19);

data:  "2006-07-06_08:00:00"

i assume this is "valid time" 2006-07-06_08:00:00,
but the "run time" of 2006-07-06_11:00:00 doesnt seem to be in there

in
wrfout_d01_2006-07-06_130000.DPG_PFCST.nc

i see

:START_DATE = "2006-07-06_10:00:00";
:SIMULATION_START_DATE = "2006-07-03_12:00:00";

char Times(Time=1, DateStrLen=19);
  data:"2006-07-06_13:00:00"

So again theres no indication of the run time.

if we can figure out this, we can probably use FMRC aggregation.



The 2006070614 directory has 14:00Z gen time data, with final analysis data
for 11:00, 12:00 and 13:00. The 14:00 data in that directory is the 0 hour forecast, 
the 15:00 data is the 1 hour forecast, etc.
Also, the timesExplicit.xml file is where I tried to put the time coordinates explicitly, with the intention of being able to retrieve times, from Java, without opening the files - to no avail (proven by moving the data files out of the way).
The problem is that all of the variables in the file have the time dimension, and so get aggregated, including some that we'd rather not. 
When the coordinate system is constructed, we read in the ZNU field, which has 
a time dimension, and so we have to access all files.
Ill have to see if the FMRC aggregation has the same problem. 
-----

Another problem with this file is that there is no coordinate variable, ie 
Time(Time), which the joinExisting spec requires. It seems to be tolerating it 
ok, but it screws things up some.

The only thing you could do is add it in the NcML, but you have to give it 
values, eg:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>

 <variable name="Time" shape="Time" type="String">
   <values>2006-07-06T08:00:00Z 2006-07-06T09:00:00Z etc </values>
 </variable>

 <aggregation dimName="Time" type="joinExisting">
   <netcdf location="file:C:/data/rap/2006070611/wrfout_d01_2006-07-06_080000.DPG_F.nc" 
coordValue="2006-07-06T08:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070611/wrfout_d01_2006-07-06_090000.DPG_F.nc" 
coordValue="2006-07-06T09:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070611/wrfout_d01_2006-07-06_100000.DPG_F.nc" 
coordValue="2006-07-06T10:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070611/wrfout_d01_2006-07-06_110000.DPG_PFCST.nc" 
coordValue="2006-07-06T11:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070611/wrfout_d01_2006-07-06_120000.DPG_PFCST.nc" 
coordValue="2006-07-06T12:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070611/wrfout_d01_2006-07-06_130000.DPG_PFCST.nc" 
coordValue="2006-07-06T13:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070611/wrfout_d01_2006-07-06_140000.DPG_PFCST.nc" 
coordValue="2006-07-06T14:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070611/wrfout_d01_2006-07-06_150000.DPG_PFCST.nc" 
coordValue="2006-07-06T15:00:00Z"/>

   <netcdf location="file:C:/data/rap/2006070614/wrfout_d01_2006-07-06_110000.DPG_F.nc" 
coordValue="2006-07-06T11:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070614/wrfout_d01_2006-07-06_120000.DPG_F.nc" 
coordValue="2006-07-06T12:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070614/wrfout_d01_2006-07-06_130000.DPG_F.nc" 
coordValue="2006-07-06T13:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070614/wrfout_d01_2006-07-06_140000.DPG_PFCST.nc" 
coordValue="2006-07-06T14:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070614/wrfout_d01_2006-07-06_150000.DPG_PFCST.nc" 
coordValue="2006-07-06T15:00:00Z"/>
   <netcdf location="file:C:/data/rap/2006070614/wrfout_d01_2006-07-06_160000.DPG_PFCST.nc" 
coordValue="2006-07-06T16:00:00Z"/>
 </aggregation>
</netcdf>

more later...
The 
timesFromFiles.xml
file is working fine, though retrieves times from the files, albeit slowly (esp if we have 
tons of files).
Lastly, is there even a way thru the *API*, where I can extract times without using a GeoGrid (or GridDatatype)? - It seems to me that once I have a GeoGrid, I have 
already opened the file(s).


Normally the time coordinate would be available through any of the interfaces 
(NetcdfFile, NetcdfDataset, Grid) without having to read all the data (assuming 
you've specified it in the NcML), but the problems above are interfering with 
that.





My apologies for my confusion ;-)

John Caron wrote:


probably better pput them on an http or ftp server

Rob Weingruber wrote:


   Hola -
They're about 18M each - ok via email? Else Ill find somewhere to put em for ya.... 

Lemme know ;-)

Thanks.

John Caron wrote:
send me a few of your files, or links to them, and ill put an NcML file together for you. 

Rob Weingruber wrote:


   Hola -

How busy are you?  ;-))))  I was wondering if you might be willing
to come on over here and show me a thing or two, wrt nj22 and ncml.
Im struggling with the time, aggregation and the caching issues, and I
bet you'd be able to clear lots of things up for me. The real jist is, 
Im not sure I really know how to use the nj22/ncml classes, and I dont
know if I/we can do it efficiently, especially for time list requests.

Or, maybe snippets of code could help too?

Whadayathink?

;-)

John Caron wrote:




Rob Weingruber wrote:


   Hi John -

Im back at it.....
Ok, so after perusing GridDatatype andGridCoordSystem, I have a few questions:
1. Can you explain the ensemble to me? Is that for climatologies? (if so, i understand 
the ensemble ;-).
models are sometimes run several (many) times with slight variations of the input, these are called "ensemble" runs. The set of results then show the fuuture in a more statistically meaningful way. these results could be represented as an extra "ensemble" dimension. 

we havent yet tested this with any real world examples yet.
2. Is the RunTime the same as what I termed the 'generate time'? If so, might be better to use 'gen time' rather than run time, since a model run may have a gen time of 12Z, but it happens to be running at 12:18Z (my $0.02 ;-).
im not sure, i could also imagine the argument "a model run may have a runtime of 12Z, but it happens to be generated at 12:18Z". 

maybe we can have a bakeoff, i've also heard "analysis time".



Related to NCML:

3. Can I use the 'Forecast Model Run' yet?  If so, whudoIdo?
we are just finishing the "Forecast Model Run Collection" (fmrc) aggregation, you can try it out if you want, but i havent written docs yet. (there was an older "Forecast Model Run" aggregation which is now deprecated.)
lets pick a simple dataset you want to do, and try serving it through the TDS:
1. have a look at the motherlode test server models (http://motherlode.ucar.edu:9080/thredds/idd/models.html) 2. install latest 3.13 development version of thredds.war (from http://www.unidata.ucar.edu/downloads/thredds/index.jsp) 3. pick a simple dataset, i will show you what the catalog should be. 
 4. ill try to get some docs written.
now that ive said all this, maybe you just want to access the fmrc aggregation dataset from java code?
4. How can I aggregate with the 2D time coordinate? I looked at the joinExisting doc http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html, 
and I still dunno what to do....





its likely you need the fmrc aggregation.



;-)  Thanks John!





yer welcome!