Re: netCDF for the CM-5

> Organization: LANL
> Keywords: 199408102030.AA03168

Hi Rick,

> I thought you might like to know the status
> of my port of netCDF to the CM-5; and I might
> need your help to speed things up.
> 
> The library is now completely operational in
> both data parallel and nodal versions on the CM-5,
> including the shadowing of metadata, with the
> caviat that chars, bytes and shorts are not
> supported in the data parallel version; thinking
> that in general parallel scientific data does
> not include these.  If necessary, we can expand
> the library to support these in the future.

Congratulations on getting it working on the CM-5!  Other users have asked
me about netCDF for a CM-5, and I'll continue to refer their questions to
you, if you don't mind.

Although we have little experience with parallel applications for netCDF
data, we have seen examples of the use of bytes for scientific data, both
for compressing low resolution floating-point data and for storing satellite
image data.  We have also seen some use of shorts for the same purposes.

> The problem I face now; or WE face now; is that
> the nodal version is extremely slow, according to
> one friendly user who has been testing it.   This
> is essentially vanilla serial netCDF, with a couple
> minor mods to make it work as a nodal library.
> I did modify xdr_NCvdata() in the nodal library to
> call xdr_opaque() instead of looping for count items
> calling xdr_float() etc.  This sped it up by an order
> of magnitude (literally!).  But it's not enough. So
> I think this is a problem we need to solve together.

If the use of xdr_opaque() means the resulting data is no longer portable
but can only be read from another CM-5, that may be a high price to pay for
the resulting speedup for users who want portable data.  In many uses of
netCDF, the time required for the I/O is dwarfed by the time taken to
generate or analyze the data, so making netCDF optimally efficient doesn't
result in much speedup in such applications.  However I know there are also
some applications where time for netCDF access is significant and that in
optimizing parallel applications, a little slow serial code can cause major
slowdowns.

> Since everything does work reliably, and you probably
> know how to tune this baby better than I do,
> I am ready to turn the library over
> to you.  And to work with you to come up with and test
> various mods to make both versions as efficient as 
> possible.

Actually we have not tuned the netCDF library for any particular platform,
so we really don't have much experience in this area, and we have no access
to a CM-5 or any short-term resources to work on platform-specific netCDF
optimizations.  If we had such resources, tuning on a Cray platform would
currently be a higher priority, from the requests we have seen.

My past approach has been to try to encourage someone with a vested interest
in speeding up netCDF on a particular platform to tune it, but this has so
far not resulted in changes of general use that we can reliably integrate
back into the source distribution and maintain.

> The approach I have taken is to document all of my assumptions,
> to document which files have been modified, and to include
> CM-5 related #ifdefs throughout the code.  I have also noted
> all changes with comments, usually something like
> 
>       /*  ---------- cm5 addition ---------------  */
> 
>       ...
> 
>       /*  ---------- end cm5 addition -----------  */
> 
> 
> 
> Anyway, I think things are in pretty good shape to give you.
> I can create for you a tar file of everything, and email you
> my file of notes for "the netCDF maintainers".  Then when you
> are ready to get into this I will be happy to help you in any
> way I can.

Despite what I said above, I'd be interested in getting a copy of your tar
file.  I'm especially interested in your modifications for shadowing of
metadata.  I can't promise we'll incorporate your changes, but it sounds
like you've made them in a way that would facilitate integrating them into
our sources.

> I know you are traveling this week, and I will be gone from
> Aug 18-30th.  So perhaps September is a better time for us
> to begin this.  Please let me know how you would like to
> incorporate this work into your own netCDF schedules.

It looks like I will be working on preparations for a workshop (unrelated to
netCDF) during all of September, so I won't be able to look at this until
October.

> Thanks for your help.  

And thanks for your contributions.

--
Russ Rew                                                   UCAR Unidata Program
address@hidden                                             P.O. Box 3000
http://www.unidata.ucar.edu/                             Boulder, CO 80307-3000