Parallel netcdf

> WRT T3E parallel netCDF...this is the impetus for my inquiry.  We (GFDL)
> will be conducting a review of the parallel-status of our applications
> here, and netCDF is an obvious component.  I have had embarrassingly
> little time to keep up with the NERSC work, so I need to carve out some
> time to get back into that.  I know of several approaches that were
> being tried: a "global" FFIO layer, application source code changes,
> and intermediate layers which write out partial files which are later
> combined into the full dataset.  Do you have any readily available
> information on how the T3E parallel netCDF works?

It uses "global" FFIO.

The following functions are added to the public netcdf interface:

    MPP FUNCTION DESCRIPTIONS
          Additional  functions  for  use  on  SGI/Cray  MPP  machines
          (_CRAYMPP).   These  are used to set and inquire which PE is
          the base for MPP for a particular  netCDF.  These  are  only
          relevant  when  using  the SGI/Cray ``global'' Flexible File
          I/O layer and desire to have only a subset of  PEs  to  open
          the  specific  netCDF  file.   For  technical reasons, these
          functions are available on all  platforms.   On  a  platform
          other  than  SGI/Cray MPP, it is as if only processor avail-
          able were processor 0.

          To use this feature, you  need  to  specify  a  communicator
          group  and call glio_group_mpi() or glio_group_shmem() prior
          to the netCDF nc_open() and nc_create() calls.

          int nc__create_mp(const char path[], int cmode, size_t  ini-
               tialsize, int pe, size_t* chunksize, int* ncid)

               Like nc__create() but allows the base PE to be set.

               The argument pe sets the base PE at creation  time.  In
               the  MPP  environment, nc__create() and nc_create() set
               the base PE to processor zero by default.

          int nc__open_mp(const char path[], int mode, int pe, size_t*
               chunksize, int* ncid)

               Like nc__open() but allows the base PE to be set.   The
               argument  pe  sets the base PE at creation time. In the
               MPP environment, nc__open() and nc_open() set the  base
               PE to processor zero by default.

          int nc_inq_base_pe(int ncid, int* pe)

               Inquires of the netCDF dataset which PE is  being  used
               as  the  base  for MPP use.  This is safe to use at any
               time.

          int nc_set_base_pe(int ncid, int pe)

               Resets the base PE for the netCDF dataset.   Only  per-
               form  this  operation  when  the  affected communicator
               group synchronizes before and  after  the  call.   This
               operation is very risky and should only be contemplated
               under only the most extreme cases.

    ENVIRONMENT VARIABLES
          NETCDF_FFIOSPEC
              Specifies the Flexible File I/O buffers for  netCDF  I/O
              when  executing  under  the UNICOS operating system (the
              variable is ignored on other operating systems).  An ap-
              propriate  specification  can greatly increase the effi-
              ciency of netCDF I/O -- to the extent that it can  actu-
              ally surpass FORTRAN binary I/O.  This environment vari-
              able has been made a little more generalized, such  that
              other  FFIO option specifications can now be added.  The
              default specification is bufa:336:2,  unless  a  current
              FFIO  specification  is  in  operation,  which  will  be
              honored.  See UNICOS Flexible File I/O for more informa-
              tion.