[netCDF #YWP-403246]: compiling netCDF

Greetings Chad!

> Sean,
> 
> I just got HDF5 to support, but there is no "make install:" option.

`make install` should work - did you follow the instructions at 
http://www.hdfgroup.org/ftp/HDF5/current/src/unpacked/release_docs/INSTALL 
regarding the ./configure setup?

> Where do I put it so that when I compile netCDF again, it can find
> what it needs.  Can I just copy everything in the HDF5/bin folder to
> /usr/local/bin, or maybe I should make it /usr/local/hdf5/bin.  Are
> there any library or include files that need to go into
> /usr/local/hdf5?  I figure if I use a separate pat like
> /usr/local/hdf5/bin, and then add that to my $PATH or remove it from
> my $PATH, I should be able to compile netCDF with or without HDF5
> support.  It would be easier if there was an environmental variable
> that the netCDF compile script recognized, like
> export HDF5=/usr/local/hdf5.  Would that work?  

The configure script for netCDF4 includes an option --with-hdf5=<path to top 
level hdf5>. If you run the config script for hdf5 using --prefix=<path where 
you want hdf libs, bin, and include to go>, then simply use this path when you 
compile netCDF4. For example, you suggested /usr/local/hdf5 as a place to stuff 
hdf5. In that case, when configuring the build for hdf5, use

./configure --prefix=/usr/local/hdf5 [other options, as needed]

Then, when compiling netCDF, use:

./configure --with-hdf5=/usr/local/hdf5 [other options, as needed]

> That way I could
> enable or disable that variable for compiling netCDF with or without
> HDF5 support.
> 

Changing $PATH as needed should work for what you need - just make sure to keep 
close watch on what $PATH says because it can bite pretty hard.

> 
<paste>
>
> I do not know why /bin/sh points to dash by default. That is weird.
> I just tried an apt-get install bash, and it says I have the latest
> version. I double-checked this by typing
>
> chad@chad-MacBookPro:/bin$ whereis bash
> bash: /bin/bash /etc/bash.bashrc /usr/share/man/man1/bash.1.gz
>
> So, /bin/bash is the real bash. Should I change the symbolic link so
> that /bin/sh points to /bin/bash? Would that help? I do not know why
> /bin/sh points to dash instead of bash. Any idea? Seems stupid to
> me. I am going to go ahead and rm /bin/sh and point it to /bin/bash
> instead.
>

The Ubuntu devs decided to make /bin/dash the default, and I am not sure of why 
they chose to do so. Does the Intel ifortvars.sh start with #!/bin/sh? If so, 
then that explains everything, as /bin/sh points to /bin/dash, and therefore 
/bin/dash was being used to run the script even though you are running 
/bin/bash.

> I just tried your idea and I think it worked. When I ran a make
> -check, it says it passed all the tests. I will attach log files for
> you to double check.

Things look good to me.

>
> I do not have HDF5 compiled on this machine, because wrfhelp@ucar told
> me not to. But, I do have a compiled version of netCDF 363 (borrowed
> the binaries from a CentOS server at the university). So, I will have
> to keep both netCDF 4 without HDF5 for compiling WRF and doing WRF
> simulations and need to keep netCDF 4 with HDF5 suport, for using
> nccopy to convert the WRF output data into a format with HDF5 that
> Tecplot can read. As long a we can get netCDF to compile with HDF5,
> then I should be in good shape. Thanks for telling me about nccopy.
> Saves me a bunch of time so I do not have to write a custom data
> converter. Incidentally, I did ask the WRF people about their plans
> regarding support for netCDF4 with HDF5 support and they said its at
> least a year away.
>
> Now I need to re-compile it with HDF5. Can I use pre-compiled
> binaries, or should I compile HDF5 with the intel compilers to make
> sure that its fully compatible? There are linux 64 bit binaries of
> HDF5 available at
>
> http://www.hdfgroup.org/HDF5/release/obtain5.html
>
> I am leaning toward trying to compile HDF5 first. I know that WRF
> will not compile unless its compiled under the same OS with the same
> compilers. Is it the same for netCDF and HDF5?
>
> I am attaching logs of configure, make and make check. Tell me if the
> little trick in switching /bin/sh -> /bin/bash actually worked.
>
> I have one important question. I just compiled netCDF 4.1.2 without
> HDF5 support, and now want to compile WRF using it. If I have nccopy
> from a netCDF 4.1.2 compilation with HDF5 support, will nccopy still
> work, converting a WRF output file from 4.1.2 to 4.1.2 with HDF5
> support, or does nccopy only work for converting from netCDF 3.x to
> netCDF 4.x with HDF5 support?
>

When you configured netCDF4, did you specify --enable-netcdf-4? If so, then I 
think nccopy will work.

Sean



>
> Thanks,
>
> Chad
> 
> On Thu, Jun 2, 2011 at 2:47 PM, Unidata netCDF Support
> <address@hidden> wrote:
> > Chad,
> >
> >> Sean,
> >>
> >> I am using the atest ubuntu 11.0.4, just downloaded and installed las
> >> week.  Here is the ouput you requested:
> >>
> >> chad@chad-MacBookPro:~$ echo $SHELL
> >> /bin/bash
> >> chad@chad-MacBookPro:~$ ls -la /bin/sh
> >> lrwxrwxrwx 1 root root 4 May 25 21:07 /bin/sh -> dash*
> >> chad@chad-MacBookPro:~$
> >
> > The failures you get for `make check` show up when using ifort during the 
> > tests, and once again ifort is having issues finding the Intel shared 
> > objects. Just out of curiosity, is /bin/bash also a symlink to /bin/dash? 
> > If so, then that is the problem - dash is not 100% compatible with bash, 
> > and thus the Intel scripts may not be fully functional. If it is a link to 
> > dash, use apt-get to install bash and then try to rebuild.
> >
> >>
> >> I am not sure if I should open a separate support issue for this, but
> >> I am running into a data incompatibility problem.  I want to use
> >> Tecplot 360 (tecplot.com) for visualzing and making movies of output
> >> from WRF.  Tecplot 360 support HDF5 data, but WRF only suports netCDF
> >> without HDF5, from what I have read and been told by wrfhelp.  How can
> >> I get my data into Tecplot 360 from a netCDF file that has no HDF5?
> >> Do you know if WRF planps to upgrade to HDF5 support soon?
> >
> > I am not sure what the plans are with WRF,as it is not developed at 
> > Unidata. The folks at NCAR who are involved in the administration of the 
> > codebase should be able to answer that question (I'd check with WRFhelp). 
> > The questions to ask would be if/when WRF plans to use netCDF4.
> >
> >>
> >> I suppose we would need to get this netCDF 4 with HDF5 support to
> >> compile, then I could write a program that read the netCDF without
> >> HDF5 data, then wrote it back with netCDF with HDF5.  Is netCDF 4 with
> >> HDF5 backwards compatible with output from say netCDF 363?  So, if I
> >> write a code using netCDF4 with HDF5, can I read netCDF 363 data and
> >> then write it back in HDF5 format?
> >
> > netCDF4 will read data written with netCDF3 - no problems :-) That said.. .
> >
> >> Do you guys have any utiities that
> >> already do this?  A simple utility that reads in a netCDF 363 data
> >> file and upgrades it to netCDF4 with HDF5 support?  If you do not, you
> >> should write one.  It would be simple for you guys that know netCDF so
> >> well.   If I have to do it, its going to take me a while to figure out
> >> how.... time I really do not have right now.
> >
> > If you check out the netCDF FAQ 
> > (http://www.unidata.ucar.edu/software/netcdf/docs/faq.html), there is an 
> > entry about how to do this 
> > (http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#fv13).
> >
> >> I can write code in
> >> Fortran and C, but would prefer to use Python.  Python can usually
> >> call C libraries, so do you know for sure if Python can link to
> >> netCDF4 with HDF5 support?
> >
> > Since nccopy can do what you need it to do (once we get it to compile, that 
> > is), it may be of general interest to know that there is a netCDF4 python 
> > wrapper that is functional, although a few features are missing (see 
> > http://code.google.com/p/netcdf4-python/). I've compiled and used this on 
> > MacOSX and it works great so far.
> >
> > Sean
> >
> >> If you can think of any other solution to
> >> this problem, please let me know.
> >>
> >>
> >> Many Thanks,
> >>
> >> Chad
> >>
> >> On Wed, Jun 1, 2011 at 3:03 PM, Unidata netCDF Support
> >> <address@hidden> wrote:
> >> > Hi Chad,
> >>
> >>
> >>
> >> Iatest ubuntu 11.0.4, just downloaded and installed las week.  Here is
> >> the ouput you requested:
> >>
> >> chad@chad-MacBookPro:~$ echo $SHELL
> >> /bin/bash
> >> chad@chad-MacBookPro:~$ ls -la /bin/sh
> >> lrwxrwxrwx 1 root root 4 May 25 21:07 /bin/sh -> dash*
> >> chad@chad-MacBookPro:~$
> >>
> >> I am not sure if I should
> >>
> >>
> >> On Wed, Jun 1, 2011 at 3:03 PM, Unidata netCDF Support
> >> <address@hidden> wrote:
> >> > Hi Chad,
> >> >
> >> > It seems as though netCDF is still having issues finding the Intel libs. 
> >> > First, which version of Ubuntu are you running? Second, could you send 
> >> > me the output from the following two commands:
> >> >
> >> > 1) echo $SHELL
> >> >
> >> > 2) ls -la /bin/sh
> >> >
> >> > Thanks!
> >> >
> >> > Sean
> >> >
> >> >> Hi Sean,
> >> >>
> >> >> I did as you suggested.  I cleared all the shell environmental
> >> >> variables and used
> >> >> source /opt/intel/bin/iccvars.sh intel64
> >> >> source /opt/intel/bin/ifortvars.sh intel64
> >> >>
> >> >> The configure and make seemed to go ok, but when I ran "make check" it
> >> >> failed 4 out of 5 of the tests.
> >> >>
> >> >> I made log files of each stage and attached them to this email for you
> >> >> to examine.  Do you think gc/gortran would work better or stay with
> >> >> intel on ubuntu?  Others here at ASU reported better success with
> >> >> intel compilers, which is why I am using them.
> >> >>
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Chad
> >> >>
> >> >>
> >> >> On Tue, May 31, 2011 at 1:00 PM, Unidata netCDF Support
> >> >> <address@hidden> wrote:
> >> >> > Greetings Dr. Coulliette!
> >> >> >
> >> >> >> Hello,
> >> >> >>
> >> >> >> My name is Dr. Chad Coulliette.  I am working with Professor Alex
> >> >> >> Mahalov at Arizona State University.  I have been trying to compile
> >> >> >> netCDF on Ubuntu Linux 64-bit.  The hardware is a MacBook Pro with a
> >> >> >> Intel Duo Core 64-bit processor.  I want to compile and use WRF, but
> >> >> >> need netCDF first.  I have netCDF installed as a package using
> >> >> >> synaptic, but I suspect there is something deficient about it, so we
> >> >> >> have been trying to compile and install netCDF from source code.  I
> >> >> >> have netCDF-4.1.2 and have tried with both GNU compilers and Intel
> >> >> >> compilers.  The GNU compilers seem to compile ok....no errors during
> >> >> >> make, but when I run "make check" it finds problems, and the message
> >> >> >> actually says to contact this email address.
> >> >> >
> >> >> > From looking through the make-gcc.log, you seem to be using the Intel 
> >> >> > fortran compiler `ifort` along side the gcc `cc` compiler - this is 
> >> >> > because .configure could not find any other fortran compiler (i.e. 
> >> >> > gfortran) in your $PATH. The error you are getting, as indicated in 
> >> >> > the make-test-gcc.log file, is due to problems finding the Intel 
> >> >> > shared objects. This is probably because when you building with a GNU 
> >> >> > environment in mind and so you didn't source ifortvars.sh. If you 
> >> >> > want to go the GNU route, try installing gfortran .
> >> >> >
> >> >> >> With the Intel
> >> >> >> compilers, it crashes during the make.  I included log files of the
> >> >> >> configure, make and make check for the GNU compilers, and included 
> >> >> >> log
> >> >> >> files for the configure and make for the Intel compilers.
> >> >> >>
> >> >> >> For the Intel compiler, I have been following the instructions at
> >> >> >>
> >> >> >> http://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-netcdf-with-the-intel-compilers/
> >> >> >>
> >> >> >> Are these the best compile instructions?  They might be too old,
> >> >> >> because I noticed a complaint about the compiler flag -xT being
> >> >> >> depreciated.  I tried to take it out and run everything over, but the
> >> >> >> results were the same.
> >> >> >>
> >> >> >
> >> >> > The instructions above are for an older version of netCDF. I would 
> >> >> > check out the Unidata netCDF docs, which can be found here:
> >> >> >
> >> >> > http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-install/Building-on-Unix.html#Building-on-Unix
> >> >> >
> >> >> > These docs do not specifically look at using the Intel compiler, but 
> >> >> > it should give you some general reference.
> >> >> >
> >> >> > What happens if you do not set any flags by hand, but only source 
> >> >> > iccvars.sh and ifortvars.sh before building?
> >> >> >
> >> >> > Cheers!
> >> >> >
> >> >> > Sean Arms
> >> >> >
> >> >> >> Could you examine the log files and let me know what to do?
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Thanks,
> >> >> >>
> >> >> >> Chad
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> > Ticket Details
> >> >> > ===================
> >> >> > Ticket ID: YWP-403246
> >> >> > Department: Support netCDF
> >> >> > Priority: Normal
> >> >> > Status: Open
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >
> >> >
> >> > Ticket Details
> >> > ===================
> >> > Ticket ID: YWP-403246
> >> > Department: Support netCDF
> >> > Priority: Normal
> >> > Status: Open
> >> >
> >> >
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: YWP-403246
> > Department: Support netCDF
> > Priority: Normal
> > Status: Open
> >
> >
> 
> 


Ticket Details
===================
Ticket ID: YWP-403246
Department: Support netCDF
Priority: Normal
Status: Open