Category Archives: Programming

Open MPI and Debian

Branden sent me a link from Planet Debian where one of the Debian developers talks about improving the packaging for Open MPI in Debian. Nice to see people outside the core development community get excited about Open MPI.

Now if only we could get people to spell it Open MPI instead of OpenMPI 😉

Open MPI in Mac OS X!

http://www.apple.com/macosx/leopard/technology/multicore.html
Apple finally announced that Mac OS X 10.5 (Leopard) will include Open MPI as part of its developer tools. The discussion to make this happen started over 18 months ago, and it’s great to finally see it announced. I’ve been working on this for a long time and it looks like we’re finally there. Woo hoo!

Sun’s working on adding some DTrace hooks into Open MPI for tracking things like message arrival and the like. Won’t make it into 1.2, but should be nice once they’re in, especially considering the announced support for DTrace on Leopard.

Happiness on the Open MPI front!

Friday night update

So it’s Friday night, it’s been 6 weeks and 2 days since my last update. Which is a bit pathetic, but I’m sure that everyone will survive without too much difficultly. Anyway, since I don’t remember most of what has happened, we’re going for list format.

  • I now have 99.5 hours of flying time (so close, yet so not 100 hours), including a bunch of flights since my last update:
    • A flight with Branden and Kyle to Taos and back in N80790 just to fly around and log me some cross country time. Branden took some pictures
    • A flight with Laura in early December to Taos for lunch (yummy Mexican) and log me some cross country time. There was still snow on the north sides of the slopes from the November snow storm, so it was really pretty.
    • A bunch of test flights in N813T, Galen’s plane. It’s a blast to fly — very responsive and very quick. The most fun I’ve ever had while flying.
    • A long (~300 nautical mile) cross country flight around Colorado in N813T, solo because it was a test flight. The entire flight was only about 2 hours, since the plane is so fast.
  • Arun, Shelece and I drove down to White Sands in December in hope that Discovery was going to land there (it was looking promising). Just as we arrived at White Sands National Monument, it was announced they were going to land at Kennedy. Good for the NASA budget, not so fun for us. But we did get to run around White Sands, which was fun. We stopped for lunch at a place along the way Arun and Shelece had read about and had one of the best green chili cheeseburgers I’ve ever eaten. Fun trip, but would have been better if there was a shuttle landing overhead.
  • I went home for Christmas break, just barely escaping the giant snow storm in Denver to get to Chicago. Getting home was delayed by yet another snow storm in Denver, but it wasn’t too bad. Just had to spend a boring New Years in South Bend. We spent Christmas afternoon in Detroit, and got to see the Barrett clan, which is always fun. Also got to see Anne while we were in Grand Rapids visiting Cathy and Dan.
  • Rich Graham (the group/team/project leader I was working for) announced that was leaving LANL for Oak Ridge. Adolfy Hoisie took over the group effective January 8th. While a bit scary for a while due to the uncertainty of change, the change should work out pretty well for Galen and I. We’ve been able to re-align what we’re working on to more match our interests (and for me, my thesis work), so that’s happy.
  • I discovered that Pete is still updating his log, so now I have another log to read when I’m wasting time. Yay Pete.
  • It’s ski season! Mary took my skis from B’ton to South Bend and I brought them back from Christmas break with me. Went skiing last Sunday at Ski Santa Fe, which is about an hour drive from my apartment in Los Alamos. My legs remembered what to do much quicker than I expected. Snow wasn’t great, but I won’t complain for such a short drive.
  • I’ve been working with Galen on getting Open MPI to operate properly on heterogeneous environments, particularly using OpenIB between Linux boxes running PowerPC and Opteron processors. This was mostly motivated by the Road Runner machine purchased by Los Alamos, which is a combination of Opteron and Cell blades, likely connected with InfiniBand. It’s been a month long process, with lots of code audits about padding, alignment, and endian swapping. I’ll write a post about the gory details really soon.
  • On Friday, January 19th, I’m moving to Albuquerque. My stuff should show up from Bloomington a week or two after that. I’ll be commuting up to Los Alamos every day in N813T. Woo. One thing this means is more compute time. Another is that I can finally get high speed internet. Working at home with dialup has been interesting and I think I’ve adopted pretty well, but I can’t wait to be able to download stuff *now*, rather than while I sleep.
  • Thesis proposal is going entirely too slowly, but I’m still working on it. Hopefully get the proposal scheduled next week, for sometime in early March. Depends on how travel falls out.

{C++, Objective C, Fortran 77, Fortran} link compatibility with C

A not-unfrequent problem we have with Open MPI is users specifying incompatible compiler flags for the different languages used in Open MPI. This frequently occurs when a user wants to build 64-bit on a system that defaults to 32-bit and provides the right CFLAGS, but skips one of the FFLAGS, FCFLAGS, CXXFLAGS, or OBJCFLAGS. If you’re lucky, you’ll get an amorphous error during configure time saying something completely unrelated to the original problem went wrong. If you’re unlucky, nothing bad will happen until you try to link ompi_info, which is one of the last things that gets built.

I wrote a macro for Open MPI today that tries to link a small C function in a .o file with a main function in either Fortran (either 90 or 77) or one of the C-like languages (C++ / Objective C) to see if they link. The result is not pretty, and I’m hoping there’s a better way to do this, but this is what I came up with (below). The only part I’m really not happy with is all the code to deal with the Fortran calling. I’m sure there’s a better way than defining a particular m4 variable then looking at that for the rest of the function. But such is life…

dnl -*- shell-script -*-
dnl
dnl Copyright (c) 2006      Los Alamos National Security, LLC.  All rights
dnl                         reserved. 
dnl $COPYRIGHT$
dnl 
dnl Additional copyrights may follow
dnl 
dnl $HEADER$
dnl

# OMPI_LANG_LINK_WITH_C(language)
# -------------------------------
# Try to link a small test program against a C object file to make
# sure the compiler for the given language is compatible with the C
# compiler.
AC_DEFUN([OMPI_LANG_LINK_WITH_C], [
  AS_VAR_PUSHDEF([lang_var], [ompi_cv_c_link_$1])

  AC_CACHE_CHECK([if C and $1 are link compatible],
    lang_var,
    [m4_if([$1], [Fortran], 
       [m4_define([ompi_lang_link_with_c_fortran], 1)],
       [m4_if([$1], [Fortran 77],
          [m4_define([ompi_lang_link_with_c_fortran], 1)],
          [m4_define([ompi_lang_link_with_c_fortran], 0)])])
     m4_if(ompi_lang_link_with_c_fortran, 1,
       [OMPI_F77_MAKE_C_FUNCTION([testfunc_name], [testfunc])],
       [testfunc_name="testfunc"])

     # Write out C part
     AC_LANG_PUSH(C)
     rm -f conftest_c.$ac_ext
      cat > conftest_c.$ac_ext << EOF
int $testfunc_name(int a);
int $testfunc_name(int a) { return a; }
EOF

     # Now compile both parts
     OMPI_LOG_COMMAND([$CC -c $CFLAGS $CPPFLAGS conftest_c.$ac_ext],
       [AC_LANG_PUSH($1)
        ompi_lang_link_with_c_libs="$LIBS"
        LIBS="conftest_c.o $LIBS"
        ompi_lang_link_with_c_fortran
        m4_if(ompi_lang_link_with_c_fortran, 1, 
          [AC_LINK_IFELSE([AC_LANG_PROGRAM([], [
       external testfunc
       call testfunc(1)
])],
             [AS_VAR_SET(lang_var, ["yes"])], [AS_VAR_SET(lang_var, ["no"])])],
          [AC_LINK_IFELSE([AC_LANG_PROGRAM([
#if defined(c_plusplus) || defined(__cplusplus)
extern "C" int testfunc(int);
#else
extern int testfunc(int);
#endif
], 
             [return testfunc(0);])],
             [AS_VAR_SET(lang_var, ["yes"])], [AS_VAR_SET(lang_var, ["no"])])])
        LIBS="$ompi_lang_link_with_c_libs"
        AC_LANG_POP($1)],
       [AS_VAR_SET(lang_var, ["no"])])
     rm -f conftest_c.$ac_ext
     AC_LANG_POP(C)])

  AS_IF([test "AS_VAR_GET([lang_var])" = "yes"], [$2], [$3])
  AS_VAR_POPDEF([lang_var])dnl
])

AC and AM version numbers

I ran into a problem with Open MPI’s build system today and have come up with a reasonable, but sub-optimal solution. The basics of the problem are that we need to support both the combinations of Autoconf 2.59/Automake 1.9.6 and Autoconf 2.60+/Automake 1.10+ in a project using Objective C (only for a very small, optional component). AC 2.59/AM 1.9.6 have basically zero support for Objective C, but AC 2.60+/AM 1.10 have full support for Objective C. I had back-ported in a bunch of macros to get the bare minimum support we needed for Objective C with AC 2.59, but that all conflicted with AC 2.60, so needed to be optionally provided. So there were two problems:

  • How to determine whether to provide compatibility AC_PROG_OBJC or use AC’s macro
  • How to prevent using AC 2.60 with AM 1.9.6

The problem with the first one is that looking at AC_PROG_OBJC seemed to cause all kinds of entertaining things to happen with AC 2.60. So the solution there was to peek at the internal structures for AC and get the version number, then act accordingly. After figuring out all the macros I needed to provide, this worked reasonably well. Look at config/ompi_objc.m4 to see which ones we ended up needing.

Automake doesn’t seem to provide an internal macro with its version number, so I couldn’t do the obvious to find the version number and do all the logic to make sure we never see AC 2.60 used with AM 1.9.6 (which would give us half support for Objective C and be a major pain). So the best I could come up with was to use the version requirement checking feature of AM_INIT_AUTOMAKE:

m4_if(m4_version_compare(m4_defn([m4_PACKAGE_VERSION]), [2.60]), -1,
  [AM_INIT_AUTOMAKE([foreign dist-bzip2 subdir-objects no-define])],
  [AM_INIT_AUTOMAKE([foreign dist-bzip2 subdir-objects no-define 1.10])])

It’s really vile, but it does work…

Heterogeneous Open MPI

Committed a reasonably large patch to the Open MPI development trunk last night so that the datatype engine does a reasonable first approximation of the “right” thing in heterogeneous environments. Right now, it deals with endian differences and the differences in representation (size and such) of C++ bool and Fortran LOGICAL. Some work still needs to be done, such as dealing with different representations of long double values. The run-time environment and PML code had already been fixed up to be endian-clean. Still have one issue to fix in the run-time layer before mixing 32 bit and 64 bit code will work properly, however. Hopefully, that will come in the not too distant future.

This work won’t be part of Open MPI 1.0.2, but it should make it into some release down the line, most likely 1.1.

Open MPI quickies

Some random quickies on the Open MPI front that may or may not interest you…

  • Jeff has left the building. Well, Jeff left the building a long time ago, but he’s leaving IU really soon now to take a job a Cisco. More information on his blog.
  • The group has gotten a lot bigger lately. New members include:
    • Cisco Systems (who hired Jeff away from us to work on Open MPI)
    • Voltaire
    • Sun, who has a couple blog posts on the move here and here.
  • The one-sided implementation in Open MPI continues to be debugged, but I have high confidence it will be ready to ship as part of Open MPI v1.1. Woo hoo!
  • George has gone on a performance binge lately, which has resulted in much lower latency for all our interconnects and much better bandwidth for our shared memory transport.

We’re done now, right?

Committed implementations of MPI_Win_lock and MPI_Win_unlock yesterday, which I believe are the last two functions from MPI-1 and MPI-2 for which Open MPI did not have implemented. Now all we have to do is make it bug free, make it go faster, and add a bunch of cool research projects so we can all graduate. Not necessarily in that order.