Author Archives: bbarrett

To Munich we go!

Branden: “Let’s go to Munich to visit Pete during our birthdays.”
Brian: “Sounds like a plan.”

Ok, maybe that isn’t exactly correct, but close enough. A couple of months ago, Branden Moore and Pete Rijks convinced me to go on a birthday trip to Munich, Germany, the current home of Pete. I needed a vacation and getting out of the country seemed like a good idea. While you won’t read it until I’m safely on the ground in Albuquerque, I write this log of the trip while at 37,000′ over the North Atlantic in a United 777 in desperate need of an interior refresh, but safely cruising at 470 knots. During the trip, I’ve drank lots of really good beer, had a traditional Bavarian breakfast, lost a game of Risk to a Brit, driven 200 kph on the Autobahn, and remembered just how allergic I really can be to cats. A more detailed review requires rewinding to Wednesday, March 28th…

Pictures of the trip are on my Gallery.

Wednesday, March 28th

In my limited experience, the worst part about any trip to Europe is getting there. For this trip, I’d be taking the 9:55 United flight from Albuquerque to Dulles, then a 4.5 hour layover, then the 19:55 Lufthansa flight from Dulles to Munich. In typical fashion, United was unable to get me a seat assignment on the Lufthansa flight, even though they code share and the flights were bought on United.com. I later found out they also screwed up the tickets somehow such that I didn’t have a confirmed seat and only by the heroics of Lufthansa did I get on the plane at all. No problem, I’ll deal with that in Washington, DC when I get there. Flight from Albuquerque to Washington was uneventful. The United 737 had the feature where passengers could listen to the flight deck during the flight, so I tuned that in for a while. Before I became a pilot (and even early in my training), I had done this a couple of times and been totally lost by what they were saying. Now it was completely different — I could visualize where everyone on the radio was and why ABQ departure was moving us where they did. ABQ center was quiet as always, so I plugged in the iPod and started thesis writing. Soon enough, we’re descending through 10,000′ into Dulles — time to shut down the laptop.

Arrive in Dulles and find I can’t deal with the lack of a seat for 2 hours so Lufthansa can deal with earlier flights. The Dulles international terminal, unlike the domestic terminal I last spent a layover in, is very clean and modern looking. $8.00 buys me wireless internet through AT&T and I find a seat with power in cord’s reach, so time to catch up on thesis work and download some data I realized wasn’t on my laptop. While I hate paying so much for internet access, in this case it allowed me to grab some data I’d need for the a section of the thesis proposal I need to get written by end of the trip. The food selection appears to be greasy fast food, expensive big sit-down, or subs. Not wanting a huge expensive meal or fast food, I go for the subs The place is called Potbelly’s and it appears the Italian sub is their specialty. For the price it gets about 2.5 out of 5 stars on the airport restaurant scale. After writing more on the thesis, I manage to time the line just right so that I’m 3rd in line for our flight (as the previous flight finishes boarding). The Lufthansa gate attendent was able to straighten things out and even managed to get me an aisle seat.We board (late, as usual) and get push back from the gate. An Airbus A340 fully loaded with passengers, luggage, baggage, and transatlantic fuel has a really long takeoff run, but we make it and are now airborn. After a mediocre pasta dish for dinner, I nod off for the night with the help of some Tylenol PM…

Thursday, March 29th

After 5 hours of mediocre sleep on the plane, I wake up as breakfast is being served. The breakfast was a small roll with a piece of meat, a slice of tomato, and some lettuce, which is apparently a standard breakfast sandwich in Germany. It wasn’t too bad and the accompaning caffeine helps to wake me up. After a smooth landing and a long deplaning (I’m in the back of the plane, after all), I quickly grab my luggage, clear customs, and find Pete and Branden waiting for me in the airport terminal. But I’m now in Germany and almost awake. Woo!

On the trip from the airport to Pete’s apartment, we stop at a little grocery store to pick up some more Coke Light, some snacks, and some fresh bread for lunch. The grocery store was entertaining, although it didn’t take long to realize that I was going to be able to read next to nothing for the next five days. Becoming functionally illiterate overnight is a frightening prospect, but at least I had Pete to translate. After arriving at Pete’s very nice apartment, lunch is had, I take a quick shower, and we’re ready to conquer Munich.

Pete hadn’t thrown a football since coming to Germany last year, so we went out to the English Gardens along the river and find a field. The gardens is a huge park that runs along the Isar River for most of the city of Munich. Some strange looks were received while we stupid Americans threw around an odd-shaped ball, but it was definitely entertaining. A quick light rain shower ends the game, although we were probably ready to quit already anyway. Next came finding an air mattress, as Pete doesn’t usually have to sleep three guys in the apartment. The store used to be a WalMart, but is now a German company. Apparently, WalMart tried to run the place like they run the American stores and the results were predictably unsuccessful. The concept clearly works in Germany as there are many super-store style stores in Germany, but the implementation was just a little off.

For dinner, we wander into the center of Munich, which like most of Europe has some really old architecture. Moving around the city is trivial, thanks to a wonderful public transportation system. We have dinner at the Augustiner restaurant on Kaufingerstraße, a beer hall that is in the tourist books and fairly popular. My dinner is a plate of different types of sausage and potato salad. While I was expecting boring food in Germany, I was pleasantly surprised. But I really like meat and potatos, so maybe I’m just not picky. Oh, and dinner, like most of our activities in Munich, involves beer. In this case, a nice half-liter glass of a Dunkel. Sara, Pete’s good friend, joins us for dinner and is definitely fun. After a fairly long meal of catching up, we head back to Pete’s apartment to crash. I fall asleep in no time.

Friday, March 30th

Friday was Munich tourist day. We started with the Olympic Park from the 1972 Olympics, which is still largely in place. Munich worked very hard to make sure all the infrastructure created for the Olympics was later usable for the general populace. Part of the Olympic park area is a large hill which was created after World War II from the rubble that remained after the frequent bombings of Munich. The hill allows an amazing view of the city and is a reminder of just how devastating the war was. Not quite as much as the US cemetary in Normandy, but still staggering.

How does one follow up a walk around the top of the hill and the beautiful view of the city? With a stop in one of the beer gardens in the park, of course. Fries and Weisen beer are consumed. Both are yummy. Weisen beer in Munich is available in liter or half liter mugs — I go with the half liter, as we haven’t even had lunch yet.

One of the sights nearby is the headquarters of BMW. It’s a beatiful building, with the white and blue logo visible from the top. Nearby is the BMW museum, which we decide is worth a trip. The history of BMW is actually quite amazing — it started as an airplane engine company and grew into cars after World War I. The BMW engines were so good and the allies so fearful of Germany rebuilding an armed forces that they forbid BMW from building airplane engines. BMW started with bicycles, then motorcycles, and eventually cars and was generally successful. The cars at the museum were impressive, although the collection was pretty small. A much larger museum that will have a much larger collection and much more history of the company is under construction, in a crazy looking building that looks to be about 1/3rd complete.

Having seen Olympic Park, we start wandering again. Pete takes us to a huge desolate field with nothing but parking lots on the outskirts and a subway stop. It’s the location of the Oktoberfest. During Oktoberfest, it would be jammed with over a hundred thousand people, all drinking large amounts of beer in what must be the largest beer garden ever created. But today, it’s empty and we quickly return to the subway in search of a more active beer garden (read: one with beer being served).

We head back to the English Gardens, this time closer to the city center. There’s a beer garden there at the Chinesischer Turm, or Chinese tower. During the weekend, the beer garden is packed, but today it is lightly populated. We grab beer and pretzels and sit down for a break. This time, I believe we went with a Helles, which is amazingly smooth. Pete promises a return on the weekend, when a oompa band will be playing from the tower.

There’s a small restaurant / beer garden, Sankt Emmeramsmühle, next to the English Garden near Pete’s place, and we head there for dinner. I get potato soup, salomi, onions, and cheese “salad”, and a weizen beer. It’s all wonderful. The meat salad is especially entertaining, and royally defeats me — there was no way I could finish all that food.

The day was long and full of beer, somehow we decide that we should watch Josie and the Pussycats before calling it a night.

Saturday, March 31st

Saturday I wake up feeling like crap. I (rightly) assume that Pete’s cat has sent my allergies into overdrive. I forgot my meds, so we stop at a pharmacy on the way to breakfast. Unlike the US, almost everything in Germany is behind the counter, even the non-prescription stuff. I explain what I need, happy that the pharmisist knows English, and walk out with the equivalent of Claratin. It takes a bit, but I feel better as the day goes on and Ghandi the cat fails to cause further problems.

Breakfast was at Hofbräuhaus, a famous beer hall that serves Weißwurst, a traditional Bavarian breakfast of white veal sausage, a pretzel, and beer. I’m feeling really bad still, so I go with Orange Juice instead of beer. The scorn from the barmaid was as expected, but I wanted to be able to function for the rest of the day and that wouldn’t happen with beer. Breakfast was excellent, and Sara even showed up in a traditional Bavarian outfit, which made things even better.

The rest of the day was spent at Deutsches Museum, a large Smithsonian-like museaum in Munich. We were there for five hours and saw maybe a tiny fraction of what there was to offer. Their computer and technology wing includes some historic old computers, including a Cray-1. I now have a picture of me using the Cray-1 as furniture, which is somehow wonderfully geeky. There’s also an amazing demonstration of the power of electricity, including simulated lightening setting wires on fire and the like. Wish I would have had another three to four days to wonder, but there is more to see in the city.

Sara holds a game night for a bunch of friends on Sunday evening, and we are in charge of beer. For €36, we bought 20 liters of beer and three sodas, including about €6 of deposits. Unlike the US, beer bottles in Germany are reused and can be purchased in milk-crate-like crates of 10 1 liter bottles. Yay Munich beer prices! Sara makes lots of finger food and we goof around until everyone arrives. A game of Risk among four teams of two starts and consumes most of the evening. In the end, Sara and David (a Brit in Munich on an archeological post-doc) win after taking Australia, Africa, and South America. Branden and I manage to squeak out a second place finish by narrowly defeating Pete’s forces in Asia. Sometime after midnight we head home, as tomorrow is Palm Sunday and we’re heading to 10:00 mass at the cathedral in Munich.

Sunday, April 1st

Sunday was Palm Sunday, so we started with mass at Frauenkirche. It was a very long mass, over two hours, but was interesting to watch. Of course, since I don’t speak any German, I didn’t really understand most of it.

From there, it was off to Schloss Nymphenburg, a rather large palace built over about 200 years from the mid 1600’s to the mid 1800’s. It is on a huge amount of land, which is currently a free park that is popular for running and the like. Two geese got in a fight, which was mildly entertaining. The whole grounds were beautiful.

From there, back to the Chinese Tower to experience the Oompa band. The place was a mad house compared to any of the beer gardens we saw earlier in the week. More beer was had by Branden and Pete. Oh, and of course, there were pretzels. The beer garden was amazing with the number of people hanging out on a Sunday afternoon. I think I could definitely get used to that kind of a social scene.

Dinner was at Brauhaus zur Dicken Sophie, another traditional Bavarian restaurant. This was the only place we went the entire trip that did not have an English menu. Making it more entertaining, the menu used a lot of Bavarian words that Pete had trouble translating. So it was a bit of a surprise when we got our food — we thought we knew what we would get and were pretty close, but weren’t sure until it actually showed up. I had a really good steak with mediocre mashed potatoes. By the way, Dicke Sophie roughly translates to Fat Sophie.

Monday, April 2nd

My birthday. I’m 28. Ugh.

Today, we leave Munich and wander other parts of Germany. Our primary destination is Neuschwanstein Castle, in the Alps near Austria. The best way to get there is to drive and this is Germany, so we splurge and rent a Mercedes Benz E-class. Since we can, we put all three of us on the rental agreement and Pete drives us out of Munich. I start in the back seat, which will become known as the seat of sleep. Neither Branden nor I will spend more than 15 minutes in that seat and not fall asleep. I fall asleep before we’re out of Munich and wake up as we’re pulling into our first stop.

Our first stop was to be the Dachau concentration camp memorial. Unfortunately, we can’t read and failed to notice that it isn’t open for tours on Mondays. This is both good and bad. On the bad side, we have to skip a reminder of a hugely important part of history, which still causes global conflict today. On the good side, it’s more time to drive on the Autobahn, and there’s no feeling guilty for being alive. So we start towards the Alps. Pete gets us further out of the city area and then hands the wheel over to me.

Autobahn driving is basically the same as travelling on the US interstate, but the “rules of the road” are more strictly enforced and actually obeyed. There is never any passing on the right and you are expected to get over for faster traffic. When in unlimited speed areas, spacing from the car in front of you is expected. Finally, the areas with unlimitted speed limits are maintained in immaculate conditions. Cracks are fixed by replacing large blocks of pavement, rather than patching the crack. The road is smooth and well marked. Contrary to popular belief, much of the Autobahn has a speed limit — today approximately 1/3rd of Autobahn roads have a speed limit (generally 120 kph, although lower where pavement conditions or traffic call for it). The rest of the EU wants to make the entire system limited to 120, although that is unlikely to happen in the near future. In fact, Germany is expanding much of the high traffic area to include an extra lane to remove the traffic condition speed limits in many areas.

For the first part of the trip, we’re in areas limited to 120kph (about 75 MPH). I get the feel of the Benz, which is large but very smooth. We get off the Autobahn between Munich and Stutguart to continue to Neuschwanstein, which is much less frequently used and has no speed restrictions. The Benz is comfortable cruising at about 160 kph (about 100 MPH) and that keeps me in the flow of traffic for a while until the traffic thins out. When we get to lower traffic, we start cruising at a bit over 200 kph (125 MPH), which is about as comfortable as I feel driving the rental car. After a bit under an hour, we find our turn off Neuschwanstein and resume more normal speeds. A quick drive through the countryside and we’re at the town at the base of the castle. The drive was a blast, and the GPS navigation system in the car convinced me I want one in my next car.

Neuschwanstein Castle was built by King Ludwig in the late 1800s and was intended to be the greatest castle in Germany. It is also rumored to be the Walt Disney’s basis for the castle in Disney Land. It’s about a 30 minute hike from the town to the castle, but the views are amazing the entire way up. Touring the inside of the castle costs a bit of money, so we elect to wander around outside instead. There’s a bridge with a great view from above. Technically, the trail is closed, but it’s pretty clear no one is paying attention to that, so neither shall we. The views are amazing, as were the paragliders hanging out on the thermals on the edge of the Alps. After a couple hours of wandering, we find a restaurant half way between the castle and town. Bratwurst, fries, Weisen beer, and fried doughnut like things for dessert. Very yummy, but time to head back to Munich.

Branden does most of the drive back to Munich, and Pete does the final navigation back into the city. We turn in the rental car (after forgetting to top off the last bit) and head back to Pete’s house to pack up for our departure on Tuesday.

There’s an Irish Pub, Gunther-Murphy’s in Munich that kind of acts as an expat bar, where English is still the expected language. Monday night is trivia night, so we head there for dinner and trivia. The Irish beer selection was Guiness and Kilkenney’s, so I opt for a Munich Weisen. Dinner, however, is a Sheperd’s Pie and it is excellent. Sara, Pete, Branden and I come in 5th out of 10 in trivia. We were in second going into the last round, but were crushed by “movie star birthdays” and dropped quickly. We decided we were proud we didn’t know those things. Headed home early because we had to be at the airport at 7:00.

Tuesday, April 3rd

Tuesday is travel home day. I caught some shut-eye on the first part of the transatlantic flight, so now it’s just staying awake until a reasonable time this evening. Coming home is definitely easier to adjust to than heading to Europe. Trip is fairly uneventful. United screwed up again, but made it reasonably alright after I got to Dulles. Flight from Dulles to Albuquerque is tiring and bumpy. I’m a zombie once I get back to my apartment, but refreshed from a wonderful trip.

Summary

It’s now Wednesday night and I’m finally recovered from travel. The trip was a whole lot of fun and I hope I’m able to go back to Munich to visit Pete again sometime in the future. If not for the whole extremely expensive to fly thing, I could see myself spending a couple years over there. Definitely a fun time, and recommend it to anyone who can go visit Pete for a week or so…

Which way am I pointing?

Galen and I went out to Belen today, to both check out some work we did on the compass and so that I could shoot some approaches. I hadn’t been flying much, and I’m much more rusty than I’d really like. I thought I was getting back in the swing of things, as I made three reasonable landings at Belen, but then had an awful landing at ABQ on runway 3. I came in a little fast, so flared a bit to slow down, over flared and was slow and high, with the nose high. Pushed the nose over and started pushing in power to go around. Air speed came back up, I had 8000′ of runway left, and was in a reasonable attitude, so I pushed the nose while pulling the power, flared again, and the back wheels squeaked on. Nose came down, and we rolled out. In the end, bad part was that I missed the F-1 exit and had to go to F-3, so I had a long high-speed taxi. Contributing factors to the bad landing were:

  • Frustration from problems flying the vectors given from approach due to compass problems.
  • EFIS going insane when I loaded a synthetic approach to 3 into the EFIS and then Galen loaded the ILS approach to 3 in the GNS430 without progressing past the IAF (lots of warnings, but should not have been an annoyance).
  • Approach to a 150′ wide runway after flying approaches to the 60′ wide runway at E80.

I’m not sure why, but I haven’t been comfortable in 813T since we brought it back from Colorado. I’m not sure if it’s having someone in the right seat (the CO time was all solo time while test flying). It’s getting better and I’m now starting to fly ahead of the plane instead of just trying to keep it doing the right thing. But all the little things built up this afternoon to me being a bit behind the plane while landing on 3. At least things are going in the right direction. On the good side, with one exception, I kept the nosewheel off the ground for as long as able. The one exception was the second approach at Belen. Galen had his foot resting on the right rudder pedal. We started drifting to the right after touchdown and I was feeling stiff resistance on the left rudder pedal when I tried to correct. I let the nose wheel down in case I had to brake the left wheel (which is touchy at those speeds). So at least that part seems to be getting better. And takeoffs were very smooth, although I still come out of ground effect faster than Galen likes. I prefer to establish a Vy climb with a shallow climb, while Galen prefers to stay in ground effect until Vy then establish the climb. I’m shallow enough on the climb that Galen appears satisfied on the point, so I’ll take that.

We’ve had a problem with the plane the last week with the magnetic heading reported by the EFIS. On both Monday and Tuesday this week, we were vectored into ABQ by approach. Both times, the EFIS reported a heading +/- 5 degrees of the assigned heading but GPS reported a very different ground track. The ground track matched what the controller was seeing, but the heading matched what the controller expected to see (so we weren’t going the direction the controller expected, which is bad). At the time, the only compass we had in the plane was the EFIS, since it’s a magnetic compass (gyro stabilizd) and that’s all the FARs require. We decided to order a traditional wet compass to help debug the problem and to make sure we had a backup in case of power failure. At this point, it’s probably worth a note on the directions used in planes:

Magnetic heading
The direction the plane is pointed, relative to magnetic north. Generally determined with a magnetic compass and/or a directional gyro calibrated to a magnetic compass. Headings assigned by approach / center controllers are in magnetic heading
True heading
The direction the plane is pointed, relative to true north. Generally determined in flight by finding the magnetic heading and adjusting with the local variance. Aviation charts are all relative to true north, leaving the pilot to convert between true and magnetic north.
Ground track
The direction the plane is actually going, almost always given relative to magnetic north when determined by an instrument (like a GPS), but in true north when doing preflight ground track planning based on charts.

The traditional method for determining magnetic heading is a wet compass, a card with a small magnet sitting in alcohol so that it spins freely and points towards north. Magnetic compasses are hard to use because they move an awful lot just due to vibration in the airplane. They also spin ‘backward’. When you’re pointed north, the compass reads north. To the left on the card is the reading 30, but to turn to 30, you make a right turn. This requires more thought than one really wants in a high stress environment and if the plane is in turbulent air, getting a reading off the compass can be near impossible. So most planes have what is called a directional gyro (DG) installed. The DG doesn’t have any concept of which way is north. But it uses gyroscopes to determine if you’ve turned left or right and how much. So if you set the DG to have a known heading (like the compass reading before takeoff), it will smoothly display the current heading. And it’s mechanically set to have the proper right/left turning orientation. There are some problems, like slow drift over time, but it basically works as long as you calibrate it against the compass every now and then.

N813T doesn’t have a traditional DG, instead it has gyros coupled with a magnetometer and a computer uses both inputs to develop a stabilized heading indicator on the EFIS. If either the gyros or the magnetometer are wonky, then the heading is going to be wrong. The EFIS control software is supposed to detect this and return an error to the pilot. It had been doing this (we the pilots just didn’t know where to look for that error), which caused us to spend most of yesterday working on the problem. We got that problem fixed and so we test flew it today. On departure, we noticed that the EFIS again didn’t match the ground track or mag compass. Odd, but ok. I knew there wasn’t much wind, so I just flew the ground track indicated on the GNS430 GPS (and about 3 other places), and it was fine. On the way home, however, the EFIS heading, compass, and GNS430 all had different headings. Figuring out which to fly was problematic and the source of much confusion for me the pilot. Thankfully, the vectors were pretty simple, so I was able to guestimate with landmarks and get the plane lined up for runway 3 without getting close to other traffic.

On the rollout after the crappy landing (and since I missed the early exit, it was a long rollout), I noticed the wet compass was definitely off and started thinking about something my instructor in B’ton said — that if the compass ever seemed wrong, double check by first turning off high current electrical items. I shut off the landing light and the compass swung right back to the proper heading. Now, I always turn on the landing light when in the vicinity of the airport or when in a practice area to make it easier for other traffic to see me. Not a big deal in N813T, since it has strobes, but is a problem on the C-150 I used to fly. So the entire flight, the landing light was turned on. In testing after shutdown, there is about a 20-30 degree swing in the wet compass caused by the landing light. And the EFIS computer, attempting to compensate for the heading reported by the magnetometer, will eventually report the new (wrong) heading, similar to the wet compass.

Ok, so now we know there was something wrong with the magnetometer wiring which we fixed. And we know the problem persisted, but in a different way. We know the landing light causes that problem. So we now at least understand the problems we’re seeing. I noticed that the ground wires for the landing, taxi, and nav lights were wired to the wing tip rather than back to the central grounding block (close to where the +12V line is routed). This set off alarm bells in my head, so I started a discussion with Perk and Chris about the problem. Sure enough, Perk breaks out some math that seems to indicate that the proximity of the +12VDC line for the landing and taxi lights and the lack of a matching ground line back could cause a magnetic field in the plane that would cause a swing of a compass in the right ballpark. So it looks like we should ground back to the cockpit instead of the wing tip and this might help quite down the magnetic compass interference. Way too much time working on the plane, but it was kind of cool to see math being used to figure out an electrical problem…

In other news, I now have 60.1 hours of pilot in command time and 55.9 hours of dual received time. Which means that I have more time being in charge of the plane than I have being taught how to fly the plane. Note that these times are not mutually exclusive. I have about 9 hours of time that is both pilot in command and dual received, for things like checkouts in new planes. But it’s still fun to think about…

Friday night update

So it’s Friday night, it’s been 6 weeks and 2 days since my last update. Which is a bit pathetic, but I’m sure that everyone will survive without too much difficultly. Anyway, since I don’t remember most of what has happened, we’re going for list format.

  • I now have 99.5 hours of flying time (so close, yet so not 100 hours), including a bunch of flights since my last update:
    • A flight with Branden and Kyle to Taos and back in N80790 just to fly around and log me some cross country time. Branden took some pictures
    • A flight with Laura in early December to Taos for lunch (yummy Mexican) and log me some cross country time. There was still snow on the north sides of the slopes from the November snow storm, so it was really pretty.
    • A bunch of test flights in N813T, Galen’s plane. It’s a blast to fly — very responsive and very quick. The most fun I’ve ever had while flying.
    • A long (~300 nautical mile) cross country flight around Colorado in N813T, solo because it was a test flight. The entire flight was only about 2 hours, since the plane is so fast.
  • Arun, Shelece and I drove down to White Sands in December in hope that Discovery was going to land there (it was looking promising). Just as we arrived at White Sands National Monument, it was announced they were going to land at Kennedy. Good for the NASA budget, not so fun for us. But we did get to run around White Sands, which was fun. We stopped for lunch at a place along the way Arun and Shelece had read about and had one of the best green chili cheeseburgers I’ve ever eaten. Fun trip, but would have been better if there was a shuttle landing overhead.
  • I went home for Christmas break, just barely escaping the giant snow storm in Denver to get to Chicago. Getting home was delayed by yet another snow storm in Denver, but it wasn’t too bad. Just had to spend a boring New Years in South Bend. We spent Christmas afternoon in Detroit, and got to see the Barrett clan, which is always fun. Also got to see Anne while we were in Grand Rapids visiting Cathy and Dan.
  • Rich Graham (the group/team/project leader I was working for) announced that was leaving LANL for Oak Ridge. Adolfy Hoisie took over the group effective January 8th. While a bit scary for a while due to the uncertainty of change, the change should work out pretty well for Galen and I. We’ve been able to re-align what we’re working on to more match our interests (and for me, my thesis work), so that’s happy.
  • I discovered that Pete is still updating his log, so now I have another log to read when I’m wasting time. Yay Pete.
  • It’s ski season! Mary took my skis from B’ton to South Bend and I brought them back from Christmas break with me. Went skiing last Sunday at Ski Santa Fe, which is about an hour drive from my apartment in Los Alamos. My legs remembered what to do much quicker than I expected. Snow wasn’t great, but I won’t complain for such a short drive.
  • I’ve been working with Galen on getting Open MPI to operate properly on heterogeneous environments, particularly using OpenIB between Linux boxes running PowerPC and Opteron processors. This was mostly motivated by the Road Runner machine purchased by Los Alamos, which is a combination of Opteron and Cell blades, likely connected with InfiniBand. It’s been a month long process, with lots of code audits about padding, alignment, and endian swapping. I’ll write a post about the gory details really soon.
  • On Friday, January 19th, I’m moving to Albuquerque. My stuff should show up from Bloomington a week or two after that. I’ll be commuting up to Los Alamos every day in N813T. Woo. One thing this means is more compute time. Another is that I can finally get high speed internet. Working at home with dialup has been interesting and I think I’ve adopted pretty well, but I can’t wait to be able to download stuff *now*, rather than while I sleep.
  • Thesis proposal is going entirely too slowly, but I’m still working on it. Hopefully get the proposal scheduled next week, for sometime in early March. Depends on how travel falls out.

{C++, Objective C, Fortran 77, Fortran} link compatibility with C

A not-unfrequent problem we have with Open MPI is users specifying incompatible compiler flags for the different languages used in Open MPI. This frequently occurs when a user wants to build 64-bit on a system that defaults to 32-bit and provides the right CFLAGS, but skips one of the FFLAGS, FCFLAGS, CXXFLAGS, or OBJCFLAGS. If you’re lucky, you’ll get an amorphous error during configure time saying something completely unrelated to the original problem went wrong. If you’re unlucky, nothing bad will happen until you try to link ompi_info, which is one of the last things that gets built.

I wrote a macro for Open MPI today that tries to link a small C function in a .o file with a main function in either Fortran (either 90 or 77) or one of the C-like languages (C++ / Objective C) to see if they link. The result is not pretty, and I’m hoping there’s a better way to do this, but this is what I came up with (below). The only part I’m really not happy with is all the code to deal with the Fortran calling. I’m sure there’s a better way than defining a particular m4 variable then looking at that for the rest of the function. But such is life…

dnl -*- shell-script -*-
dnl
dnl Copyright (c) 2006      Los Alamos National Security, LLC.  All rights
dnl                         reserved. 
dnl $COPYRIGHT$
dnl 
dnl Additional copyrights may follow
dnl 
dnl $HEADER$
dnl

# OMPI_LANG_LINK_WITH_C(language)
# -------------------------------
# Try to link a small test program against a C object file to make
# sure the compiler for the given language is compatible with the C
# compiler.
AC_DEFUN([OMPI_LANG_LINK_WITH_C], [
  AS_VAR_PUSHDEF([lang_var], [ompi_cv_c_link_$1])

  AC_CACHE_CHECK([if C and $1 are link compatible],
    lang_var,
    [m4_if([$1], [Fortran], 
       [m4_define([ompi_lang_link_with_c_fortran], 1)],
       [m4_if([$1], [Fortran 77],
          [m4_define([ompi_lang_link_with_c_fortran], 1)],
          [m4_define([ompi_lang_link_with_c_fortran], 0)])])
     m4_if(ompi_lang_link_with_c_fortran, 1,
       [OMPI_F77_MAKE_C_FUNCTION([testfunc_name], [testfunc])],
       [testfunc_name="testfunc"])

     # Write out C part
     AC_LANG_PUSH(C)
     rm -f conftest_c.$ac_ext
      cat > conftest_c.$ac_ext << EOF
int $testfunc_name(int a);
int $testfunc_name(int a) { return a; }
EOF

     # Now compile both parts
     OMPI_LOG_COMMAND([$CC -c $CFLAGS $CPPFLAGS conftest_c.$ac_ext],
       [AC_LANG_PUSH($1)
        ompi_lang_link_with_c_libs="$LIBS"
        LIBS="conftest_c.o $LIBS"
        ompi_lang_link_with_c_fortran
        m4_if(ompi_lang_link_with_c_fortran, 1, 
          [AC_LINK_IFELSE([AC_LANG_PROGRAM([], [
       external testfunc
       call testfunc(1)
])],
             [AS_VAR_SET(lang_var, ["yes"])], [AS_VAR_SET(lang_var, ["no"])])],
          [AC_LINK_IFELSE([AC_LANG_PROGRAM([
#if defined(c_plusplus) || defined(__cplusplus)
extern "C" int testfunc(int);
#else
extern int testfunc(int);
#endif
], 
             [return testfunc(0);])],
             [AS_VAR_SET(lang_var, ["yes"])], [AS_VAR_SET(lang_var, ["no"])])])
        LIBS="$ompi_lang_link_with_c_libs"
        AC_LANG_POP($1)],
       [AS_VAR_SET(lang_var, ["no"])])
     rm -f conftest_c.$ac_ext
     AC_LANG_POP(C)])

  AS_IF([test "AS_VAR_GET([lang_var])" = "yes"], [$2], [$3])
  AS_VAR_POPDEF([lang_var])dnl
])

AC and AM version numbers

I ran into a problem with Open MPI’s build system today and have come up with a reasonable, but sub-optimal solution. The basics of the problem are that we need to support both the combinations of Autoconf 2.59/Automake 1.9.6 and Autoconf 2.60+/Automake 1.10+ in a project using Objective C (only for a very small, optional component). AC 2.59/AM 1.9.6 have basically zero support for Objective C, but AC 2.60+/AM 1.10 have full support for Objective C. I had back-ported in a bunch of macros to get the bare minimum support we needed for Objective C with AC 2.59, but that all conflicted with AC 2.60, so needed to be optionally provided. So there were two problems:

  • How to determine whether to provide compatibility AC_PROG_OBJC or use AC’s macro
  • How to prevent using AC 2.60 with AM 1.9.6

The problem with the first one is that looking at AC_PROG_OBJC seemed to cause all kinds of entertaining things to happen with AC 2.60. So the solution there was to peek at the internal structures for AC and get the version number, then act accordingly. After figuring out all the macros I needed to provide, this worked reasonably well. Look at config/ompi_objc.m4 to see which ones we ended up needing.

Automake doesn’t seem to provide an internal macro with its version number, so I couldn’t do the obvious to find the version number and do all the logic to make sure we never see AC 2.60 used with AM 1.9.6 (which would give us half support for Objective C and be a major pain). So the best I could come up with was to use the version requirement checking feature of AM_INIT_AUTOMAKE:

m4_if(m4_version_compare(m4_defn([m4_PACKAGE_VERSION]), [2.60]), -1,
  [AM_INIT_AUTOMAKE([foreign dist-bzip2 subdir-objects no-define])],
  [AM_INIT_AUTOMAKE([foreign dist-bzip2 subdir-objects no-define 1.10])])

It’s really vile, but it does work…

Welcome to Los Alamos National Laboratory

Today I officially accepted Los Alamos’s offer for a position as a technical staff member in CCS-1. In reality, there are only a few changes as a result of changing from a graduate student to technical staff:

  • I get paid more (Woo!)
  • I get vacation and sick days (Woo!)
  • I can take my LANL-owned laptop home without writing a safety plan for my apartment.

And before anyone asks, YES, I STILL INTEND TO GRADUATE AS SOON AS POSSIBLE! It looks like my duties over the next year will include:

  • Writing a health monitoring subsystem for ORTE on non-BProc systems
  • Modifying the BTL interface to better support RDMA communication (For a bunch of stuff, including my thesis)
  • Working with various groups at LANL trying to generate performance data on Open MPI
  • Planning LANL’s Open MPI presentations at SC’06
  • Open MPI support (ugh!)

So same old thing, slightly different title.

Flying is Fun

A couple weekends ago, Kyle and I went for a short flight around New Mexico. So that I could count it as a cross-country flight, we flew from Santa Fe to Taos and back. The flight was pretty smooth and the route was pretty basic. For my first cross-country flight since getting my license, I thought it went pretty well. Kyle took some pictures, available on his website.

Onward to victory!

Last year at Super Computing, Notre Dame had a raffle in their booth for a football signed by Charlie Weis, which I won. Of course, they didn’t have said football with them, but it would be mailed to me soon after the conference ended. I quickly forgot about it, and never heard back from ND. I realized this sometime in the Spring when I was packing to move out to Los Alamos, and figured it was just one of those things. Monday, I got an e-mail from the ND engineering department saying my football had finally been signed and they would send it out to me. Apparently, ND doesn’t mess around with their signed footballs, as it was sent overnight FedEx. So after 11 months, I have a ND football signed by Charlie Weis. Woo!

Sigh….

There have been two articles in HPCWire during the last 2 months on the use of RDMA in HPC, particularly for implementing the MPI specification. The first, A Critique of RDMA, is written by Patrick Geoffray of Myricom. Patrick has worked with the Open MPI team on improving our support for the Myrinet/MX communication interface. The article assumes a good deal of knowledge about HPC, but is a good read if you know anything about MPI. The summary is that RDMA interfaces with explicit memory registration are difficult to use when implementing the matching send/receive rules of MPI.

In response, a chief engineer at IBM (who makes InfiniBand cards, which are RDMA with explicit memory registration), wrote an attempt at a reasonable reply, A Tutorial of the RDMA Model. The article is both sad and insulting to MPI implementors. The article’s opening paragraph is:

RDMA encompasses more than can be encapsulated by a reference to RDMA Writes and RDMA Reads. The reduction of the RDMA programming model by describing a poor mapping over MPI or for that matter Sockets indicates a limited understanding of the extensive capabilities of the full RDMA Model.

The problem with this statement is that you’ll find few MPI implementors that believe RDMA is a good model for implementing MPI-1 send/receive semantics, especially those that have tried. And Patrick has maintained MPICH-GM, an implementation of MPI-1 over the Myrinet/GM interface, which is a RDMA with explicit memory registration interface. It is also clear that Mr. Recio is unfamiliar with the MPI standard and it’s nuances. For example, in response to Patrick’s comments about copying small messages and registration/deregistration usage for large messages, Mr. Recio claims that “long-lived registrations provide the lowest overhead”. This statement is true, but misses Patrick’s point. The following code is perfectly legal in MPI:

char *foo = malloc(16);
[populate foo with data]
MPI_Send(foo, 16, MPI_CHAR, ....);
MPI_Send(foo, 16, MPI_CHAR, ....);
free(foo);

An MPI implementation over InfiniBand (using OpenIB or mVAPI) has a couple of choices to implement MPI send. The straight-forward solution is to pin the buffer, send the data to the remote process, and unpin the buffer. The problem with this is that the registration/deregistration cost will generally be higher than the cost of the send itself. So one option would be to leave the buffer registered and hope the user re-uses the buffer. Ok, so now we get to the free() call. The results of free()ing pinned memory differs from OS to OS, but it’s never good [1].

So what’s an MPI implementor to do? The short message answer starts with a bounce buffer, a pre-registered buffer held internally by the MPI implementation. The data for the send is copied into the bounce buffer, where it is then sent. If the MPI is really trying to get aggressive about latency, it might use RDMA instead of send/receive for the short messages, but it’s still being pushed out of a bounce buffer. On the receiver side, I’ve yet to hear of an MPI implementation over RDMA with explicit registration do anything but receive the short message into yet another bounce buffer. Why? The short message isn’t the only thing being sent. Because there’s only ordered matching for send/receive on these RDMA networks, a MPI-internal header has to be sent as well. Until that header is analyzed, it’s impossible to know where the message is supposed to be delivered.

Longer messages are a different story. There are a number of options. For medium sized messages, a pipeline of copies and sends works well. For large messages (>128K on modern networks), the copy pipeline protocol results in much lower bandwidth than the network is capable of delivering. For optimal performance, it is better to pin the user buffer and RDMA directly into the user’s receive buffer. This can be done by pipelining the registration / rdma / deregistration (an algorithm the Open MPI team has worked hard to optimize and has published on), or by leaving the user buffer pinned, which is how you get optimal bandwidth on NetPIPE. Pinning such a large buffer has high initial cost, so buffer reuse is critical in this “leave pinned” case. A third option, developed by the Open MPI team, is a combination of the two. A registration pipeline is used to optimize the speed of the first send, but the buffer can be left pinned for later reuse. While we implement the leave pinned options, they aren’t the default and have to be explicitly enabled. Why? because of the free() problem described earlier. We have to track memory manager usage by intercepting free(), munmap(), and friends in order to deregister the memory and update our caches before giving the memory back to the OS. This is error prone and frequently causes problems with applications that need to do their own memory management (which is not uncommon) in HPC apps. Other MPI implementations deal with it in other ways (like not allowing malloc/free to give memory back to the OS). These MPI implementations are frequently known for crashing on applications with aggressive use of the memory manager.

The final point that really annoyed me in Mr. Recio’s article was the comment:

For MPI, both Mvapich and Open MPI have moved beyond N-1 RDMA connections and use dynamic and adaptive mechanisms for managing and restricting RDMA connections to large data transfers and frequently communicating processes.

This is true, in that Open MPI has done all of these things. However, in order to implement support for Open IB in Open MPI, quite a bit more work was required than to implement support for MX. Proof can be shown in a simple LOC count (includes comments, but both are similarily commented):

Device Lines of code
Open IB BTL 5751
MX BTL 1780
OB1 PML 6283
CM PML 2137
MX MTL 1260

The PML components (OB1 / CM) both implement the MPI point-to-point semantics. OB1 is designed to drive RDMA devices, implemented as BTLs. The CM PML is designed to drive library-level matching devices (MX, InfiniPath, and Portals), implemented as MTLs. OB1 includes logic to handle the various pinning modes described above. The Open IB BTL includes short message RDMA, short message send/receive, and true RDMA. The MX MTL includes short and long message send/receive. The CM PML is a thin wrapper around the MTLs, which are very thin wrappers around the device libraries. As you can see, it takes significantly less code to implement an MX BTL than a Open IB BTL. The difference is even more startling when you compare the MX MTL/CM PML (3397 LOC) and the Open IB BTL/OB1 PML (12034 LOC). This isn’t exactly a fair comparison, as OB1 includes support for multi-device stripping. On the other hand, the MX library handles those details internally, so perhaps it is a fair comparison.

As an MPI implementor, I dislike RDMA interfaces with explicit memory registration. Quadrics, which can do RDMA without explicit memory registration by linking the NIC with the kernel’s memory manager, offer many of the benefits of RDMA devices without the registration problems. But there are still copies for short messages in many cases. Most importantly, Quadrics is much more expensive than InfiniBand, frequently an unjustifiable cost when building a cluster. Portals offers a good combination of RDMA and send/receive that is extremely powerful. Implementing an MPI is more difficult than over MX, but it is possible to implement interfaces other than MPI, which is a useful feature. MX and InfiniPath offer a trivial MPI implementation, with excellent latency and bandwidth.

There is one good thing about InfiniBand Mr. Recio doesn’t mention. It is so hard to implement an MPI over these devices that two groups (Open IB and MVAPICH) have had great success at publishing papers about hacks to get decent performance out of the interconnect.

[1] On Linux, the memory will be deregistered and returned to the OS implicitly. But the MPI’s tables on which pages are pinned haven’t been updated. So when you inevitably get that page back from the OS for a new malloc() call and try to send from it, the cache will think the page is register it and not try to register it. Leading to the MPI sending from an unregistered page, which frequently leads to incorrect data transmission. On OS X, on the other hand, free() will block until all pages in the allocation are deregistered. Which means you’ll deadlock.