We brainstormed with Network Operations to get a better idea of how both groups would like a virtual server environment to work. The gist was that ops wanted to own the base OS and application/host config, and engineering wanted to own the application.

There are tools that make it relatively straightforward to build and update servers in such an environment, and virtualization adds the flexibility that disk images for use by a virtual server instance can be created on-demand. Where a sysadmin might previously have had to drive to the colo and rebuild a disk, a script could accomplish the same remotely with much less time and effort.

As we started prototyping a virtual server architecture, it occurred to us that we might be able to use a mostly read-only runtime configuration. Our production systems were based on CentOS 5.x which already had support for running from a read-only root filesystem using bind mounts for files and subdirectories that really need (or want) to be writable at runtime /etc/fstab, such as /etc/fstab. As an added benefit, the way that CentOS figures out how to mount the filesystems that contain read-write resources for bind mounting gave us an idea for implementing the separation of ownership goal.

With virtual servers, this model is even easier because what appears as a disk device to a virtual guest OS can be a disk image on the host. All virtual servers can share a single copy of an authoritative base OS image and application image, and each virtual server only needs very small images for application and host configuration. Changing the OS or application images can be done by updating the virtual server's configuration file and restarting the server. Changing application or host configuration can be done by modifying the server's configuration images and restarting the server.

As we adopted this model, we also scripted most of it to make it easy to create and update servers in our application stack. In the end, we had a script that could generate virtual instances for all services in our application stack and have everything up and running in less than five minutes.

One of the unintended benefits of this approach was the flexibility it gave the development team in our test environments. Previously it had been difficult to allocate hardware for testing roles; in a virtual world we needed only a few real servers and were free to define and bring up and down virtual servers as testing needs dictated.

We were also able to define a application personality for a simulated client that ran in a very small (32-64MB) memory footprint and run 10-15 virtual clients on a modestly equipped PC. By repurposing retired displays, we were able to load-test our development servers stacks in environments very similar to production with virtually no hardware costs.

When I started in 2004, the system used CVS for content management and RPM files for content distribution. CVS made sense as all content changes were managed in a CVS repository; but as the network grew, the use of RPMs for distribution became problematic. Every time a new version of a channel was pushed out, the server created a new RPM file which each display transferred and installed. In most cases, channel updates involved changes to a small subset of files, but the displays had to transfer all files as part of the RPM file even if they already had a copy from a previous version. This was especially annoying when the display's network connection was slow or intermittent and the transfer failed at 95% and had to restart from the beginning. So it was time for something new...

I found that git already had most of the functionality we would need. We could use it to store multiple versions of content in a repository on the server, to transport content files to each display, and to install new and updated content in the filesystem on each display. There was one thing missing though - our displays communicated to the central server using HTTPS, and the git HTTP transport didn't support some of the SSL configuration we needed. Since git is open source software, I set foot on a very slippery slope and started submitting changes. In the beginning, it was just additional SSL settings. Later I delved into the internals of HTTP transport and added other features that Reactrix would need, some that I thought Reactrix would someday need (I was wrong...) and some that just seemed cool. In the end, we were able to swap in a new content management and distribution system that performed well, used much less bandwidth to each display, and was much more tolerant of periodic network hiccups.

Since then we switched from Perforce to git for the projects my group worked on (other engineering projects stuck with Perforce, and the content folks still use CVS.) We also incorporated git into a CMS redesign as a more generic mechanism for moving data between the server and displays. Another coworker, Scott Chacon, got a little crazy about git and has since become involved in related software development and training. If you're interested in learning more about how git works, I'd highly recommend watching his presentation from Railsconf 2008.

When my friend Vijay invited me, he wrote that they could cover Ventures, Pink Floyd, Bach, Yes, Beethoven, and Queen. They played all that and more, and stuck around to enjoy some quality Belgian beer, sign CDs, and talk.

If you get a chance to see them, go - you won't be disappointed.

I've always liked learning on the job, and I had plenty of opportunities to do so. There was very little turnover in Engineering, and I got to work with and learn from some smart people for over 4 years.

The first major change for me was moving from systems administration to software development. My original title there was Systems Engineer and I later moved on to Lead Systems Engineer and Manager of Platform Engineering. Suffice it to say that this change was absolutely for the better.

For the first couple of years, I spent most of my time continuing development of a Perl client/server app that managed a network of remote Linux systems. As that product was nearing the end of its life I was able to architect and develop a new version of the application, which involved picking up a bunch of new skills including git, OSX, Ruby/Rails, Xen, Scrum, and a bunch of other related stuff.

I'm not sure where I'll end up next, but I know the Bay Area has no shortage of smart people working on fun and interesting projects in very similar development environments; and I'm excited about the prospect of being part of that.

I still have not seen any Bernese Mountain Dogs, and in the words of a gentleman I passed at Queen's Day in Amsterdam, "het f**king regent" (translation: "it's raining".) I've done a bit of walking about the city, and know where to find a few ATMs and how to get Swiss Francs out of them, I know where to buy crossbows, paintbal guns, spiderman costumes, large stuffed bears, umbrellas, possibly new digital cameras, and decent pizza (thanks elderly guy!)

Just before dark, I took Fezzik to Crestview Park for a little exercise and training. I noticed a new plastic fence at the other side of the park and thought this was rather cool as Fezzik has been known to venture in to that area and this seemed like an easy (for me) way to keep him out. Surprise #1 was that the fence was there not only to keep out the likes of Fezzik and the local coyote population, but to keep in a herd of goats recently employed to rid the hillside of all vegetation. I further noticed that this plastic fence had little yellow warning signs stating that the fence was electric. As I was unable to relate the nature of this warning to Fezzik, he unfortunately discovered surprise #2 on his own by sniffing around the fence while I was busy pulling burrs out of his tail. He voiced his surprise rather loudly which attracted surprise #3 - two very large Great Pyrenees dogs who communicated to us that if we presented any danger to the goats, they would not hesitate to get all medieval on our asses.

Oh, and surprise #4: I learned a super fast and easy way to fold t-shirts. I love the internet...

One of the receipts that caught my eye was from a business trip I took to New York City in August 2001. It turns out that I'm relatively boring when I travel, as even though I may hit one or two of the more obvious tourist attractions (in this case, it was the Metropolitan Museum of Art) I also tend to sit in my hotel room and read alot.

A few years after this trip, I went back and Judi went along. She tends to have an itinerary of interesting places to visit, and I'm always glad to follow her around as I would otherwise miss out on plenty of cool sights. During this trip, she was flabbergasted that I'd been to New York City at least twice and not only hadn't visited the World Trade Center but had no idea where it was and had never seen it (I do find myself wondering how I managed to miss a couple of 1300' buildings.)

That trip in August 2001 was clearly my last chance to see the WTC, but I am somewhat comforted that I probably won't look back on silly missed opportunities like that as long as I include Judi in all future travel plans.

Except to Merrimack, NH. That place is boring.

Lately I've been playing with a few online music services; I'm still figuring out what works for me but so far I'm encouraged. Audioscrobbler tracks what I play (compatible music with compatible formats, at least) and after quite a bit of lag produces a list of similar music and people with similar tastes. liveplasma provides a search tool for music artists (and movies) which offers an interesting visualization with links to similar artists. Yahoo Music provides access to a large music library along with search and recommendation tools, and sells music for $0.99/track ($0.79 for subscribers.) If you subscribe to Yahoo Music you can download most anything in their library to play when you like, although they use Windows WMA format which is not universally supported - of particular annoyance is that the iPod cannot currently play this format and I'd really like to buy one.

While none of these services does it all, I find that using the three together has made it much easier for me to discover artists and buy what I like. I very much like that Yahoo's service allows full access to the library rather than just 30-second samples, but I'm currently stuck without a portable music player because of the WMA format problem. The iPod makes the most sense for a variety of reasons but I have yet to find word on whether that will ever work - to be fair, this problem could also be solved by Yahoo using an iPod-compatible format instead of or in addition to WMA.

For now, I've got a way to find new music and at least listen to it in the car by burning what I like to CD.

Now we just need to find someone that wants a free aquarium.

After graduation, I went to college in the area and learned computer science, then moved to Grand Rapids, MI and worked in IT for 8 years, and finally ended up being a geek in California. It was interesting to hear what some of my classmates have been up to since graduation. Some are still in the area, some have moved around the country a bit, most have kids, and I don't think anyone else ended up in a tech job.

One of my classmates ended up in the Bay Area for business the week after the reunion so I had a chance to catch up some more. He's about to retire after 20 years of military service and has two kids, 13 and 16 years old. That caused a bit of reflection on how different my life could have been.

The whole experience did inspire me somewhat. With apologies to The Village People, this is what came to me:

On the surface, Moore certainly succeeded in making the Bush administration and Congress look bad. He traced a line of people who know people to get from Bush to the Saudi government and tied this relationship to a coverup since so many of the 9/11 hijackers were Saudi citizens. He showed members of Congress unwilling to send their children into harm's way. He exposed the dark side of the war in Iraq with disillusioned soldiers, prisoner abuse, and Iraqi families devastated by civilian casualties.

Some straightforward fact checking provides explanations that contradict many of Moore's assertions and demonstrates the extent to which he twisted events to support his point of view. Regarding elements of the coverup such as a blacked-out name on National Guard records, it turns out that federal laws require this. Regarding Congress's unwillingness to have their children serve in the military, it turns out that not only have a high percentage of members of Congress served, but several of them currently have children in the military (Moore quoted statistics of only those serving in Iraq.) It also turns out that one of the senators who Moore showed walking away doesn't even have any children. Moore took quotes from the Bush administration out of context in some cases to suggest they were making the completely opposite point. While the points he raised regarding the military in Iraq were valid, I had to wonder whether this was also a one-sided view designed to prove Moore's point. I assume that there are also soldiers who are not disillusioned with the war and who find prisoner abuse repulsive, and that there are Iraqi families who are glad that Saddam is gone; but that Moore left this out of the film because it didn't agree with his thinking.

What bothers me the most is the idea that viewers may accept this as a documentary. I believe that there are important questions regarding the 9/11 attacks and the war in Iraq, and hope that someone is able to ask them and truthfully represent all sides of the story. If nothing else, Moore demonstrated that there is another side to what we've been seeing so far.

I understand that Moore hoped the film would turn voters away from Bush this November. I had already been seriously considering this after reading Al Franken's book . I'm not sure how other viewers are reacting, but after seeing 9/11 I've found myself turned off by the anti-Bush camp. I realize that Moore is not their official representative, but I have also realized that I don't know why Kerry/Edwards is a better choice. At least I have a few months to try and figure that out...

If this defense can be applied to software, the situation can be worse with closed source. If the government specifications were not precise enough to disallow a system compromise due to malicious code, not only will it be much more difficult to determine the nature of the problem but the vendor won't bear any responsibility and will be the only source for a fix. The situtation can be improved somewhat by limited release of the source under something like an escrow agreement; but in any case, open source will certainly allow more people to see the code faster in the event of such a compromise.

In the context of military defense systems, it seems to come down to this: with systems based on open source we must trust the government to protect systems from compromise by providing technical resources that can insure system integrity; and with systems based on closed source we must trust the government to insure responsibility in the event of compromise by providing technical and legal resources that can insure bulletproof system specifications.

Later on, I enjoyed some live music at Slim's with Judi, Vincent, and Patrick. Petty Booka opened: ukelele punk from Tokyo. I found it amusing that with some of the songs they covered I could understand the lyrics better than the original even though their native language is Japanese. We now own two of their CDs.

The main act was Aquabats, who I hadn't seen or heard of before. I'm sure I would have enjoyed the music more if I'd heard their songs before, but I sure liked the show. They didn't seem to take themselves too seriously (how could they, with those uniforms?) and their background video was great. I think my personal highlight was their spreading punk culture to the next generation by getting the little kids in the audience to stage dive (the smallest one with more than a little help leaving the stage.)