« Looking back: discovering git | Main
November 08, 2008
Looking back: dynamic server appliances with Xen
As Reactrix started deploying the new version of the web application I had been working on, virtualization products like VMWare and Xen were becoming increasingly popular. It seemed like virtualization would be a better way to utilize our server hardware - particularly systems with newer multicore CPUs - and to simplify the process of building and deploying new servers.
We brainstormed with Network Operations to get a better idea of how both groups would like a virtual server environment to work. The gist was that ops wanted to own the base OS and application/host config, and engineering wanted to own the application.
There are tools that make it relatively straightforward to build and update servers in such an environment, and virtualization adds the flexibility that disk images for use by a virtual server instance can be created on-demand. Where a sysadmin might previously have had to drive to the colo and rebuild a disk, a script could accomplish the same remotely with much less time and effort.
As we started prototyping a virtual server architecture, it occurred to us that we might be able to use a mostly read-only runtime configuration. Our production systems were based on CentOS 5.x which already had support for running from a read-only root filesystem using bind mounts for files and subdirectories that really need (or want) to be writable at runtime /etc/fstab, such as /etc/fstab. As an added benefit, the way that CentOS figures out how to mount the filesystems that contain read-write resources for bind mounting gave us an idea for implementing the separation of ownership goal.
The concept was a virtual version of a real server configured with two CD-ROM drives and two USB ports. Such a server could boot from one CD-ROM containing a read-only root filesystem with a base OS image, and automatically detect and mount filesystems on USB flash filesystems for bind-mounting read-write resources. With an application on the second CD the system would be pretty close to our goal. During the boot process, it could automatically figure out its personality by mounting filesystem with a known label and running appropriate initialization for that personality, such as using chkconfig to enable/disable services before the system enters its default runlevel. With a rack full of such servers, certain maintenance tasks could be simplified - for example, switching to a different version or even distribution of the OS could be done by replacing all the OS CDs and rebooting; upgrading or changing the application personality could be done the same way.
With virtual servers, this model is even easier because what appears as a disk device to a virtual guest OS can be a disk image on the host. All virtual servers can share a single copy of an authoritative base OS image and application image, and each virtual server only needs very small images for application and host configuration. Changing the OS or application images can be done by updating the virtual server's configuration file and restarting the server. Changing application or host configuration can be done by modifying the server's configuration images and restarting the server.
As we adopted this model, we also scripted most of it to make it easy to create and update servers in our application stack. In the end, we had a script that could generate virtual instances for all services in our application stack and have everything up and running in less than five minutes.
One of the unintended benefits of this approach was the flexibility it gave the development team in our test environments. Previously it had been difficult to allocate hardware for testing roles; in a virtual world we needed only a few real servers and were free to define and bring up and down virtual servers as testing needs dictated.
We were also able to define a application personality for a simulated client that ran in a very small (32-64MB) memory footprint and run 10-15 virtual clients on a modestly equipped PC. By repurposing retired displays, we were able to load-test our development servers stacks in environments very similar to production with virtually no hardware costs.
Posted by nickh at November 8, 2008 11:38 AM