projects::arch linux on ec2

summary

Arch Linux's minimalistic and user-centric approach makes it a great candidate for compute cloud deployment. It's light, fast, scalable, and (perhaps most importantly) it's fun!

I am doing pseudo-weekly releases of Arch Linux as Amazon EC2 AMIs. I do frequent releases because I'm still refining the build process. I'm eventually going to make my tools public, but I am not quite satisfied with them yet.

current releases

Release 2013.05.16

Release 2013.05.01

recent changes and news

• 2013-02-05

  Nothing too exciting lately. Today's release has Linux 3.7.6.

  I have also built some Debian stable and testing AMIs for x86_64 in each region. You can find them with the EC2 API tools: ec2-describe-images -o 460511294004 -F "name=debian-*"

• 2012-11-22

  New AMI releases, now with cloud-init. Thanks to Jeremy D for contributing his time and effort to making cloud-init work well on Arch Linux.

• 2012-11-12

  Released new AMIs, primarily for the new AWS region in Sydney, Australia (ap-southeast-2). This new build also includes EC2-hosted Arch Linux mirrors for packages, local to each region.

• 2012-11-08

  Today's AMIs are released. Nothing too fancy in this build: just updated packages, including linux-ec2 3.6.6-1.

• 2012-10-21

  I've added a new linux-ec2 package which contains a patched v3.6.2 kernel. There are a few major differences between this kernel and the Arch Linux stock kernel:

  • Hangs on Xen fixed (patches from 3.6.3 stable-queue).
  • CONFIG_PREEMPT_VOLUNTARY instead of CONFIG_PREEMPT, this will allow for better scheduling as a domU.
  • CONFIG_HZ=100 instead of CONFIG_HZ=300, this allows for better performance on many-CPU instances, as there are fewer timer interrupts to preempt other tasks.
  • Many drivers removed, particularly those that didn't make sense for running in an EC2 instance. I've left drivers for my own hardware so I can experiment with it as a dom0 kernel as well. The kernel size is roughly half the stock Arch Linux kernel due to the stripped drivers.

  I am also building new AMIs right now, and am beating the i386 AMIs into working order. Once done I'll publish the next release (which should be 2012.10.21). Once it's available, it will show in the tables above.

• 2012-10-16

  Do not upgrade HVM instance kernels to anything between 3.6.0 and 3.6.2 inclusive. You must wait for 3.6.3 or else your instance will not boot. We're currently waiting on this patch to be integrated into the mainline stable tree. This is also why I am probably not doing an AMI release this week, as the HVM AMIs would be totally broken.

  I've also taken a look at building i386 (well, i686) AMIs. I'm not really sure that it's worth the effort. Nobody really uses 32-bit AMIs anymore, and we'd need to fork the kernel just to make it happen. For now, i686 is on ice.

• 2012-10-14

  Implemented my own fix for FS#31250. The issue was that SSH connections weren't properly closed before shutting down, leaving connections hanging on occasion. The issue seems to be that there's no order dependency in sshd.service with relation to network.target (such that sshd gets stopped before the network goes down). So I've added network.target to the 'After' list in sshd.service. The developers didn't believe me that it actually worked, however. They closed the bug a bit aggressively, so I haven't pushed the issue further.

  The nice thing about doing this in the cloud is that I can do mass tests of this kind of thing with relative ease. I started 100 m1.small instances, and did a series of 10 reboots without changing sshd.service. I got a failure rate of approximately 3% (2-3 instances didn't close the connection on each reboot).

  After modifying sshd.service, and a series of 10 more reboots, I saw a 0% failure rate. I don't know what to say, devs. It works.

release notes

These AMIs are as close to a "vanilla" install as I can make them without making them functionally impaired on EC2. But here's the complete list of differences between the EC2 builds and a stock install:

• High performance kernel specifically for EC2, including paravirtualization support on i386 and x86_64 AMIs, and more Xen-friendly process scheduling.

• Kernel modules included in initrd, some of which are only there for the initial bootstrap (which I do in KVM): virtio virtio-blk virtio-net virtio-pci virtio-ring xen-blkfront xen-fbfront xen-kbdfront xen-netfront xen-pcifront xen-privcmd button ipmi-msghandler ipmi-poweroff

• Packages removed: initscripts, sysvinit

• Extra packages installed: netcfg, openssh, dhclient, ec2-keyring, ec2-pacman-mirrors, rng-tools, systemd-sysvcompat

• Added an extra package source for ec2-specific packages. the repository currently contains: ec2-keyring, ec2-pacman-mirrors, package-query, yaourt. I'm working on porting cloud-init to Arch Linux so that we don't use the really funky rc.local setup I have it currently using.

• systemd used instead of sysvinit. I also have a flag in my build process which allows me to build it with sysvinit as the default, but I think since systemd is the future I'll keep that the default for now.

• Additional services enabled at boot: rngd, sshd, netcfg

• User's public key is pulled from the EC2 instance metadata service at startup, and added to /root/.ssh/authorized_keys

• SSH configured with 'PasswordAuthentication no', enforcing public key authentication

• pacman loads (and automatically lsigns) the 'archlinux' and 'ec2' keyrings on the first boot (the latter keyring contains my public key used for package signing in the ec2 repo).

• pacman mirror list is automatically selected at boot based on a list I created (based on rankmirrors run on instances in each region). These lists are provided by the package ec2-pacman-mirrors, which is in the ec2 repo.

• dhclient is used instead of dhcpcd for robustness reasons. I found that dhcpcd gave up too quickly if it tried to do a DHCPREQUEST when the vif wasn't completely up, making the EC2 instance inaccessible.

• dhclient is configured to retry forever, and request the following dhcp options: subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-name-servers, host-name, interface-mtu, fqdn

• /usr/bin/pinentry is symlinked to /usr/bin/pinentry-curses instead of the default pinentry-gtk, since gtk isn't available in this install and the primary access method is SSH.

caveats/bugs

There is one major caveat I know of at the moment. If you launch one of these and create an EBS AMI snapshot based on your instance, any launches of the derived AMI can only be accessed by your SSH key pair. This means you effectively cannot share any derived AMIs unless you first remove /root/.ssh/authorized_keys and /etc/pacman.d/gnupg/* before snapshotting. This is one of the reasons I'm investigating cloud-init.

contact

If you find a bug or have any other comments, please send me an email.