projects::arch linux on ec2

Arch Linux
Amazon Web Services

summary

Arch Linux's minimalistic philosophy and high degree of customizability makes it a great choice for compute cloud deployment. It's light, fast, and scalable.

I am doing releases of Arch Linux as Amazon EC2 AMIs roughly twice per month.

contact

IRC Join us on irc.freenode.net in #archlinux-ec2!

If you find a bug or have any other comments, please send me an email, or ping me on twitter.

NOTE: This project is not run, sponsored, or endorsed by my employer.

current releases

Release 2015.06.23

HVM Images

region ebs
hvm
x86_64
s3
hvm
x86_64
ebs
hvm gpu
x86_64
s3
hvm gpu
x86_64
us-east-1 ami-45c53d2e ami-41c53d2a
us-west-1 ami-f92fd8bd ami-ff2fd8bb ami-6929de2d ami-6f29de2b
us-west-2 ami-8d3e3abd ami-8b3e3abb ami-33c7c303 ami-31c7c301
eu-central-1 ami-447c4759 ami-4a7c4757 ami-08714a15 ami-0e714a13
eu-west-1 ami-6285c115 ami-6685c111 ami-3c93d74b ami-3e93d749
ap-southeast-1 ami-34565166 ami-36565164
ap-southeast-2 ami-a102479b ami-a3024799 ami-31094c0b ami-33094c09
ap-northeast-1 ami-a29a3ca2 ami-a09a3ca0 ami-026ec802 ami-006ec800
sa-east-1 ami-a364e6be ami-a164e6bc ami-816cee9c ami-876cee9a
cn-north-1 ami-405dc079 ami-4e5dc077

Paravirtual Images

region ebs
paravirtual
x86_64
s3
paravirtual
x86_64
ebs
paravirtual
i386
s3
paravirtual
i386
us-east-1 ami-f5c53d9e ami-f7c53d9c ami-47c53d2c ami-43c53d28
us-west-1 ami-8b2fd8cf ami-892fd8cd ami-e92fd8ad ami-ed2fd8a9
us-west-2 ami-093f3b39 ami-073f3b37 ami-393f3b09 ami-373f3b07
eu-central-1 ami-16714a0b ami-14714a09 ami-8a7c4797 ami-887c4795
eu-west-1 ami-bc85c1cb ami-be85c1c9 ami-9a85c1ed ami-9285c1e5
ap-southeast-1 ami-9a5651c8 ami-945651c6 ami-c856519a ami-ca565198
ap-southeast-2 ami-f90247c3 ami-fb0247c1 ami-9d0247a7 ami-9f0247a5
ap-northeast-1 ami-be62c4be ami-bc62c4bc ami-bc9a3cbc ami-ba9a3cba
sa-east-1 ami-ad64e6b0 ami-b364e6ae ami-d364e6ce ami-d164e6cc
cn-north-1 ami-f25dc0cb ami-f05dc0c9 ami-fa5dc0c3 ami-f85dc0c1

Release 2015.06.15

HVM Images

region ebs
hvm
x86_64
s3
hvm
x86_64
ebs
hvm gpu
x86_64
s3
hvm gpu
x86_64
us-east-1 ami-1b37c770 ami-0537c76e ami-2538c84e ami-2738c84c
us-west-1 ami-11eb1f55 ami-17eb1f53 ami-6beb1f2f ami-6feb1f2b
us-west-2 ami-5f4b716f ami-5d4b716d ami-fb4a70cb ami-f94a70c9
eu-central-1 ami-a81921b5 ami-ae1921b3 ami-a41921b9 ami-aa1921b7
eu-west-1 ami-8f83fbf8 ami-8183fbf6 ami-8983fbfe ami-8b83fbfc
ap-southeast-1 ami-369a9f64 ami-309a9f62 ami-ee9a9fbc ami-e89a9fba
ap-southeast-2 ami-47c7bc7d ami-41c7bc7b ami-89c7bcb3 ami-8bc7bcb1
ap-northeast-1 ami-a4b56fa4 ami-a2b56fa2 ami-7eb56f7e ami-7cb56f7c
sa-east-1 ami-8b6dee96 ami-896dee94 ami-d56deec8 ami-db6deec6
cn-north-1 ami-86b12cbf ami-84b12cbd ami-f0b12cc9 ami-feb12cc7

Paravirtual Images

region ebs
paravirtual
x86_64
s3
paravirtual
x86_64
ebs
paravirtual
i386
s3
paravirtual
i386
us-east-1 ami-e53aca8e ami-e73aca8c ami-9b39c9f0 ami-8539c9ee
us-west-1 ami-25d42061 ami-1bd4205f ami-adeb1fe9 ami-a3eb1fe7
us-west-2 ami-b1487281 ami-4f48727f ami-6d48725d ami-6b48725b
eu-central-1 ami-3e1a2223 ami-3c1a2221 ami-161a220b ami-141a2209
eu-west-1 ami-4f86fe38 ami-4186fe36 ami-5386fe24 ami-5586fe22
ap-southeast-1 ami-50a5a002 ami-52a5a000 ami-caa5a098 ami-c4a5a096
ap-southeast-2 ami-cdc7bcf7 ami-cfc7bcf5 ami-efc7bcd5 ami-e9c7bcd3
ap-northeast-1 ami-46ae7446 ami-44ae7444 ami-1eaa701e ami-1aaa701a
sa-east-1 ami-3f6cef22 ami-3d6cef20 ami-156cef08 ami-1b6cef06
cn-north-1 ami-dcb12ce5 ami-dab12ce3 ami-d0b12ce9 ami-deb12ce7

tools

The EC2 image build process is public, but the AMI registration portion is not. Here are the necessary tools to create an image file, but see the 2013-05-26 news post for information on how to register the images in EC2.

  • ami-build-backend - These files are held on the PXE server, and fetched when the guest boots.
  • ami-builder-image - This is a fork of archiso with some changes to automatically pull down my install script and do a few other things.
  • ec2-packages - These are the sources for all the packages contained in the 'ec2' Pacman repository.

recent changes and news

  • 2014-07-26

    I've added AMIs for the Beijing, China (cn-north-1) region.

  • 2014-06-27

    We're now up to Linux 3.15.2. I've removed xen-fbfront from the initramfs, because the module was causing 30-second boot delays:

    [    2.050081] tsc: Refined TSC clocksource calibration: 2793.267 MHz
    [    6.370066] xenbus_probe_frontend: Waiting for devices to initialise: 25s...20s...
    [   12.390306] random: nonblocking pool is initialized
    [   16.370070] 15s...10s...5s...0s...
    [   31.371241] xenbus_probe_frontend: Timeout connecting to device: device/vfb/0 (local state 3, remote state 1)
    

    The module is not required for an instance to boot correctly, so it can be removed from the initramfs. If you are running an AMI older than the 2014.06.27 release and would like to improve your instance's boot time, you can prune the module yourself:

    [root@ip-10-0-155-25 ~]# grep fbfront /etc/mkinitcpio.conf 
    MODULES="button ipmi-msghandler ipmi-poweroff virtio virtio-blk virtio-net virtio-pci virtio-ring
    xen-blkfront xen-fbfront xen-netfront xen-pcifront xen-privcmd hv_storvsc hv_balloon
    hv_vmbus hv_utils hv_netvsc ixgbevf"
    [root@ip-10-0-155-25 ~]# sed -ri 's/xen-fbfront //g' /etc/mkinitcpio.conf 
    [root@ip-10-0-155-25 ~]# mkinitcpio -p linux-ec2
    ==> Building image from preset: /etc/mkinitcpio.d/linux-ec2.preset: 'default'
      -> -k /boot/vmlinuz-linux-ec2 -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-ec2.img -S autodetect
    ==> Starting build: 3.15.2-1-ec2
      -> Running build hook: [base]
      -> Running build hook: [udev]
      -> Running build hook: [modconf]
      -> Running build hook: [block]
      -> Running build hook: [filesystems]
      -> Running build hook: [growfs]
      -> Running build hook: [keyboard]
      -> Running build hook: [fsck]
    ==> Generating module dependencies
    ==> Creating gzip initcpio image: /boot/initramfs-linux-ec2.img
    ==> Image generation successful
    [root@ip-10-0-155-25 ~]#
    

    After removing xen-fbfront from mkinitcpio.conf's MODULES section, subsequent reboots will be 30 seconds shorter. Before:

    [root@ip-10-0-155-25 ~]# systemd-analyze
    Startup finished in 32.530s (kernel) + 3.997s (userspace) = 36.528s

    After:

    [root@ip-10-0-155-25 ~]# systemd-analyze
    Startup finished in 2.345s (kernel) + 2.092s (userspace) = 4.438s
  • 2014-06-19

    The AMI now uses systemd's networkd, timesyncd, and resolved services. This makes the AMI have a significantly smaller footprint. Right now our biggest non-core packages are CUDA (in the GPU AMI) and cloud-init, which has a large dependency chain. I'd like to slim things even further, but I'll need to investigate how to do so.

  • 2014-03-24

    We're up to Linux 3.13.7 for the ec2 kernel and 3.10.34 for the ec2-lts kernel. I didn't make a news post earlier, but kernels are now built with 'debug' and 'strip' options, which will create split-out debug information packages (i.e. linux-ec2-debug, linux-ec2-lts-debug). This is useful for tools like perf, oprofile, and systemtap. Note that the -debug packages are compressed with 'lrzip'. New AMI builds have lrzip preinstalled, but if you're running an instance based on one of the older AMIs, you will need to install lrzip before you can make use of the -debug packages.

  • 2013-11-28

    New AMIs are being built right now and contain a couple changes:

    • EBS root volumes are now automatically resized to fill the block device. You can take advantage of this feature by launching an instance with a root volume size larger than the snapshot.
    • The resolv.conf file permissions are now 0644, allowing non-root users to resolve hostnames.

  • 2013-11-26

    Geoff H. and David B. both reported an issue with the current AMI release. The /etc/resolv.conf permissions are set to 0600 rather than 0644, which means that non-root users cannot resolve hostnames to IP addresses. This is an unintentional regression, most likely caused by a default 'umask' change in some package. dhclient will create a new resolv.conf and copy it over any existing file, which preserves the target file's permissions. But if no such file exists, then the permissions of the source file are copied. Previously, this worked fine because the file was generated with 0644 permissions, but now it's being generated with 0600. I've implemented a fix for future AMI builds. In the meantime, if non-root users need to perform DNS requests in your instances, be sure to do 'chmod 0644 /etc/resolv.conf'.

  • 2013-11-06

    A new ec2-pacman-mirrors package is available, and will provide your instances with optimal Arch Linux mirrors for your EC2 region. The upgrade path is as follows:

    1. Edit /etc/pacman.conf, change 'ec2' mirror URL to https://s3.amazonaws.com/arch-linux-ami/repo/$arch
    2. Run 'pacman -Sy ec2-pacman-mirrors'

    New AMIs will be published very shortly which use the new mirror list and point to the new EC2 package repository.

  • 2013-05-26

    I've added some links to this page, which are the complete set of files needed to do an EC2 image build. This does not include the AMI registration process, however. The tools Amazon provides for HVM AMI registration are still under NDA at the moment, and the bits necessary to do that are included in my AMI registration tools. So I can't make those public right now. The process itself can be replicated relatively easily, though:

    1. Build your VM image using the build-backend and builder-image repos above. PXE is what I use, but you could just as easily make it into an ISO or something. If you intend to do an S3-backed AMI, you will need to make the image no larger than 10GB (I use 8GB).
    2. Trim the image down (I do a 'mount -o loop,discard' on the image, then 'fstrim' the mount point, making the image into a sparse file).
    3. Tarball the image (tar cSzf, S to preserve the sparseness).
    4. Upload the tarball to S3.
    5. In each region, launch an instance and attach an empty 8GB EBS volume to them.
    6. On each of those instances, download the tarball and extract with 'tar xSf'.
    7. Use 'ddpt', an enhanced dd which pays attention to the sparseness of the image, to copy the raw image file into the EBS device. I use "ddpt if=<imagefile> of=/dev/xvdf bs=512 conv=sparse oflag=sparse,fsync". The sparseness aspect is important, because otherwise you're copying empty blocks onto the EBS device, which makes the snapshot take much longer, and is really just a waste of time. EBS volumes already read-as-zero, so there's no sense copying zero blocks.
    8. Detach the EBS volume and terminate the instances.
    9. Snapshot the EBS volume.
    10. Delete the volume (not needed now).
    11. Use ec2-register to create an AMI from the snapshotted volume.

  • 2013-03-22

    I've started creating AMIs which have CUDA preinstalled. These are for the cg1.4xlarge instance type.

  • 2013-02-05

    Nothing too exciting lately. Today's release has Linux 3.7.6.

  • 2012-11-22

    New AMI releases, now with cloud-init. Thanks to Jeremy D for contributing his time and effort to making cloud-init work well on Arch Linux.

  • 2012-11-12

    Released new AMIs, primarily for the new AWS region in Sydney, Australia (ap-southeast-2).

  • 2012-11-08

    Today's AMIs are released. Nothing too fancy in this build: just updated packages, including linux-ec2 3.6.6-1.

  • 2012-10-21

    I've added a new linux-ec2 package which contains a patched v3.6.2 kernel. There are a few major differences between this kernel and the Arch Linux stock kernel:

    • Hangs on Xen fixed (patches from 3.6.3 stable-queue).
    • CONFIG_PREEMPT_VOLUNTARY instead of CONFIG_PREEMPT, this will allow for better scheduling as a domU.
    • CONFIG_HZ=100 instead of CONFIG_HZ=300, this allows for better performance on many-CPU instances, as there are fewer timer interrupts to preempt other tasks.
    • Many drivers removed, particularly those that didn't make sense for running in an EC2 instance. I've left drivers for my own hardware so I can experiment with it as a dom0 kernel as well. The kernel size is roughly half the stock Arch Linux kernel due to the stripped drivers.

    I am also building new AMIs right now, and am beating the i386 AMIs into working order. Once done I'll publish the next release (which should be 2012.10.21). Once it's available, it will show in the tables above.

  • 2012-10-16

    Do not upgrade HVM instance kernels to anything between 3.6.0 and 3.6.2 inclusive. You must wait for 3.6.3 or else your instance will not boot. We're currently waiting on this patch to be integrated into the mainline stable tree. This is also why I am probably not doing an AMI release this week, as the HVM AMIs would be totally broken.

    I've also taken a look at building i386 (well, i686) AMIs. I'm not really sure that it's worth the effort. Nobody really uses 32-bit AMIs anymore, and we'd need to fork the kernel just to make it happen. For now, i686 is on ice.

release notes

These AMIs are as close to a "vanilla" install as I can make them without making them functionally impaired on EC2. But here's the complete list of differences between the EC2 builds and a stock install:

  • High performance kernel specifically for EC2, including paravirtualization support on i386 and x86_64 AMIs, and more Xen-friendly process scheduling.

  • Kernel modules included in initrd, some of which are relevant outside of EC2 contexts (e.g. if you want to run the image in a non-EC2 environment such as KVM or Hyper-V):

    • KVM: virtio virtio-blk virtio-net virtio-pci virtio-ring
    • Xen: xen-blkfront xen-netfront xen-pcifront xen-privcmd
    • Hyper-V: hv_storvsc hv_balloon hv_vmbus hv_utils hv_netvsc
    • IPMI (e.g. EC2 reboot request): button ipmi-msghandler ipmi-poweroff
    • EC2 enhanced networking SR-IOV driver: ixgbevf
  • Extra packages installed: audit, cloud-init, ec2-keyring, ec2-pacman-mirrors, irqbalance, lrzip, openssh, rng-tools, rsync, systemd-sysvcompat

  • Added an extra package source for ec2-specific packages. The repository currently contains numerous packages useful on EC2. You can view the list of packages by doing 'pacman -Sy; pacman -Ss | grep ^ec2'

  • Additional services enabled at boot: rngd, sshd, cloud-init, irqbalance, auditd, systemd-networkd, systemd-timesyncd, systemd-resolved

  • User's public key is pulled from the EC2 instance metadata service at startup, and added to /root/.ssh/authorized_keys

  • SSH configured with 'PasswordAuthentication no', enforcing public key authentication

  • pacman loads (and automatically lsigns) the 'archlinux' and 'ec2' keyrings on the first boot (the latter keyring contains my public key used for package signing in the ec2 repo).

  • pacman mirror list is automatically selected at boot based on a list I created (based on rankmirrors run on instances in each region). These lists are provided by the package ec2-pacman-mirrors, which is in the ec2 repo.

  • dhclient is used instead of dhcpcd for robustness reasons. I found that dhcpcd gave up too quickly if it tried to do a DHCPREQUEST when the vif wasn't completely up, making the EC2 instance inaccessible.

  • dhclient is configured to retry forever, and request the following dhcp options: subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-name-servers, host-name, interface-mtu, fqdn

  • /usr/bin/pinentry is symlinked to /usr/bin/pinentry-curses instead of the default pinentry-gtk, since gtk isn't available in this install and the primary access method is SSH.