Gandi Kitchen - Hosting

Storage Migration

Leland Vandervort — Thu, 05 May 2011 15:09:00 +0200

For the past few months perhaps you might have made use of the servers in the US. The changes in the storage technology was one of the strong points in the hosting infrastructure. Prior to this, we had to adapt the infrastructure so that it could understand "n" datacenters. The implementation of this new storage platform was not as complicated as it seems since it is completely independent to the architecture in France. With a new datacenter, it was therefore quite a trivial matter to build from scratch, and all of the new servers in the US made use of this new platform from day one.

On the French side, an inevitable migration was required in order to arrive at a standardised platform to utilise the new functionalities. Here, however, the problem is different and a little more complicated, with the coexistence of two different storage solutions. In reality, we were confronted with a number of challenges so that the machines hosting the servers could happily play ball with two different storage platforms at the same time. The opening of our US datacenter was already a few months ago, and so a large proportion of our efforts have been dedicated to this migration. This is, of course, proceeding and certainly takes [a lot of] time, and will soon be available to all of our customers. We will thus be in a position to run the two storage platforms together in order to perform the migrations efficiently.

Once active, all new disk creations will take place on the new storage platform. At this point we will enter the migration phase, and specifically the phase which directly impacts you since it also means the migration of your disks.

There are a number of ways to perform the migration:

Create a new disk:

The creation of this new disk will be on the new platform.

Next, simply attach this new disk to your server and copy the relevant data to the new disk. This would be a good occasion to do some housekeeping and get rid of any old data that you no longer need. Such commands as 'cp' or 'rsync' would do the job nicely.

Create a new disk from the image of an existing one:

This function has been available through the API for several months: disk.create_from(apikey, disk_spec, src_disk_id) -- see the API documentation for more details at http://doc.rpc.gandi.net/hosting/reference.html#disk.create_from

The next Gandi Website update will include the ability to create a new disk from and existing one to make life easier for you if you are not a user of our API. This method, nevertheless, requires either that the server that has the original disk attached be stopped, or that the disk be detached from the server.

Please note that the time take to make the copy will be directly dependent upon the size of the disk, so you should have some patience and/or a good coffee break, if you decide to employ this method on a comparatively large disk.

If the new disk is to be a system disk, then you will need to define the disk as a boot-disk either via the web interface ("Boot Disk" in the server details) or using the API : vm.disk_attach(apikey, vm_id, disk_id {'position' : 0} ) ( see http://doc.rpc.gandi.net/hosting/reference.html#vm.disk_attach )

This migration will enable you, among other things, to make use of the new storage features such as snapshots, resizing, and rapid-copy, which will be available in the coming weeks. Aside from the platform which is seeing new features every month, we hope to soon be able to talk to you about these new functions which may change the way you use your servers.

How to Create a System Images for Your Server

Leland Vandervort — Tue, 16 Nov 2010 16:46:00 +0100

There are many reasons to create a system image for your servers: to build a custom system with your preferred applications pre-installed, to create an image of a game server that can be easily deployed, to simply duplicate a custom server, or simply to backup one's system...

The procedure is relatively simple and can be performed by anybody, as long as you pay careful attention to the detail.

Create a Data Disk

You need to create a data disk via the disk creation interface of your hosting account. If you wish to make a copy of an existing disk, the data disk must be of sufficient capacity.

Simply attach your new disk to the target server ; the server that contains the virtual disk to copy, or the server used to perform the base installation.

Create a System Image

By making a copy of the data from a virtual disk.

Warning: You must have sufficient space on the destination disk.

Copy the data from the source disk to the destination with the 'tar' command:

tar cC /srv/disk1 . | tar xC /srv/disk2

If /srv/disk1 is the directory where the source disk is mounted and /srv/disk2 is the mount point of the destination disk. Using 'tar' will only copy the data and not empty blocks, so the operation is relatively quick.

Warning: if you want to copy the local system disk, you must add a few exclusions:

tar
--exclude=/proc --exclude=/sys --exclude=/srv/disk2 -c \ | tar xC /srv/disk2

By installing a base system

Preamble

Your new disk attached to a server is ready for use. You can partition your disk to prepare a swap partition for the image, but we advise that you do not create partitions and to create the file system directly on the disk.

In this way you have the advantage of flexibility if you intend to resize the disk later (no need to calculate the partition table), whilst being better adapted to the block storage on our physical filers, and to be able to decide later whether or not to use a swap file.

In the near future, all of the OS images provided by Gandi will be without partitions, and the Gandi hosting platform will automatically provide additional swap space.

Meanwhile, the virtual servers will still see the first disk as "xvda1" and the swap disk as "xvda2".

Whether or not you decide to partition, the Gandi hosting platform will be able to start a system disk with your image as source by detecting the disk boot sector and adapting the kernel boot options. Gandi also provide a server containing copies of the distributions for which we provide system images. You can use this server to speed up your installation : mirrors.gandi.net

Preparation

Initially, verify that the virtual disk is not already mounted (for example in /proc/mounts). Unmount the disk if it is mounted. Partition the disk if required and then format it.

We recommend that you format the disk directly and use the ext4 file system. If, for example, your disk is xvdc : mkfs.ext4 -j -m0 /dev/xvdc and then mount the disk to a directory of your system : mount -o rw /dev/xvdc /var/tmp

Bootstrap installation for distributions using .deb packages

The base system installation can be accomplished with debootstrap in a specific directory. Once the installation is completed and a few modifications are made, the disk will contain a bootable and fully functional GNU/Linux system. The system will have a basic set of applications and you will need to adapt the system to your requirements by installing any required applications with apt-get and configuring the locales.

We highly recommend the installation of a system with the amd64/x86_64 architecture, as this will be faster on the Gandi platform. When it comes to selecting the kernel, you should choose the appropriate x86_64 kernel (2.6.32-3831, for example).

An example installation for an Ubuntu Maverick 10.10 distribution on a previously prepared and mounted disk:

debootstrap
--arch=amd64 --verbose --components=main,universe,multiverse \

--include=openssh-server,openssh-client
maverick /var/tmp \

http://mirrors.gandi.net/ubuntu/

Should debootstrap happen to complain that no configuration file for maverick is accessible, you only need create a link in /usr/share/debootstrap/scripts from lucid to maverick. The following step will modify the file containing the source package media locations. Edit the file /etc/apt/sources.list in the directory by adding the distribution media and the Gandi package media.

Example :

# cat
/var/tmp/etc/apt/sources.list

deb
http://mirrors.gandi.net/ubuntu maverick main universe multiverse

deb
http://mirrors.gandi.net/ubuntu maverick-security main universe multiverse

deb
http://mirrors.gandi.net/ubuntu maverick-updates main universe multiverse

deb
http://mirrors.gandi.net/gandi/ubuntu maverick main

Once the debootstrap command and the configuration has been completed, you will be able to access the newly installed system via chroot to the directory.

For example: chroot /var/tmp

In this way, you can refresh the package media with apt-get update and install the various applications and packages that you may require. Beware, however, that the installation of packages in a chroot require a few corrections. Notably, you will need to mount /proc and /dev/pts which are usually available in /etc/fstab : mount -a then complete the package configurations with dpkg --configure -a. Certain packages attempt to automatically start the daemons and servers as they are installed. You will have to stop these services to umount your image later.

In order to complete the installation of these packages, you will need to modify the postinst files installed in /var/lib/dpkg/info by commenting out the calls to invoke-rc.d or start, such as with procps : #start procps in procps.postinst or for rsyslog : #invoke-rc.d in rsyslog.posinst.

For recent Ubuntu distributions, you will need to add an entry for /dev in the /etc/fstab file: dev /dev tmpfs rw 0 0 in order to boot correctly.

Next, copy the files /etc/hosts, /etc/resolv.conf and /etc/fstab of the server to the directory containing the newly installed system (/var/tmp in our example). Change these by deleting the hostname of the current machine in /etc/hosts and adapting /etc/fstab to your new disk image.

We recommend installing the gandi-hosting-agent and gandi-hosting-vm packages available on the Gandi distribution media server. To do so, add the Gandi maintainer key as follows:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D8EAC2F4DAFE3FA5

then :

apt-get install gandi-hosting-agent gandi-hosting-vm

The packages will install their required dependencies. For a description of the functions of these packages, please refer to the previous article in the Gandi Kitchen.

If your virtual server uses python version 2.6, you will need to install the package gandi-hosting-agent-py2.6 instead of gandi-hosting-agent.

Basic Installation for Distributions using .rpm

The installation of a distribution based on .rpm packages follows the same method as for a .deb based distribution, but using rinse or a specific application for the chosen distribution.

For example, using the same directory :

rinse --arch=amd64 --directory=/var/tmp --distribution=centos-5

According to your package management system, you will need to then add the package media locations in the configuration in order to install the necessary applications.

Please refer to the debootstrap / .deb base system installation for the remaining step : Gandi specific package installation, copy of system configuration file.

Other Ways to Install a Base System

Other distributions (often more specific) have other methods of installing the base system. Sometimes an image is already available, thus you need only mount it in 'loop' and copy the files from the image.

Taking the above example :

mkdir /srv/a

mount -o loop,ro /my/image/directory /srv/a

cp -raf /srv/a/ /var/tmp/

Yet a More Radical Method

If you have a source disk to copy, you could also copy the entire disk; Ensure that the destination disk that was just attached to the server is not mounted, and verify that the disk nameis not present in /proc/mounts.

Warning : this method only works where the destination disk is the same size as the source.

To create your system image and copy it to your new data disk, you will use the 'dd' command. You need to pay close attention to the syntax of the command.

Here is an example, which will be explained afterwards:

dd if=/dev/xvda of=/dev/xvdc conv=sync

The parameter if= refers to the source disk; of= refers to, of course, the destination disk. Any data on the destination disk will be completed erased. In order to distinguish between the disks, we recommend first using the 'df' command.

Note that the image will not be bootable. You will need to reconfigure the network settings of your server using DHCP before proceeding with the creation of the image. Copying using 'dd' takes significant time and resources as it copies byte-per-byte the source disk to the destination, without any distinction between a data block and an empty block. The Linux kernel for the Gandi hosting platform boots by default on the first partition ( root=/dev/xvda1 on the boot command line). If you copy an existing system disk, the partition table will be correct on the destination. Otherwise, you should ensure that the first partition of the destination disk contains the system files and is flagged as bootable. You can also additionally create a swap partition.

Requirements to Boot

In the case of a copy of an existing virtual disk, the source disk having already booted, you only need to check that the network configuration is using DHCP so that the produced images will be correctly bootable. In the case of an image created by a base system installation, there are a few items to verify and/or modify:

All of the services or modules related to physical elements of the server must be deactivated (such as, the service associated with the system clock - hwclock)

Some services at boot-time, such as ureadahead, console-setup, ondemand, plymouth, must be deactivated. Move away the init-scripts associated with these services. In an Ubuntu distribution, these scripts are found in /etc/init.

Add the gandi-hosting-agent and gandi-hosting-vm packages. The agent allows the installation of your server to complete. The scripts contained in the gandi-hosting-vm package allow the hosting platform to automatically and dynamically manage the server resources. For a system with python 2.6, you should install gandi-hosting-agent-py2.6 instead. For more details on these packages, please read the previous post on the Gandi Kitchen.

Delete the files related to ssh keys generated during the installation of the sshd package to avoid having the same key present on all of the servers generated from the same source image.

Verify that xinetd/inetd is started at boot -- if not, then activate it. The Gandi-agent is spawed via xinetd/inetd and will enable the final configuration of your server. The package gandi-hosting-agent installs its configuration file in /etc/gandi/agent.yml.

Verify that the permissions of your installation directory structure and the directories lib/ root/ and tmp/ of the installation directory. If in doubt, apply the same permissions to these directories as your existing server outside of the chroot.

chmod 0755 /var/tmp/lib

chmod 1777 /var/tmp/tmp

chmod 0750 /var/tmp/root

chmod 0755 /var/tmp/

Create the base files in /dev of your installation for the first boot stages, for example:

[ -e "$chroot"/dev/xvc0 ] || mknod "$chroot"/dev/xvc0 c 204 191

[ -e "$chroot"/dev/console ] || mknod "$chroot"/dev/console c 5 1

[ -e "$chroot"/dev/null ] || mknod "$chroot"/dev/null c 1 3

[ -e "$chroot"/dev/ptmx ] || mknod "$chroot"/dev/ptmx c 5 2

[ -e "$chroot"/dev/zero ] || mknod "$chroot"/dev/zero c 1 5

Add the necessary kernel modules corresponding to the kernel version in /lib/modules of your installation directory. The modules are available on mirrors.gandi.net/kernel/ (See the associated article on the wiki)

At the end of the configuration, if you are inside the chroot, leave it using 'exit' then unmount the various elements of your images in /proc/mounts. Kill any processes and daemons that may have been started during package installation within the chroot/directory. For example : grep /var/tmp/proc/mounts and then umount /var/tmp/proc and any others that may be listed. End by umount /var/tmp which should work without errors.

Detach the Disk

Return to your disk management interface in your Gandi hosting account, and detach the disk to which you have just copied, or on which you have just prepared the installation.

Make the disk bootable by associating a kernel

In the administration interface, you have the posibility to define a kernel for a disk. Select the virtual disk and associate it with a kernel suitable for the image you have just created. Let's take a "data" disk as an example, that we want to transform into a bootable virtual disk:

Use the link to change the disk information, and at the bottom of the page you will find an option to change the disk type:

You have now three advanced options, as for a bootable virtual disk:

By associating a kernel, the disk becomes usable as a custom image and will be shown in the list of images, just like the Gandi AI or expert images provided by Gandi during the creation of a new service via the administration interface.

Create a server from this image

During the creation of a server via your administration interface on the website, you will find your new custom image in the list of available images. The server creation will thus use this image as the source for the system disk. The server will then normally boot from a disk which is a copy of your image. As such, you have the possibility of creating serveral identical service from the same custom image.

Troubleshooting the server during boot

The Gandi hosting platform gives you the ability to access the command line of your server via the emergency console, which is accessible via an ssh session, and provides access to the console commandline of your server.

If you configure a getty on the console by default (tty1, xvc0, hvc0, depending on the chosen kernel version), you will have an emergency shell in case of boot errors, or a login prompt in the case of a successful server boot.

This emergency console will allow you to view the boot messages, and more importantly, any errors that may occur and thereby allowing you to debug your image.

Correcting the Image

Later, if you notice errors, or have forgotten anything in your custom image, you only need re-attach your new disk image to one of your servers (ultimately, and preferably, a server which was created from the same source image). Then you only need make the changes on the source image and then detach the disk. The image will thus be available to create new servers without error.

Gandi 10th Anniversary - The Experience

Leland Vandervort — Wed, 17 Mar 2010 19:33:00 +0100

To celebrate Gandi's 10th anniversary, this hair-brained idea to give away, in ten days, 55000 domains, raise a very practical question. How, once we open the floodgates on such an operation, to maintain the highest quality of service on the site? The festive spirit could well have transformed into a nightmare for our customers if they were suddenly unable to access their management interface.

So we took the decision to host the event on a dedicated site. This was a hitherto dreamed of occasion to put ourselves into our customers' shoes, and use our hosting infrastructure for this event. We defined the rules of play: Using only the tools provided to our customers, build an architecture which was easily scalable and didn't break the bank, and to demonstrate our renowned flexibility.

Keep it Simple, Stupid.

We had the "luxury" of one week from design to implementation. As a result, the charming idea to demonstrate our "cloudlike" site based on modern technology was summarily put out of our minds. To be perfectly honest, given the time scale, the lucky chosen developer had a nifty precept to select the technology: "You have full choice of the technology, but you have one week." It would be PHP/MySQL, which isn't exactly everybody's favourite! It would, nevertheless, allow us to release a tested site within the tight time frame.

To adequately sustain the load generated by such an event, several servers would be needed. We then hit our first stumbling block: Gandi does not [yet] have a load balancing solution for the hosting solution! Nevermind, we'll use the old yet faithful round-robin DNS method, with a low TTL to be able to quickly remove a front-end server from production in case of an incident.

Our Linux distribution for this occasion would be Ubuntu 9.10 - because it is reasonably up-to-date, with the 2.6.27 kernel.

Small Servers - Go Forth and Multiply!

The best way to sustain the high loads for our solution is to split the functionality among several small servers. This way we would maintain a minimum level of "vertical" scalability (you can dynamically increase the memory and CPU allocated to a server), and the architecture provides "horizontal" scalability.

In this way we could easily add resources if we started to feel the pinch, and add or migrate shares from one server to another as load requirements necessitate. There are numerous advantages of using multiple small servers:

each server gets a minimum of one core, burstable, of the CPU (yes, one whole core, even with one share -- that's new, by the way!)
assured resilience, with shares spread somewhat randomly across a few hundred different physical servers.
specifically in a virtualised environment, the memory performance is best with less than 1GB of memory.
if you have 4 servers of one share each, rather than one big server of 4 shares, they can dynamically increase to 8x4 shares, or 24x4 shares with a reboot, and all of this without modification to the architecture. A big server, however, would only scale to 8 shares without a reboot, or 24 after reboot.
resources may be easily moved towards servers that need it most

We commence with a simple architecture:

24 (!) servers of one share each, to manage the PHP website: 10 for the English site, 10 for the French site, and an additional 2 of each for IPv6.
2 servers of 4 shares each for replicated memcached to reduce database load and manage sessions.
1 MySQL server of 4 shares, which contains the pre-generated promotion codes (they actually all fit within memory, so the database itself should really be pretty bored doing nothing...)

After a couple of lovely overloads and a bit of code review, the database would finally be greatly spared by memcached (see the section "lightweight coding...").

One of our administrators would put his fingers to work on the site to create and configure 24 servers -- at the same time! Obviously the release of the hosting API or an admin interface function would have been welcome. (Thanks to cssh in this case!)

Lock Down (somewhat) the Machines

A default installation always needs a few finishing touches. The very fact of opening a MySQL database on the "public" network made us a little edgy. So, swooping 'netstat' and shutting down non-critical services listening on public ports. With the help of tcp wrappers (hosts.allow, hosts.deny), all of the "private" interfaces are also locked down (sshd, mysql accessibly only from the web farm).

Finally it behooved us to pay close attention to the PHP code and MySQL queries; The safest way to avoid php code injections is to bind all the parameters after a prepare(). This also helps reduce load on the database when several execute() are called.

One important detail: since the site should allow a user to send an email to any "arbitrary" address, it was absolutely critical to limit its potential for abuse by some clever black-hat as much as possible. At the very minimum, the number of sent emails per promotion code was limited, in addition to very close monitoring.

Setup the Development and Deployment Environment

The sharing of data between the sites in effect adds a single point of failure, as well as a potential architectural bottleneck. As such, we decided to deploy the content of the site locally on each of the servers. We would use one server for developing and staging, and ultimately for the development and testing of updates. A quick script and some 'rsync' would allow rapid deployment across the entire front-end architecture. Simple! (some would say ;) )

Resource Monitoring

A few moments before the operation, more as a precaution rather than a cure, all of the virtual machines from one to two shares. Using the statistics interface, from day one, one can see that the the virtual machines were essentially sitting "twiddling their thumbs" from boredom ;) :

It would have been cool, at this very moment, to reduce back to a single share per server, or make use of Gandi "Autoflex", or even given the actual load observed, set up scheduled flex for each hour to hand out the promotion codes! Unfortunately, with all hands on deck, we missed this opportunity to demonstrate this [econono-techno-ecological ;)] feature.

Lightweight Code is Worth More than a Thousand Beefy CPUs

Even though we physically had several thousand CPUs and a few Terabytes of RAM at our fingertips, Tuesday turned out to be somewhat chaotic and worthy of note here. After Monday, which managed the load very well, the "smooth" execution of our one and only SELECT COUNT brutally altered and became excruciatingly slow (300ms). We had naively thought that this "only" query, on a table held exclusively in memory, wouldn't be an issue. As such, it was executed on every page of the site. The multiple simultaneous accesses to the database, coupled with the UPDATE operations for the promotion codes, resulted in the database, despite the near-idle system performance, started causing database lock contention.

The usual knee-jerk reaction to such a situation is to increase the number of shares to support the load. It's great for a quick-fix temporary solution, but it's not enough!

A new analysis of the system, questions about the code, and the use (or salvation) of memcached resulted in recovering the optimal performance. Equally, a modification of the database queries used probably would have been prudent.

The moral of the story: the code, indexes, architecture (etc.) are the cornerstones of your ability to support usage load, and if they are "CPU friendly", they will save the day. Otherwise a catastrophe could be lurking, or at the very least, the unnecessary purchase of additional shares.

Also, as we said earlier somewhat tongue-in-cheek -- it's eco-friendly!

Some Numbers

36 shares total, but we could have done it with less (*sniff*)
5% CPU usage at peak
4000 requests per front-end web server in the first minute of each hour (roughly 1400 requests/second total)
a minimum of 11 seconds to hand out 1000 promotion codes.
a maximum of 40 minutes to hand out the same number of promotion codes, during the Tuesday incident described above.