Gandi's Thomas Stocking went to the Cloud Expo 2011 West in Santa Clara this week, and found that marketing is alive and well in the Cloudsphere. Despite the fluff, there were some interesting ideas getting discussed, and companies developing strategies to leverage the evolving cloudscape...

For the past few months perhaps you might have made use of the servers in the US.  The changes in the storage technology was one of the strong points in the hosting infrastructure.  Prior to this, we had to adapt the infrastructure so that it could understand "n" datacenters.  The implementation of this new storage platform was not as complicated as it seems since it is completely independent to the architecture in France.  With a new datacenter, it was therefore quite a trivial matter to build from scratch, and all of the new servers in the US made use of this new platform from day one.

As most of you may have already noticed over the past 18 months, there have been several periods of scheduled maintenance on the Gandi network. Some of these have been fairly intrusive, while others have taken place quietly behind the scenes. I have made a number of innuendos here on the Gandi Kitchen and also on the Gandi Bar over the past year dropping hints about some of the things to come, so I decided to take this opportunity to reveal a little more about what our network operations team here at Gandi is up to and what it means for you over the coming months.

Why are we calling this "Operation Dragonfly" ?? well.. that is for you to try to guess! We will be giving a Gandi T-Shirt to the first five people who find the link/significance of the name of this project, and send us a short one paragraph description of this significance by email to dragonfly AT gandi DOT net :) Contest ends on 14 February 2011 when the first five people to send us a correct answer will be sent their Gandi T-Shirts! (Please include your size and address in your email so that we know where to send it to, and to make sure we don't send you one for a Barbie doll!) Oh .. and before I forget, there are a few subtle hints embedded throughout this article to help you, and after the end of the competition we will also reveal the clues for everyone !

With the opening of our Baltimore datacenter, we have decided to provide a quick FAQ to respond to a few of the commonly asked questions about the hosting product and how it will work with the two datacenters, as well as some of the other features that we are putting in place.

If you have any questions not addressed here, then please feel free to let us know!

What are the modification that Gandi staff makes on standard installed OS to be used on Gandi hosting?

Local modifications

Gandi hosting infrastructure is using Xen virtualization in paravirtualization mode (for the moment). As such we have to build a Linux kernel with specific options to allow it to boot your virtual server. Moreover as we allow customers to dynamically add or remove resources, the hotplug system in the kernel was patched by our team to allow a correct use of this features (mainly correct udev call). All kernel modules are available at each new kernel release on http://mirrors.gandi.net/kernel/. As of the 2.6.32, we now use upstream kernel source for building the xenU kernel and you should find the buildconfig file in /proc/config.gz on your virtual server. We add external patches such as drbd (before upstream integration).

Each Linux base system that we provide on Gandi hosting contains modifications by our team. For example, we removed services based on hardware clock as Xen does not provide direct access to it. On some distribution we had to disable boot features such as ureadhead or plymouth to allow a flawless boot of the virtual server. The main configuration is done during the boot process, especially the first boot process.

On a side note, on x86_32 architecture, to use the hardware capability of Xen, the libc could use the nosegneg hwcap with the correct libc-xen package.

Package gandi-hosting-vm

The idea which triggered this article was the release of a new version of gandi-hosting-vm. The package contains a collection of scripts to setup the local system of your virtual server at each boot and when specific events about hosting resources happen.

Changing hosting resources

When you add or remove resources dynamically to your virtual server, the Linux kernel receives information from Xen - the system managing all the virtualization. Each of these events are passed to the udevd daemon which apply configured rules to these events. It mainly creates files in directory /dev to allow access to the newly discovered resources.

For resources that Gandi hosting allows you to dynamically change, we wrote some udev rules (located in /etc/udev/rules.d/86-gandi.rules) to start a script when a virtual disk, a virtual interface or even a virtual cpu is attached (or removed) to the virtual server.

On a more detailed level, when a virtual interface is attached, the script /etc/gandi/manage_iface.sh is called by udev and a DHCP request is sent for this interface. A couple of other scripts setup the default route (/etc/gandi/dhcp-postconf) and store network configuration (/etc/dchp-hostname) in a tmpfs directory for further configuration at the end of the boot process. When the virtual interface is removed, the script simply removes the local network interface.

When a virtual disk is attached a similar script (/etc/gandi/manage_data_disk.py) is called. It tries to check the file system on the device or in its partitions (only in GandiAI mode) and mounts the file system in a specific mountpoint using the file system label /srv/<FS label>. If no label is setup on the file system, it uses the device or partition name as mount point (/srv/xvdc1 for example). To change the default mount options, please edit the variable mount_options in the beginning of the Python script.

During the boot process

The gandi-hosting-vm package provide two services called on boot : gandi-mount and gandi-config. The first one mounts already attached virtual disks in the local system in the /srv directory (see the description of /etc/gandi/manage_data_disk.py). You can start the service again once your server is booted and it will mount attached disks to the server (if you remove udev packages for example).

The second service starts a couple of specific plugins to setup your local system. Some of these configurations are optional and a configuration file is available for you to choose to setup each of these optional features /etc/default/gandi. Each configuration variable contains a short description in the default config file. These plugins are configuring the default local console for the hosting emergency console, configure the hostname and dns resolver, change the timezone to Europe/Paris, change the hwcap nosegneg according of your kernel version, change the motd to the default and so on.

For example, when the plugin 11-config_ssh is called, it creates SSH key for the local system if the keys are not already present. Then, depending on your configuration, it could add the Gandi SSH management key to the root user keyring (variable CONFIG_SSHMGMT) and reconfigure your sshd server by disabling password access for root, disabling empty password and enabling compression (variable CONFIG_SSHD).

Package gandi-hosting-agent

Gandi agent is used to setup the virtual machine according to customer information. In case of expert mode server, the setup of the local system is limited to setting the root password and creating the administrator user (as chosen by the customer) to avoid ssh-ing the server as root. In case of a GandiAI mode server, the agent uses specific modules to setup applications on the local system.

Once your expert server is setup after creation, you can remove gandi-hosting-agent packages. For example : dpkg -P $(dpkg -l | awk '/gandi-hosting-agent/ { print $2 }' | xargs) in deb based package system or rpm -e gandi-hosting-agent in rpm backed package system.

We are pleased to announce that you can (finally!) choose the version of your kernel that you want (from a list which will be continually expanded), and associated boot options (cmdline). 2.6.18 and 2.6.27 are "base" versions supplied by Xen (backport of Xen patches for 2.6.27). Version 2.6.32, which is is currently available, uses paravirt_ops and the "Linux" implementation of Xen patches.

We will show you the new kernels here. They can be found in the "advanced mode" within the server's management page:



And you will then be able to access:



Concerning cmdline, you may now deactivate selinux at boot, boot as a single user, change the disk and the boot partition (which is practical for working with "images"), or choose the most appropriate console for your needs. In short, everything that you need to manage your updates in a more friendly environment, or to repair your server in the most autonomous manner.

If you feel that an option is missing, please let us know.

(Translator's note: Taskigi Sequoyah would be proud! ;) )

A friend had explained to me that he had just finished installing a web server.  Having decided on a change and not to use the mammoth Apache that everyone knows and loves, my friend seemed to have found a decent alternative.  Bearing the name Cherokee, it appeared much leaner and with greater performance than Apache.  See the benchmarks from the project website at the end of this article.

Personally, I had no knowledge of Cherokee, though I had certainly heard the name pandered around once or twice, but I never really paid it any attention at the time.  So to give it a whirl, I decided to install it locally, which to my surprise exceeded my expectations.  On a debian (or derived) distribution, installation is achieved by a simple apt-get install cherokee (or 'aptitude'...)

The web management interface provided is clean and intuitive.  The basic idea is quite interesting;  It is possible to activate the PHP interpreter with a simple mouse click; same for activating different virtual hosts as well as other options.  There are also wizards to assist with the installation of such applications and frameworks as Django, Rails or Wordpress, etc.

I thought that it would be interesting to present this project during a developers meeting at Gandi, especially for those who use GandiAI.  The presentation was well received.  After a few pow-wow, and a glance through the source code, there was nothing really unusual or risky.  This doesn't mean, of course, that there aren't any bugs in it, but at any rate clean code is a good sign!  The core is written in C with the administrator interface written in Python.

Thus is the way in which Cherokee made its appearance at Gandi, and a server image was produced.  In this way you can rapidly test a server with Cherokee pre-installed.

We have also added an "expert" mode distribution with Cherokee pre-installed... You need only to select this distribution during the steps to create your server.  And, just as a reminder, in case you haven't yet tried our service, you can always request a free trial using our online form!

For further information about Cherokee, you can visit the project website.  Additionally, you can find a comprehensive article about Cherokee in the Gnu/Linux magazine France, written [in French] by Carl Chenet in issue number 125.

To celebrate Gandi's 10th anniversary, this hair-brained idea to give away, in ten days, 55000 domains, raise a very practical question.  How, once we open the floodgates on such an operation, to maintain the highest quality of service on the site?  The festive spirit could well have transformed into a nightmare for our customers if they were suddenly unable to access their management interface.

So we took the decision to host the event on a dedicated site.  This was a hitherto dreamed of occasion to put ourselves into our customers' shoes, and use our hosting infrastructure for this event.  We defined the rules of play:  Using only the tools provided to our customers, build an architecture which was easily scalable and didn't break the bank, and to demonstrate our renowned flexibility.

Many websites start with a single server solution, a box acting as a web server and database server all in one. Simply, easy, cheap. The problem comes when traffic gets too high (a victim of their own success!). Many customers want a bigger box, but the answer is actually changing your architecture from "web server" to "web infrastructure". You can duplicate web servers, use the DNS to load balance them and ramp up your capacity very fast and very far.