Gandi Kitchen

OpenSSL: heartbleed postmortem

A vulnerability in OpenSSL was released on April 7th 2014 around 11:00 CEST.

This security issue allows an attacker to get sensitive data but the full impact is hard to evaluate (could go up to the private key).

Please find the timeline below, with actions taken to secure our platform following this publication:

April 7

  • 23:30 CEST (21:30 UTC): Our technical team are aware of the OpenSSL security issue.

April 8

  • 00:19 CEST (22:19 UTC): Debian patch is now available, rebuild packages for the platforms.
  • 03:34 CEST (01:34 UTC): Customers using our web-accelerators are now protected of the heartbleed security flaw
  • 04:23 CEST (02:23 UTC): End of the upgrade of the OpenSSL library for www.gandi.net  and webmail.gandi.net
  • 12:59 CEST (10:59 UTC): Error during the SSL provisioning on our platform for www.gandi.net and webmail.gandi.net during the certificates load.
  • 13:11 CEST (11:11 UTC): Certificates for www.gandi.net and webmail.gandi.net are now up to date.

End of the security incident

For more details and instructions, please see our news post about Heartbleed.



How is a virtual machine set up?

The setup process of a virtual machine on Gandi IaaS has changed but the new process is still considered unstable and the API calls may change in the near future. If you chose to play around this new mechanism to develop new ways to commission virtual machines, let us know! And keep in mind this is an early release of our provisioning tools, for which we cannot guarantee a stable interface yet.


SimpleHosting PaaS Platform

We are pleased to write about our new Platform As A Service (PaaS), which we call SimpleHosting. We have been working on this for more than a year, and are integrating the latest versions of component software while maintaining homogeneity with the management framework of our existing Infrastructure As A Service (IaaS) hosting.

Our goal in offering this platform is full abstraction of the system configuration, using our specific knowledge and experience gained from thousands of existing hosting installations. This abstraction then allows you full latitude on the application side, within reasonable limits. We have chosen the limits to be as open as possible by default.



What's Happening at the Cloud Expo in Santa Clara?

Gandi's Thomas Stocking went to the Cloud Expo 2011 West in Santa Clara this week, and found that marketing is alive and well in the Cloudsphere. Despite the fluff, there were some interesting ideas getting discussed, and companies developing strategies to leverage the evolving cloudscape...

When Null0 and BGP May Cause Problems

If you read any networking textbook or study guide on the subject of BGP and route stability, very often you will find a mention or even a suggestion to tie your aggregated prefix to null0 to ensure that the prefix is remains in the routing table thereby increasing the stability of your BGP advertisements.

While this is a good thing, to a certain extent, there are situations where such a configuration may lead to service interruption in case of an outage.  This quick article talks a little about internet routing using BGP and some "common practices".


Storage Migration

For the past few months perhaps you might have made use of the servers in the US.  The changes in the storage technology was one of the strong points in the hosting infrastructure.  Prior to this, we had to adapt the infrastructure so that it could understand "n" datacenters.  The implementation of this new storage platform was not as complicated as it seems since it is completely independent to the architecture in France.  With a new datacenter, it was therefore quite a trivial matter to build from scratch, and all of the new servers in the US made use of this new platform from day one.



Operation Dragonfly - Next Generation Gandi Network

As most of you may have already noticed over the past 18 months, there have been several periods of scheduled maintenance on the Gandi network. Some of these have been fairly intrusive, while others have taken place quietly behind the scenes. I have made a number of innuendos here on the Gandi Kitchen and also on the Gandi Bar over the past year dropping hints about some of the things to come, so I decided to take this opportunity to reveal a little more about what our network operations team here at Gandi is up to and what it means for you over the coming months.

Why are we calling this "Operation Dragonfly" ?? well.. that is for you to try to guess! We will be giving a Gandi T-Shirt to the first five people who find the link/significance of the name of this project, and send us a short one paragraph description of this significance by email to dragonfly AT gandi DOT net :) Contest ends on 14 February 2011 when the first five people to send us a correct answer will be sent their Gandi T-Shirts! (Please include your size and address in your email so that we know where to send it to, and to make sure we don't send you one for a Barbie doll!) Oh .. and before I forget, there are a few subtle hints embedded throughout this article to help you, and after the end of the competition we will also reveal the clues for everyone !



Gandi Hosting : US and France Datacenters FAQ

With the opening of our Baltimore datacenter, we have decided to provide a quick FAQ to respond to a few of the commonly asked questions about the hosting product and how it will work with the two datacenters, as well as some of the other features that we are putting in place.

If you have any questions not addressed here, then please feel free to let us know!

How to Create a System Images for Your Server

There are many reasons to create a system image for your servers: to build a custom system with your preferred applications pre-installed, to create an image of a game server that can be easily deployed, to simply duplicate a custom server, or simply to backup one's system...

The procedure is relatively simple and can be performed by anybody, as long as you pay careful attention to the detail.


Gandi modification on standard OS

What are the modification that Gandi staff makes on standard installed OS to be used on Gandi hosting?

Local modifications

Gandi hosting infrastructure is using Xen virtualization in paravirtualization mode (for the moment). As such we have to build a Linux kernel with specific options to allow it to boot your virtual server. Moreover as we allow customers to dynamically add or remove resources, the hotplug system in the kernel was patched by our team to allow a correct use of this features (mainly correct udev call). All kernel modules are available at each new kernel release on http://mirrors.gandi.net/kernel/. As of the 2.6.32, we now use upstream kernel source for building the xenU kernel and you should find the buildconfig file in /proc/config.gz on your virtual server. We add external patches such as drbd (before upstream integration).

Each Linux base system that we provide on Gandi hosting contains modifications by our team. For example, we removed services based on hardware clock as Xen does not provide direct access to it. On some distribution we had to disable boot features such as ureadhead or plymouth to allow a flawless boot of the virtual server. The main configuration is done during the boot process, especially the first boot process.

On a side note, on x86_32 architecture, to use the hardware capability of Xen, the libc could use the nosegneg hwcap with the correct libc-xen package.

Package gandi-hosting-vm

The idea which triggered this article was the release of a new version of gandi-hosting-vm. The package contains a collection of scripts to setup the local system of your virtual server at each boot and when specific events about hosting resources happen.

Changing hosting resources

When you add or remove resources dynamically to your virtual server, the Linux kernel receives information from Xen - the system managing all the virtualization. Each of these events are passed to the udevd daemon which apply configured rules to these events. It mainly creates files in directory /dev to allow access to the newly discovered resources.

For resources that Gandi hosting allows you to dynamically change, we wrote some udev rules (located in /etc/udev/rules.d/86-gandi.rules) to start a script when a virtual disk, a virtual interface or even a virtual cpu is attached (or removed) to the virtual server.

On a more detailed level, when a virtual interface is attached, the script /etc/gandi/manage_iface.sh is called by udev and a DHCP request is sent for this interface. A couple of other scripts setup the default route (/etc/gandi/dhcp-postconf) and store network configuration (/etc/dchp-hostname) in a tmpfs directory for further configuration at the end of the boot process. When the virtual interface is removed, the script simply removes the local network interface.

When a virtual disk is attached a similar script (/etc/gandi/manage_data_disk.py) is called. It tries to check the file system on the device or in its partitions (only in GandiAI mode) and mounts the file system in a specific mountpoint using the file system label /srv/<FS label>. If no label is setup on the file system, it uses the device or partition name as mount point (/srv/xvdc1 for example). To change the default mount options, please edit the variable mount_options in the beginning of the Python script.

During the boot process

The gandi-hosting-vm package provide two services called on boot : gandi-mount and gandi-config. The first one mounts already attached virtual disks in the local system in the /srv directory (see the description of /etc/gandi/manage_data_disk.py). You can start the service again once your server is booted and it will mount attached disks to the server (if you remove udev packages for example).

The second service starts a couple of specific plugins to setup your local system. Some of these configurations are optional and a configuration file is available for you to choose to setup each of these optional features /etc/default/gandi. Each configuration variable contains a short description in the default config file. These plugins are configuring the default local console for the hosting emergency console, configure the hostname and dns resolver, change the timezone to Europe/Paris, change the hwcap nosegneg according of your kernel version, change the motd to the default and so on.

For example, when the plugin 11-config_ssh is called, it creates SSH key for the local system if the keys are not already present. Then, depending on your configuration, it could add the Gandi SSH management key to the root user keyring (variable CONFIG_SSHMGMT) and reconfigure your sshd server by disabling password access for root, disabling empty password and enabling compression (variable CONFIG_SSHD).

Package gandi-hosting-agent

Gandi agent is used to setup the virtual machine according to customer information. In case of expert mode server, the setup of the local system is limited to setting the root password and creating the administrator user (as chosen by the customer) to avoid ssh-ing the server as root. In case of a GandiAI mode server, the agent uses specific modules to setup applications on the local system.

Once your expert server is setup after creation, you can remove gandi-hosting-agent packages. For example : dpkg -P $(dpkg -l | awk '/gandi-hosting-agent/ { print $2 }' | xargs) in deb based package system or rpm -e gandi-hosting-agent in rpm backed package system.


Hosting Public API 1.0 beta

As you are probably already aware, we have been beavering away to offer you a public API to manage your resources on our Cloud hosting platform.  In order to ease the work, several things have been reorganised so as to provide an interface which is easy to use, and allow a real management of your hosting resources at Gandi.

By way of this introduction, this article will be deliberately less technical, and will only succinctly present the elements that will be developed more in-depth when the official release of the API is launched.


Kernel and cmdline

We are pleased to announce that you can (finally!) choose the version of your kernel that you want (from a list which will be continually expanded), and associated boot options (cmdline). 2.6.18 and 2.6.27 are "base" versions supplied by Xen (backport of Xen patches for 2.6.27). Version 2.6.32, which is is currently available, uses paravirt_ops and the "Linux" implementation of Xen patches.

We will show you the new kernels here. They can be found in the "advanced mode" within the server's management page:

server-info-en.jpeg

And you will then be able to access:

advanced-mode-en.jpeg

Concerning cmdline, you may now deactivate selinux at boot, boot as a single user, change the disk and the boot partition (which is practical for working with "images"), or choose the most appropriate console for your needs. In short, everything that you need to manage your updates in a more friendly environment, or to repair your server in the most autonomous manner.

If you feel that an option is missing, please let us know.

Mandriva 2010 image in alpha (updated)

Server hosting by Gandi allow customers to choose from a selection of OS images available during the creation process of the virtual server. After the creation of the image by Gandi and internal testing, a new distribution is released to a specific group of hosting customers called 'alpha'. These clients can create server using these release candidate images. This allows Gandi to increase the types of testing and usage, and to find more bugs and problems by working with a small group of its customers.

Today - May the 21th - the Mandriva 2010.0 image has been released . This new version of the Mandriva distribution boots with a 2.6.27 kernel by default. It is currently only available for the 'alpha' customer group but will shortly be available for all customers. Please contact us if you wish to participate in our alpha testing phase.

16th August 2010 : Image is now available for everybody


Cherokee Arrives at Gandi

(Translator's note: Taskigi Sequoyah would be proud! ;) )

A friend had explained to me that he had just finished installing a web server.  Having decided on a change and not to use the mammoth Apache that everyone knows and loves, my friend seemed to have found a decent alternative.  Bearing the name Cherokee, it appeared much leaner and with greater performance than Apache.  See the benchmarks from the project website at the end of this article.

Personally, I had no knowledge of Cherokee, though I had certainly heard the name pandered around once or twice, but I never really paid it any attention at the time.  So to give it a whirl, I decided to install it locally, which to my surprise exceeded my expectations.  On a debian (or derived) distribution, installation is achieved by a simple apt-get install cherokee (or 'aptitude'...)

The web management interface provided is clean and intuitive.  The basic idea is quite interesting;  It is possible to activate the PHP interpreter with a simple mouse click; same for activating different virtual hosts as well as other options.  There are also wizards to assist with the installation of such applications and frameworks as Django, Rails or Wordpress, etc.

I thought that it would be interesting to present this project during a developers meeting at Gandi, especially for those who use GandiAI.  The presentation was well received.  After a few pow-wow, and a glance through the source code, there was nothing really unusual or risky.  This doesn't mean, of course, that there aren't any bugs in it, but at any rate clean code is a good sign!  The core is written in C with the administrator interface written in Python.

Thus is the way in which Cherokee made its appearance at Gandi, and a server image was produced.  In this way you can rapidly test a server with Cherokee pre-installed.

We have also added an "expert" mode distribution with Cherokee pre-installed... You need only to select this distribution during the steps to create your server.  And, just as a reminder, in case you haven't yet tried our service, you can always request a free trial using our online form!

For further information about Cherokee, you can visit the project website.  Additionally, you can find a comprehensive article about Cherokee in the Gnu/Linux magazine France, written [in French] by Carl Chenet in issue number 125.


E-Mail 'Goes Postal' - Gandi Mail Version 2

Okay, excuse the corny punch line, but for those who haven't heard the expression, to "go postal" simply means to go crazy.  Without delving into the origins of the phrase, suffice it to say that the term these days can also mean to go crazy or emphatic in a positive sense as well.. (or so some would like to believe ;-) )

As many may know by now, the Gandi Mail platform in the past couple of years has seen a few recurring instances of performance degradation every couple of months.  As mentioned on the Gandi Bar, our teams have been working to bring you a new and more robust mail platform.  This new platform is the culmination of a considerable amount of resource investment over many months.

I am pleased to say that the new platform is fully operational and all Gandi Mail customers are fully migrated to the new system, with the migration itself taking about six weeks or so.  We decided to stage the migration gradually over a number of weeks to minimise any impact to our customers, and for the most part (with a very small handful of exceptions), the bulk of the migration was completed without anyone even noticing ;)

Anyway, without further ado, let's take a brief tour of the new mail platform and what has changed in Gandi Mail version 2.  Oh.. and before we go on, this article will be at times a little bit technical with the use of some acronyms and other geek-speak.  Don't worry, you'll soon get the gist of what we bearded technophiles blabber on about on a daily basis!  ;)


Gandi 10th Anniversary - The Experience

To celebrate Gandi's 10th anniversary, this hair-brained idea to give away, in ten days, 55000 domains, raise a very practical question.  How, once we open the floodgates on such an operation, to maintain the highest quality of service on the site?  The festive spirit could well have transformed into a nightmare for our customers if they were suddenly unable to access their management interface.

So we took the decision to host the event on a dedicated site.  This was a hitherto dreamed of occasion to put ourselves into our customers' shoes, and use our hosting infrastructure for this event.  We defined the rules of play:  Using only the tools provided to our customers, build an architecture which was easily scalable and didn't break the bank, and to demonstrate our renowned flexibility.


- page 1 of 2