How Code Enigma does hosting

A look behind the scenes at our specially developed hosting system.

Photo of Greg Harvey
Fri, 2011-09-30 15:02By greg

We'd been talking since we started trading (almost exactly a year ago, as it happens) about setting up something that would smooth our deployment process and allow us to offer solid, professional hosting to our clients. It all started when we decided to take on running a server for Economist Group, after a number of issues with their previous provider.

We set up a nice server for them (actually, Stew at Full Fat Things did most of that), got it all hooked up with Monit and Rootkit Hunter for monitoring and intrusion detection, set up Nginx, FastCGI to accelerate PHP, Varnish, Memcached, all the usual clever stuff to make things as slick as possible. And we thought "wouldn't it be nice if we could do make another one of these with a single mouse click!"

Enter Miguel Jacq (AKA mig5 if you catch him in IRC). Mig has been working for us on a retainer basis for a while now. We hired him because we realised if we wanted some of the cool stuff we were dreaming about we'd have to hire someone dedicated to the task or it wouldn't get done.

So after getting him to sort out better monitoring (a combination of Nagios and Munin, at his suggestion) and a solid VPN, we set about the whole one-click server thing once more. But this time we took it a whole lot more seriously. This time it actually happened!

We'll leave it to Mig to tell you about the "hows" - you can read his post here: 
http://mig5.net/node/340 

I'll tell you the "whys" and the "whats" - why are we doing this and what's the plan.

Why?

Well, for a number of reasons:

  1. We wanted to make our lives easier
  2. We didn't want to be tied to someone else's platform
  3. We didn't actually *like* anyone else's platform (though this was before the very recent launch of Pantheon, which does look nice)
  4. Commercially, repeat revenue work like hosting and support is smart - project-based income only is dangerous, unless you're a one-developer-shop

What?

We now have a pretty well shaped set-up. It's for our own hosting clients use only right now, simply because too many things have been hardwired to our choices of systems (e.g. GitHub, Linode, Rackspace Cloud, etc.) and we want to increase flexibility before we throw it open to a wider audience, but we're working hard on that. Well, Mig is anyway.

Right now we can spin up Linux virtual machines anywhere we like, whenever we like, in a matter of minutes. In fact, seconds for the developer. Fill in a Webform, click Submit, go make a coffee and when you get back to your desk your server is built (by a mix of Jenkins, the CI tool, Mig's Madelon app and an open source product called Puppet for managing the software on a server).

Want a dedicated Memcached machine? No problem. Varnish + Nginx + FastCGI + Memcached? Sure. Oh, I'm sorry, no Varnish - you want that separate? Easy, no problem. Mig's built that for us. It works *great*. (Sordid details on Mig's blog.) At the moment we can support servers at Linode or the Rackspace Cloud, and for now that will probably do us, but we'll probably add more at some stage.

The real killer features of this set-up for me though are the early warning and emergency systems. You build a totally custom server with a Webform and it is *instantly* covered by Nagios real time alerting, Munin monitoring for longer-term data (very useful if a server has recurring issues and you're looking for patterns). This cover is provided by a two-vector system, two monitoring servers at entirely different ISPs on different sides of the world, keeping each other honest. And if you run Drupal (which we strongly encourage you to do), we can hook Nagios up to that too, since we are now maintaining the Nagios module for Drupal. The level of reporting available from these products beats anything I've ever seen offered elsewhere.

Tied to this is instant cover for back-up. We use Duplicity to back up data to Rackspace Cloud Files in the UK, encrypted, naturally. This happens every night on all our production servers. We have another application, again written by Mig, called Felicity which restores Duplicity back-ups on to new machines. This means we can restore a broken machine in the exact state it was in in a matter of minutes. We know what that machine looked like (thank you Puppet) and we have every custom file that was on it (thank you Duplicity) so thanks to Mig's app, Felicity, we can build a replacement.

What about VM snapshots? Well, they're grand if your hosting provider is still there, but what if it isn't? What if the whole of EC2 goes down? Surely that would never happen... *ahem* ... but if it did...? Courtesy of Felicity, we could just restore your whole system somewhere else.

Anyway, enough about the now. What's next?

The future

Well, we've got loads of ideas.

One obvious one is more ISP options. Another is support for repositories other than GitHub - we'd like it to not matter where your VCS is, as long as it's either Git or Subversion and you can add our public key to it.

But probably the biggest thing we're working on right now is knitting our CI (Continuous Integration) in to this. We're working on it for ourselves primarily, but if we get it nailed, it works really well and we think other people will like it too, we're happy to show and tell. Right now we have another Webform where we can really quickly commission Jenkins jobs across a range of servers. It's pretty simple really. Build your server, go to the Jenkins form, select your server and fill in the details for your CI job. Jenkins does the rest!

We're just applying the last layers of polish to this. The hosting is ready, we're using it with a few select clients (including an international bank and a global technology brand) and we're talking to other businesses too. The CI is ready for us to use internally and will, before too long, be something we can offer to other developers too. Watch this space!

If you're interested to hear more about our hosting, do get in touch to arrange a demo.

And if you want a live demo of the hosting and CI and how that all works together, get yourself to DrupalCamp Toulouse, where myself (in person) and Mig (via Skype) will be presenting on Saturday, 26th November at 1400 CET.