It’s hard to come up with an analogy that everyone can appreciate and which fits well enough that it will help, but I’ll try. Consider the first time you saw a TV show or movie in HD – perhaps you bought an HDTV, perhaps you upgraded from digital cable to HD, or you bought a Blu-Ray or HD-DVD player. Maybe you just saw one at Best Buy playing your favourite summer blockbuster. You see what’s on the screen, and it’s still the same, really. It’s the same movie you’ve enjoyed in the past, the same TV shows you’ve always loved. The plot hasn’t changed, the writing isn’t any better. It’s quantitatively clearer, but that provides a qualitative improvement.

If you’re a gamer, the comparison is easy. Do you remember the first time you upgraded your video card to the latest and greatest? The frame rate increased, you get more texture quality, it’s smoother. There’s less jitter, scrolling and panning is smoother. You didn’t really notice it before, but now it feels better. It’s more natural, it flows better.

And that’s really the key. The iPhone isn’t faster, because ‘faster’ doesn’t connote the right differences. The iPhone is more fluid. Moving from one application to another, or from one screen to the next, or scrolling around on a website or reading your e-mail, it flows better. It’s not just that the current iPhone is slow, it’s more that it breaks your stride.

Here’s another example. Let’s say you’ve got a room in your house with doors on two sides, directly opposite. On the other two sides, you have a couch and a TV, and in the middle, there’s a coffee table. Let’s also say you have to walk through this room to get from your bedroom to the kitchen, so it’s a trip you make fairly often. If you put your coffee table directly between the two doors, then every time you cross this room, you’ll have to go around it. It’s not a big deal, but it throws off your momentum. You’ll have to step to the side twice every time you go through the room. You’ll also have to go through the line of sight from the couch to the TV, so you might have to slow down if someone’s playing games or watching a movie, waiting for the perfect opportunity, and you’re more likely to bang your knee while going around it, maybe if you’re in a hurry or it’s dark.

Using the iPhone 3G now is a lot like that. There are a lot of situations now where the flow of action from one activity to the next is disrupted. The straight path from point A to point B has a detour in the middle, an obstacle, a delay, and it slows your momentum ever so slightly. Enough of these delays and it becomes subconsciously frustrating. The iPhone 3GS does away with that.

Now here’s the key, and here’s why it’s hard to quantify. You probably didn’t notice any of those delays. You go around that coffee table so often that you don’t even realize it’s there, but on a subconscious level, it bothers you. You don’t think to move the coffee table because you don’t realize it bothers you, but if you moved it a half a foot out of the way, you’d decrease your subconscious frustration ever so slightly, and you’d find yourself enjoying your home more (and you’d have fewer bruised shins to show for it).

I didn’t dislike Twitterrific on the iPhone, not really. It was ok, but not great. The UI was nice, but I just didn’t really care for it, preferring Tweetie. I couldn’t tell you why, I just didn’t like Twitterrific’s interface, or maybe it was the colours, or maybe it was just kind of convoluted to use. I don’t know why I didn’t care for it, but I didn’t.

Now, however, I know. Twitterrific wasn’t slow, you have to understand. I wasn’t sitting there, staring at loading screens or progress bars, waiting for it to accomplish the task I’d set it to. The real reason I didn’t care for Twitterrific was because there was some subtle, imperceptible lag when moving from one action to the next. I didn’t notice it consciously, but subconsciously it bothered me, it made me dislike the app for reasons I didn’t realize or understand. On the 3GS this week, I used Twitterrific exclusively. I didn’t try any other Twitter apps, and it didn’t occur to me to want to. Twitterrific was great, and I was happy to keep using it, because that unconscious frustration with minute waits and imperceptible lag was gone.

And that’s what it feels like to move from an iPhone 3G to a 3GS. It’s a more satisfying experience, a more enjoyable device, but not for reasons that you might think. It takes better pictures, but I don’t really care. It records video, but I don’t really care. It has faster data access, but I don’t really care. It has voice control, but I don’t really care.

The real difference between the iPhone 3G and 3GS isn’t something that you’re likely to notice unless you’re looking for it. You’ll enjoy the new phone for reasons that you can’t really explain, and when people ask why you paid an extra few hundred dollars for a not-so-different phone, you won’t regret it, but you won’t be able to defend your decision either, because you can’t reduce the experience to numbers.

It just feels better.

With the release of the iPhone 3.0 software, the iPhone 3G now has an interesting new feature – tethering. In order to do this in Canada on Rogers or Fido, you must have a 3G data plan of 1GB or higher and be using an iPhone 3G (or 3G S). You must also call Rogers or Fido and get them to enable the plan on your account.

There is another option – do it yourself. All it takes is a simple config file, and because I love you so much, I will now provide them to you at no extra charge.

Here are the simple steps to enabling tethering:

1. Copy, e-mail, or text this URL to your phone: iphone.cdslash.net
2. Visit the URL on your iPhone and choose Rogers or Fido
3. You will be asked whether you want to install the file or not. Tap Install.
4. Confirm that you want to install the file
5. After installation is finished, go to Settings > General > Network, and you should see the option for tethering appear.
6. Enable it, then connect over USB or pair via Bluetooth.

That’s it. Then all you have to do is open Settings > General > Network and enable tethering, and you’re on your way. Plug in via USB or pair via Bluetooth and you’ll find tethering enabled and working.

(Credit where it’s due: I found these files at Adam McNamara’s blog post on the topic)

Update: If you install and something goes wrong, or if you change your mind or whatever, you can uninstall the profile and revert back to the default like so:

1. Open Settings > General > Profiles
2. On this page, you’ll see a heading of ‘Configuration Profiles’, under which should be the update you’ve installed either fido_com or rogers_com. Just tap on that to see more details about it (the same screen you were provided when you installed it).
3. Tap ‘Remove’. This will remove the provisioning profile from your iPhone, which will revert the changes made.
4. Done!

Update 2: This seems to have enabled MMS on Fido as well – MMS wasn’t working for me until I installed this update (but tethering was working fine!) so if MMS isn’t working for you, you can try this too, even if you don’t want to tether your iPhone.

Update 3: If you are getting the message “to enable tethering on this account contact rogers” from your iPhone, then you’ve come to the right place. Going through these steps will fix it.

Update 4: Some people have reported problems with Visual Voicemail. Users affected can’t fetch messages or change their greeting. To see if you’re affected, open your voicemail and try to play back your greeting. If it sits and spins for a few seconds, you’re affected.Thanks to a tip from @andrewe, I’ve fixed the config files. You’ll need to first remove the old ones, then install the new ones.

To uninstall, go to General > Profiles, tap the profile, and remove it. After that’s done, go back to the page and re-download for your carrier, and Visual Voicemail should work again.

Atomic Migrations

The actual act of copying the data from one server to another is simple. It’s not difficult to take a point-in-time snapshot of a Rails app and move it to another server. The difficult part, when dealing with a dynamic website, is keeping the two servers in sync.

Because of DNS propagation issues, you will continue to have clients hitting the old server while some users start hitting the new server as well. This can cause concurrency issues; if a user is hitting the old server, then they may make modifications to their account (or in our case, modifications to their team) that will not show up on the new site once their DNS changes. This can be a pretty big issue, and we came up with a pretty useful method of dealing with the issue that eliminates concurrency issues with minimal impact on the users.

Testing the Server

The first step was to copy the app over and get it ‘working’. To do this, we used a dump of the MySQL database, using the mysqldump utility included with MySQL. We got the app up and working, tested it, and made sure that it was going to work properly (i.e. all Ruby gems were installed, etc.). Next we took a fresh dump of the database using the –master-data option to mysqldump, which includes the information necessary to set up a replication slave.

MySQL Replication

MySQL provides realtime replication between two servers, and it’s pretty simple to set up. Setting up MySQL replication between the two servers required briefly opening MySQL to the outside world (having it listen on port 3306), but we used iptables firewalling rules to ensure that no one else could connect. Once replication was set up, we knew that the new server would be an exact replica of the old server. The application was working, the database was identical, and so on. Any changes happening on the old server would happen instantly on the new server.

MySQL provides functionality called multi-master replication, which allows two servers to be written to, and have them push their changes back and forth between two servers. While I’ve set this up and used it in the past, it’s more complicated to set up and can be a pain to maintain. If replication fails in one direction, you can get inconsistent results between the two. For this reason, we wanted to ensure all of our traffic would switch over all at once – an atomic migration. Either all traffic hits the new site, or all traffic hits the old site. This is where Apache comes in.

Apache’s mod_proxy

We use Apache to run our websites, including our rails applications (using Passenger’s mod_rails). The solution we came up with was to make use of Apache’s built-in modules to ease our transition. We set up two config files, the original (which we had already) and a new configuration that, instead of serving pages, proxied all traffic to the secondary server. In essence, all traffic to the old site was transparently proxied to the new server, and the responses were proxied back. This is the config snippet we used:

This had two benefits: first of all, it was instantaneous and atomic – all users would see traffic from the same server, either the old one or the new one. We wouldn’t have any frustrating situations where one user would hit the old server and another user would hit the new one. Secondly, it’s quick and easy. If there were problems on the new server that we hadn’t forseen, it would be easy for us to switch everything back to the old server (by swapping out the config files again).

The Changeover

In order to ensure we didn’t run into any collisions with replication, our solution was to stop Apache entirely for a second, then bring it back online with the new config. This ensured that if there was any latency between the master (old) and slave (new) servers, it would catch up in the second that we were down. After we made the change, we checked, and the site was up and running. Users were all logged out, but beyond that the experience was largely smooth. Once we knew the server was working, we changed the DNS to point to the new server, and the changeover had finished. We were live on our new (virtual) server.

In Part 2, I plan to discuss the initial scaling issues we found, as we tried to learn the bounds of Mosso’s virtual servers, and of our own application and server design.

Ok, the similes suck, I get it. I’ll stop.

Enter The Cloud™. The great thing about The Cloud™ is that it’s all a nebulous, formless void, allowing you to create light and form from nothingness – in this case, servers. Let me explain.

In The Cloud™, you don’t buy a server then upgrade it later. You rent an abstraction of a server from people that (presmuably) have bought thousands of servers, each of which is more expensive than my house, and combined them together into a giant cluster of ultimate power. You rent a little space on this giant cluster and bend it to your will.

One of the domains my company owns is Cricket.com, for which we also have an associated fantasy sports league thing. Now, you’ll be forgiven if you don’t know anything about the game, because god knows I don’t. In fact, the only thing I know about Cricket is that they’re bigger fans of it in India than they are of housing, money, and oxygen combined. People from India are fanatical about the game, and so when the Indian Premier League (of Cricket; like the MLB or NHL or MBA or whatever) started up this year and we were providing the only officially licensed online fantasy sportsbook kind of game, we got some traffic.

Lots. Lots of traffic.

We had previously had the site running on a server at Serverbeach (a Peer1 company), which was fine, but it was underpowered because I had specced it out for what we needed at the time, and not for an entire subcontinent hitting the server all at once, which is what started happening this week. Which sucked. The site is running on Ruby on Rails, using MySQL, memcached, and duct tape, and was running well until the IPL started.

The site started getting pretty slow, and so I thought hey, let’s stick it in the cloud and that’ll make everything better. Brian and I, harnessing the power of the free snacks the company stocks the kitchen with, migrated the server over to a roughly equivalent machine at Mosso, which is where things start to fall apart. Not because of Mosso, per se, but because we were getting dramatically more traffic than we were planning for. Still, the timing was kind of conspicuous, and we needed to update. Upgrading the machine from a 4GB machine to 8GB provided almost no increase in performance whatsoever, and we began to become concerned.

Our first step was to fork services off. Until then, the entire site had been running on one server; we added a second server at Mosso just to run MySQL, and that provided some benefit, but nothing on the scale we needed. Enough to hobble along, but not enough long-term. That was when we decided to evolve in a different direction – horizontally.

Vertical scaling involves growing up – you have a server, you make it bigger. More memory, more CPU, more disk, and so on. Horizontal scaling involves growing *out* – more smaller servers instead of one large server. Building a cluster on Rackspace’s cluster seemed like the next logical step, and that’s what we did.

What we’ve ended up with is a database server, three web servers, and a load balancer in front of them. This solution is more scalable, and also seems to perform better. How much better? Enjoy some graphs!

fantasy.cricket.com would die at around 30 accesses/second

This graph shows the apache ‘accesses per second’ for our old (8GB) server. As you can see load would rise to about 30 requests per second, after which point the server would be so overloaded that we could no longer get statistics out of it. We would still be serving pages, but only about 30 of them per second, and they would be slow – anywhere from 5-30 seconds per page load. Unacceptable. We would only start to get more data back from the server once load dropped below 30 accesses per second. This was obviously unacceptable.

One of our current servers, showing a marked increase in capacity

This graph shows the same data as above, but for one of our cluster’s web servers. The other servers show nearly identical usage, so the real numbers are three times what you see here. You can see that during peak hours, we average around 15-18 accesses per second on this server. The statistics for the other two servers are similar, giving us around 45-55 accesses/sec total, and with very little latency. Whereas before we were dying at 30 accesses/sec, we’re currently averaging that throughout the whole day.

Think about that. We’re using the same software and configuration, using the same amount of total memory, and paying about the same, but we can now handle half again the load as before, and we’re spiking to three times our previous maximum load.

The moral of the story? If you’re going to move into the cloud, especially somewhere like Mosso, you have to be willing to re-think your architecture. The old plan of ‘bigger servers’ just doesn’t work – or at least, it didn’t for us – but now, for about the same price, we can handle up to three times our previous maximum load without any trouble at all.

We won’t know what our maximum is until we hit it, but I have a sneaking suspicion that it’ll be a while before that happens, if at all. Until then. I’m going to enjoy sleeping through the night without Pingdom whispering sweet nothings into my phone at 3 AM.

Jenny: Wait. This thing doesn’t have a positronic brain. Just a bunch of wires. Also, it’s very hard to take apart.
Jenny: This phone is not waterproof and does not float at all.

If you are lol-inclined, this will lol you.