Geekery
Fido and Rogers Tethering on iPhoneOS 3.0
Jun 17th
Update: As of iPhoneOS 3.1, the iPhone will not accept unsigned config files (unless you have jailbroken); as such, this file will no longer work. If you have installed it (or any similar file for other carriers, like AT&T, T-Mobile, etc.) and upgraded to iPhoneOS 3.1 or 3.1.2, you will need to remove it. Open Settings > General > Profiles (at the bottom). Tap, then remove the profile you see there. Tethering should hopefully show up for you afterwards. I’ve preserved the rest of the entry for historical purposes, but it will no longer work on iPhone OS 3.1 or higher. More >
Memcached, gems, and Macports
Jun 10th
I actually really like Macports; it’s got a lot of software, it’s well-maintained, it works really well. Kudos to the Macports guys.
We have been having a problem with ruby though, or specifically with the memcached gem for Ruby. The problem is that it won’t compile with the latest versions of libmemcached. The latest, you see is 0.30, but it won’t compile with 0.26 or higher. In fact, if you’re using version 0.14 of the memcached gem, it won’t compile with 0.24 or lower. Oh, and it won’t compile with 0.25 either.
In fact, it won’t compile at all with any version of libmemcached for most people, and if it does it rarely works.
Unfortunately, the version of libmemcached required, 0.25.14, is only available (as far as I can find) on Evan Weaver’s blog post mentioning it. For people like me who like to keep everything package-managed when possible, this is a hassle. There’s no one-line command to make everything work, set it all up, and get it going. Kind of ugly, imho.
Fortunately, the ports system is as flexible as it is powerful, and you can easily hack the ports file to know about this new, better version. For you, I have done this, and you can find it in my github repo if you so desire, though I’ll include it here for reference. Behold!
Notice the ‘variant’ section at the bottom; that’s all I had to add to add support for this edge-case to Macports. Hopefully, as a result, the Macports maintainer will be willing to add this change into the official portfile. I suppose we’ll see. In the meantime, if you want to add this functionality, you need only replace your current portfile with the one in my GitHub repo. Instructions, paths, and commands are all at that URL. Enjoy!
Fantasy Cricket: Initial Migration (Part 1)
May 20th
We’ve recently migrated cricket.com’s Fantasy Cricket application from a single server running at ServerBeach to a multi-server cluster running on Mosso‘s Cloud Servers system. In an attempt to document the problems we’ve had and the solutions we’ve found, I’ve decided to write down my thoughts and memories, and the processes we went through, in the hopes that it will help others.
Atomic Migrations
The actual act of copying the data from one server to another is simple. It’s not difficult to take a point-in-time snapshot of a Rails app and move it to another server. The difficult part, when dealing with a dynamic website, is keeping the two servers in sync.
Because of DNS propagation issues, you will continue to have clients hitting the old server while some users start hitting the new server as well. This can cause concurrency issues; if a user is hitting the old server, then they may make modifications to their account (or in our case, modifications to their team) that will not show up on the new site once their DNS changes. This can be a pretty big issue, and we came up with a pretty useful method of dealing with the issue that eliminates concurrency issues with minimal impact on the users.
Testing the Server
The first step was to copy the app over and get it ‘working’. To do this, we used a dump of the MySQL database, using the mysqldump utility included with MySQL. We got the app up and working, tested it, and made sure that it was going to work properly (i.e. all Ruby gems were installed, etc.). Next we took a fresh dump of the database using the –master-data option to mysqldump, which includes the information necessary to set up a replication slave.
MySQL Replication
MySQL provides realtime replication between two servers, and it’s pretty simple to set up. Setting up MySQL replication between the two servers required briefly opening MySQL to the outside world (having it listen on port 3306), but we used iptables firewalling rules to ensure that no one else could connect. Once replication was set up, we knew that the new server would be an exact replica of the old server. The application was working, the database was identical, and so on. Any changes happening on the old server would happen instantly on the new server.
MySQL provides functionality called multi-master replication, which allows two servers to be written to, and have them push their changes back and forth between two servers. While I’ve set this up and used it in the past, it’s more complicated to set up and can be a pain to maintain. If replication fails in one direction, you can get inconsistent results between the two. For this reason, we wanted to ensure all of our traffic would switch over all at once – an atomic migration. Either all traffic hits the new site, or all traffic hits the old site. This is where Apache comes in.
Apache’s mod_proxy
We use Apache to run our websites, including our rails applications (using Passenger’s mod_rails). The solution we came up with was to make use of Apache’s built-in modules to ease our transition. We set up two config files, the original (which we had already) and a new configuration that, instead of serving pages, proxied all traffic to the secondary server. In essence, all traffic to the old site was transparently proxied to the new server, and the responses were proxied back. This is the config snippet we used:
This had two benefits: first of all, it was instantaneous and atomic – all users would see traffic from the same server, either the old one or the new one. We wouldn’t have any frustrating situations where one user would hit the old server and another user would hit the new one. Secondly, it’s quick and easy. If there were problems on the new server that we hadn’t forseen, it would be easy for us to switch everything back to the old server (by swapping out the config files again).
The Changeover
In order to ensure we didn’t run into any collisions with replication, our solution was to stop Apache entirely for a second, then bring it back online with the new config. This ensured that if there was any latency between the master (old) and slave (new) servers, it would catch up in the second that we were down. After we made the change, we checked, and the site was up and running. Users were all logged out, but beyond that the experience was largely smooth. Once we knew the server was working, we changed the DNS to point to the new server, and the changeover had finished. We were live on our new (virtual) server.
In Part 2, I plan to discuss the initial scaling issues we found, as we tried to learn the bounds of Mosso’s virtual servers, and of our own application and server design.
Scaling clouds: more, not bigger
Apr 27th
Scalability is a problem. It always has been. The solution in the past has always been either to buy more servers to cope with your peak load (at great expense), or to… I dunno. Crash, I guess. Or go broke. Or lose all your users. Scalability is a pain because you’ve got all manner of bottlenecks; I/O system, memory, and CPU are generally your big evildoers, and building a system that can handle all of those issues at once is like trying to build a building that is both the tallest, widest, and most expensive – complicated, and… expensive.
Ok, the similes suck, I get it. I’ll stop.
Enter The Cloud™. The great thing about The Cloud™ is that it’s all a nebulous, formless void, allowing you to create light and form from nothingness – in this case, servers. Let me explain.
In The Cloud™, you don’t buy a server then upgrade it later. You rent an abstraction of a server from people that (presmuably) have bought thousands of servers, each of which is more expensive than my house, and combined them together into a giant cluster of ultimate power. You rent a little space on this giant cluster and bend it to your will.
One of the domains my company owns is Cricket.com, for which we also have an associated fantasy sports league thing. Now, you’ll be forgiven if you don’t know anything about the game, because god knows I don’t. In fact, the only thing I know about Cricket is that they’re bigger fans of it in India than they are of housing, money, and oxygen combined. People from India are fanatical about the game, and so when the Indian Premier League (of Cricket; like the MLB or NHL or MBA or whatever) started up this year and we were providing the only officially licensed online fantasy sportsbook kind of game, we got some traffic.
Lots. Lots of traffic.
We had previously had the site running on a server at Serverbeach (a Peer1 company), which was fine, but it was underpowered because I had specced it out for what we needed at the time, and not for an entire subcontinent hitting the server all at once, which is what started happening this week. Which sucked. The site is running on Ruby on Rails, using MySQL, memcached, and duct tape, and was running well until the IPL started.
The site started getting pretty slow, and so I thought hey, let’s stick it in the cloud and that’ll make everything better. Brian and I, harnessing the power of the free snacks the company stocks the kitchen with, migrated the server over to a roughly equivalent machine at Mosso, which is where things start to fall apart. Not because of Mosso, per se, but because we were getting dramatically more traffic than we were planning for. Still, the timing was kind of conspicuous, and we needed to update. Upgrading the machine from a 4GB machine to 8GB provided almost no increase in performance whatsoever, and we began to become concerned.
Our first step was to fork services off. Until then, the entire site had been running on one server; we added a second server at Mosso just to run MySQL, and that provided some benefit, but nothing on the scale we needed. Enough to hobble along, but not enough long-term. That was when we decided to evolve in a different direction – horizontally.
Vertical scaling involves growing up – you have a server, you make it bigger. More memory, more CPU, more disk, and so on. Horizontal scaling involves growing *out* – more smaller servers instead of one large server. Building a cluster on Rackspace’s cluster seemed like the next logical step, and that’s what we did.
What we’ve ended up with is a database server, three web servers, and a load balancer in front of them. This solution is more scalable, and also seems to perform better. How much better? Enjoy some graphs!
fantasy.cricket.com would die at around 30 accesses/second
This graph shows the apache ‘accesses per second’ for our old (8GB) server. As you can see load would rise to about 30 requests per second, after which point the server would be so overloaded that we could no longer get statistics out of it. We would still be serving pages, but only about 30 of them per second, and they would be slow – anywhere from 5-30 seconds per page load. Unacceptable. We would only start to get more data back from the server once load dropped below 30 accesses per second. This was obviously unacceptable.
One of our current servers, showing a marked increase in capacity
This graph shows the same data as above, but for one of our cluster’s web servers. The other servers show nearly identical usage, so the real numbers are three times what you see here. You can see that during peak hours, we average around 15-18 accesses per second on this server. The statistics for the other two servers are similar, giving us around 45-55 accesses/sec total, and with very little latency. Whereas before we were dying at 30 accesses/sec, we’re currently averaging that throughout the whole day.
Think about that. We’re using the same software and configuration, using the same amount of total memory, and paying about the same, but we can now handle half again the load as before, and we’re spiking to three times our previous maximum load.
The moral of the story? If you’re going to move into the cloud, especially somewhere like Mosso, you have to be willing to re-think your architecture. The old plan of ‘bigger servers’ just doesn’t work – or at least, it didn’t for us – but now, for about the same price, we can handle up to three times our previous maximum load without any trouble at all.
We won’t know what our maximum is until we hit it, but I have a sneaking suspicion that it’ll be a while before that happens, if at all. Until then. I’m going to enjoy sleeping through the night without Pingdom whispering sweet nothings into my phone at 3 AM.
Digital Downloads and Analog Media
Apr 4th
Note: The following post is very Apple-centric, re: iTunes, AppleTV, etc. Substitute if you like any similar technology except Windows-specific ones. Oh wait, that only leaves Apple.
I’m not asking too much. I mean, I don’t watch a lot of TV. Any, in fact. I don’t even own a TV. I don’t have a DVD player or TiVo, and I don’t have cable.
If you consider my impact on the television industry, you’ll find it’s not really worth factoring into any equation. And yet, if they want to reach into my pockets and clean out any part of my paycheque that doesn’t go to food, rent, or Starbucks, it’s a pretty simple matter. In fact, I’ll tell them right now what needs doing.
A tweet from the lovely Shannon McKarney got me started down this line of thought, and brought up some old ideas I’d had. Not complicated ideas, nothing I’d consider revolutionary, but pretty straightforward ideas. Her question revolved around how, given the demand indicated by the Wolverine workprint leak and the feeding frenzy of downloading it set off, can movie companies use the internet to make money? There’s a market, how do you tap it?
Simple. Give me what I want the way I want it.
Movies: easy. Release in theatres on a Friday. Next Friday, release on iTunes in HD for rent. Depending on the movie, MAYBE wait two weeks. Two weeks after that, release on Blu-Ray and DVD. Down the road, THEN you come out with extras. BD discs with behind-the-scenes stuff, all the extra fluff, bonus sequences, director’s cuts, and so on.
I want to watch movies, but that’s not what theatres are about to me. Theatres are about pushing your way through a crowd, about getting the worst seats in the house because there are no two adjacent seats together. They’re about overpriced, substandard food, five dollar soft drinks, sticky floors, and people talking to each other behind you and kicking the back of your seat.
Imagine not having to pay for that. Imagine sitting down at home, cooking up some pasta, making a sandwich, grabbing a beer, blending a smoothie, or BBQing some steaks, and then putting a movie up on the big screen. Enjoying a movie even when your baby is sick, or your child is teething. Recovering from the flu, or staying in with your sweetheart on a rainy Sunday evening. It’s a better experience all around.
Some people will prefer the classic, and for groups of any reasonable size, the theatre will just make more sense. It won’t kill the theatres, but it will augment them. More people will watch movies because the barrier to entry will be lowered. The giant screens are irreplaceable, but that just justifies them staying in business and charging more. It’s an experience, but you don’t always want it.
So what about TV shows? Possibly a little more complicated, but I have an idea about that too. First things first: confessions. I download all the TV I watch. I don’t pay for it, I don’t have cable. No one gets anything from me. If I had the following solution, I would pay.
Imagine a scenario: It’s Monday night – Lost night. You’ve got a long day at work, but at the end of it, you drive home. Throw together a dinner or some leftovers, sit down on the couch, turn on your AppleTV, and flip on Heroes. Watch it, in all its HD glory. When your wife gets home from her late-late shift, she can watch it again. The key factor: you don’t have cable. You never had. No DVR, no anything.
You bought a subscription to Heroes. You paid, up-front, for the entire season. Every Monday morning, your AppleTV downloads the next episode of Heroes for you, automatically. When you get home, you don’t have to wait for it to download, you just jump right in. The AppleTV respects airdates by requesting a key from Apple’s servers before it’s allowed to play it. It knows what time it’s supposed to be available, and once that time comes, it will let you play it. It verifies with the servers, and the content is unlocked – for good. It costs a little extra to get it in HD, but it’s worth it.
The end result? High-quality digital versions of the show, each episode sent to your home so that you can watch it on your own terms, without having to subscribe to cable TV, get the HD pack, get the HD PVR. For me, it would be about $78/mo before tax to get the same thing. That’s a lot of money that most people aren’t willing to pay – especially if, like me, you only watch a few hours of TV per week. No wonder people pirate it.
So think about that. I can pay $78 to Shaw, who then turns around and buys access to the channels, who sell access to advertisers to make money for production (or syndication rights). For that, I get hundreds of channels I’ll never watch, and years of content I have no interest in. Or, I can pay $20 (or even $30!) to subscribe to an entire season of Heroes, or any other show (adjust pricing accordingly for different types of shows).
It’s a no-brainer.
So please, accept the digital age. Accept that you can skip the middleman (or at least, a lot of the middle men). Accept that you can lower your overhead. Accept that impulse-buying is a good thing. Accept that having an entire season of Heroes paid off by the fifth episode is a good thing. Accept that you’ll be able to have TV series with long-running plots, because people will be able to easily start at the beginning, rather than wherever the networks happen to be airing them at. Accept that on-demand is better over the internet than through some broken cable box UI.
How do you stop piracy? You don’t. But if the iTunes Music Store can be more popular than free, I think you’ll find that giving people quality will encourage them to part with their hard-earned money.
Accept change, because the world is changing, with or without you. After all, I can already do everything I’ve talked about above, I’m just not paying you for the privilege.