What happens if you’re Notch’d?

Edit: A nice long thread about it produced on reddit! Check it out for a few answers.

Yesterday our Farm Fortress tweet was RT by mighty @notch, sending it past his some 800.000 followers. Today I thought I’d share what came of it and what it has meant to our project (well short term, since this was yesterday). Here are some numbers!

August 16-17 (24h)

• 7863 visits, by new visitors.
• 54% were from the US.
• 3.95% use Internet Explorer, 51% use Google Chrome.
• 3614 confirmed game starts (= 14Mb download & multiplayer connection up).
• 378 players registered their email.
• 264 players have also verified/activated their email.
• 228 hours collective playtime (registered users only).

Rather immediately, our web front end machine chocked out of low memory. This was discovered to be caused by two things.

Firstly, I had forgotten some (soon to be added) database request code on the front page. The generated queries were not even optimized yet, as the function wasn’t meant to be online.

Secondly, web server is hosted on a VPS which does not allow you to have swap space. This may have been the worst factor. Memory quickly ran out and I could not send a single command to it, or connect via ssh. This affected the game since it requires a http response when it loads a list of game servers of the web server. Obviously the page embedding the game could not load either. The game data was loaded from a CDN.

After a while, I found an option to temporarily boost memory for the machine on the fly, and that help getting the site responsive again. I could then also remove the unused but heavy database queries. 

When notch retweets something, within seconds he sends a horde of 1000 or so users (and bots) to jump on your site and try to load it. This sudden wave should be manageable, but kills a lot of small sites.

Some really good news - it looks like I underestimated the load on the game servers, making the game a (potentially) better business. Technically the game can be compared to a very dynamic and persistent multiplayer 2d counter strike, for each player the server runs a full game simulation (even if player is alone in the fortress). This means grouping players together improves performance, but we can also handle a lot of “loners”. At most the game servers had 50-60 players on them, but causing very little load. A cheap AMD Operton will easily handle 150 instances by itself, maybe even over 200. Hopefully we’ll see that one day.

Having a small project and getting Notch’d is the equivalent of throwing a cup of gas on a little fire. It’ll burn real nice to start with, but pretty soon it will need some better fuel to keep burning.