Game In The Cloud #1
by Konrad Kiss · 05/12/2011 (4:58 pm) · 12 comments
This is the first part of a - hopefully - series of blogs that discuss the planning and creation of a server infrastructure in the Amazon cloud.
Creating a game where you must rely on a number of privately run game servers is a daunting task. This is not necessarily constrained to MMOs. A casual Facebook game is as good an example as any.
Creating a game that also has a server side opens up a whole new can of worms in general. You might or might not be prepared to spend half of your development time working on code that is not even a direct part of your game. A custom master server, a login service, centralized and secure authentication, syncing services (ie. global timeofday time), an automation to generate patches and installers, tools to manage your users, your website, your game, a webshop probably that will generate revenue, statistics, and so on...
While I do find making indie games romantic, I also believe that in order to be able to make more games you need to be successful. Often, making indie games is considered analogous to not making enough money off it. I think that you are successful with your game when your game is able to generate income and you are able to use this opportunity well. Greedy and puritan, perhaps, but true. There's no way to run an MMO without money, really, so you need to make sure it performs well. It also goes the other way around - there's not enough will in this world to put up with the players of an MMO for too long if it doesn't make you enough money.
If the question for you is not whether you should start making an MMO then it's whether you'll be able to pay the upkeep for it - host the servers, develop the additional services and tools, spend the additional time required to administer these and of course actually pay for everything.
Luckily for us, cloud based computing is becoming more and more mainstream. The prices are now so low, that you can run a small linux server through Amazon Web Services' (AWS) EC2 service for a little more than $10 a month. This is more than enough to host a website to start with.
To put price in perspective, we used to have 5 really strong servers co-located at a hosting company when we decided to move over to AWS. We doubled the number of (virtual) servers - we broke up tasks and spread them out on multiple servers - yet we pay less than half of what we used to with the hosting company and get much better bandwidth, speed in general and much much better future options. Cloud based computing is cheap partially due to the fact that you pay for what you use. These prices are continuously falling, too.
I'll return to Amazon prices in a later blog after I have introduced AWS services and how they can be used when making an online game. Then it will make much more sense to detail what is cheap and what you should be sure to use only as required.
The kinds of services available in the Amazon cloud are services that will take many of the more problematic issues off your shoulders. Most of these issues could be show-stoppers. See any MMO on launch day / week / month and you get the idea.
You have a great game, great servers, all the services are in place - what could go wrong? Many of the AAA MMO titles had the same background and then went down the drain.
It might sound funny at first, but take my word, your most dreaded problem should be traffic. A lot of traffic. Meaning a huge spike of new users storming your game after PC Gamer was so nice as to give you a quarter page interview about the game. Now what?
This actually happened to me with a web-based "MMO" in around 2003. I wasn't prepared and wasn't able to scale the service. I was able to keep about 10k players coming from that article, but lost far many more due to the inability of the web and database servers to endure the strain. Simply put, we were unprepared for that much traffic.
But I'm sure you've also seen this in real life with AAA desktop MMOs. Anyone been there at Rift's launch? I heard there were people who had to wait 12 hours at the login screen. While TRION did a terrific job with the game itself, they were still caught off guard with their login services for 2 weeks after launch. They had a stable, well-designed game but they were not able to scale a service as registrations exceeded their wildest dreams.
How does an MMO die? It does not scale. Scaling has many sides, only part of which is actual technical scaling of resources. I think it also means you being able to quickly and continuously adapt to changes by making the right decisions.
Through this series I will do my best to share my experiences in creating an automatically scaling MMO server infrastructure - backing up our Torque 3D based MMO - that is able to handle hundreds of thousands of players, and hopefully I'll be able to at least give you directions on how to get started with your own setup using AWS.
As a part of this series, I'll be sharing a stripped down version of our master server code, but more importantly I will explain how you can write your own and expand on Torque's current capabilities. I will also offer code snippets in various programming languages as much as possible.
This first part was just this rant. In part two I'll introduce the different AWS services that the series will refer to. In the parts following #2 I'll show you the different game services and the way they are realized through AWS. I will also show you how these services interact and how they are wired to the game. I will also spend some time on the topic of using YouTube's API to add cool features to your site and make others pay for it. I intend to share a bunch of flow charts, code and urls. Better yet, there will be no half naked centaurs! :)
...
So this was the easy part. I just had to post this to finally get moving with the series - I've been planning on starting it for a while now. Please let me know if you have any specific questions about the topic in general, so I can prepare with an answer in the following blogs. I hope this will be useful and by time we'll see more dynamically scaling game services from the Torque community.
Thanks for reading this. I know it's tough, with no pictures and all. Sorry about that.
-- Konrad
@konradkiss
KonradKiss @ GitHub
konradkiss.com
The MMO dilemma
Creating a game where you must rely on a number of privately run game servers is a daunting task. This is not necessarily constrained to MMOs. A casual Facebook game is as good an example as any.
Creating a game that also has a server side opens up a whole new can of worms in general. You might or might not be prepared to spend half of your development time working on code that is not even a direct part of your game. A custom master server, a login service, centralized and secure authentication, syncing services (ie. global timeofday time), an automation to generate patches and installers, tools to manage your users, your website, your game, a webshop probably that will generate revenue, statistics, and so on...
While I do find making indie games romantic, I also believe that in order to be able to make more games you need to be successful. Often, making indie games is considered analogous to not making enough money off it. I think that you are successful with your game when your game is able to generate income and you are able to use this opportunity well. Greedy and puritan, perhaps, but true. There's no way to run an MMO without money, really, so you need to make sure it performs well. It also goes the other way around - there's not enough will in this world to put up with the players of an MMO for too long if it doesn't make you enough money.
If the question for you is not whether you should start making an MMO then it's whether you'll be able to pay the upkeep for it - host the servers, develop the additional services and tools, spend the additional time required to administer these and of course actually pay for everything.
Luckily for us, cloud based computing is becoming more and more mainstream. The prices are now so low, that you can run a small linux server through Amazon Web Services' (AWS) EC2 service for a little more than $10 a month. This is more than enough to host a website to start with.
To put price in perspective, we used to have 5 really strong servers co-located at a hosting company when we decided to move over to AWS. We doubled the number of (virtual) servers - we broke up tasks and spread them out on multiple servers - yet we pay less than half of what we used to with the hosting company and get much better bandwidth, speed in general and much much better future options. Cloud based computing is cheap partially due to the fact that you pay for what you use. These prices are continuously falling, too.
I'll return to Amazon prices in a later blog after I have introduced AWS services and how they can be used when making an online game. Then it will make much more sense to detail what is cheap and what you should be sure to use only as required.
The kinds of services available in the Amazon cloud are services that will take many of the more problematic issues off your shoulders. Most of these issues could be show-stoppers. See any MMO on launch day / week / month and you get the idea.
How an MMO dies
You have a great game, great servers, all the services are in place - what could go wrong? Many of the AAA MMO titles had the same background and then went down the drain.
It might sound funny at first, but take my word, your most dreaded problem should be traffic. A lot of traffic. Meaning a huge spike of new users storming your game after PC Gamer was so nice as to give you a quarter page interview about the game. Now what?
This actually happened to me with a web-based "MMO" in around 2003. I wasn't prepared and wasn't able to scale the service. I was able to keep about 10k players coming from that article, but lost far many more due to the inability of the web and database servers to endure the strain. Simply put, we were unprepared for that much traffic.
But I'm sure you've also seen this in real life with AAA desktop MMOs. Anyone been there at Rift's launch? I heard there were people who had to wait 12 hours at the login screen. While TRION did a terrific job with the game itself, they were still caught off guard with their login services for 2 weeks after launch. They had a stable, well-designed game but they were not able to scale a service as registrations exceeded their wildest dreams.
How does an MMO die? It does not scale. Scaling has many sides, only part of which is actual technical scaling of resources. I think it also means you being able to quickly and continuously adapt to changes by making the right decisions.
Just tell us what you want already
Through this series I will do my best to share my experiences in creating an automatically scaling MMO server infrastructure - backing up our Torque 3D based MMO - that is able to handle hundreds of thousands of players, and hopefully I'll be able to at least give you directions on how to get started with your own setup using AWS.
As a part of this series, I'll be sharing a stripped down version of our master server code, but more importantly I will explain how you can write your own and expand on Torque's current capabilities. I will also offer code snippets in various programming languages as much as possible.
This first part was just this rant. In part two I'll introduce the different AWS services that the series will refer to. In the parts following #2 I'll show you the different game services and the way they are realized through AWS. I will also show you how these services interact and how they are wired to the game. I will also spend some time on the topic of using YouTube's API to add cool features to your site and make others pay for it. I intend to share a bunch of flow charts, code and urls. Better yet, there will be no half naked centaurs! :)
...
So this was the easy part. I just had to post this to finally get moving with the series - I've been planning on starting it for a while now. Please let me know if you have any specific questions about the topic in general, so I can prepare with an answer in the following blogs. I hope this will be useful and by time we'll see more dynamically scaling game services from the Torque community.
Thanks for reading this. I know it's tough, with no pictures and all. Sorry about that.
-- Konrad
@konradkiss
KonradKiss @ GitHub
konradkiss.com
About the author
http://about.me/konrad.kiss
#2
I can't wait for your next blog and I'm checking the AWS now!
Luck!
05/12/2011 (5:56 pm)
You absolutely got all my attention. Specially now that I'm worried about my servers.I can't wait for your next blog and I'm checking the AWS now!
Luck!
#3
At my new job, I have been primarily focused on building out our cloud based game systems (save games and social systems) so I have been working heavily with Amazon AWS since the beginning of the year and I continue to be more and more impressed with them. They are constantly adding onto their services (a big new and useful feature almost every other week this year).
This isn't just stuff for MMOs and Facebook games anymore. More and more, game players expect their gaming data to be accessible from anywhere and any device. They also expect to be able to interact with their friends in social ways in every game (even if just with leaderboards and achievements) and they expect their games to be more of a service than a product with ongoing, continual updates to keep the game fresh and interesting. You cannot serve the needs of these kinds of customers without a solid backend.
Some parts of the game industry have been getting themselves in a tizzy over how "freemium" is destroying the value proposition of games but they seem to be turning a blind eye to the sheer volume of gamers who want to consume (and pay for) games which are woven tightly into their online, always connected, always changing networks of friends.
05/12/2011 (8:05 pm)
I've been eagerly waiting for you to start this blog series =)At my new job, I have been primarily focused on building out our cloud based game systems (save games and social systems) so I have been working heavily with Amazon AWS since the beginning of the year and I continue to be more and more impressed with them. They are constantly adding onto their services (a big new and useful feature almost every other week this year).
This isn't just stuff for MMOs and Facebook games anymore. More and more, game players expect their gaming data to be accessible from anywhere and any device. They also expect to be able to interact with their friends in social ways in every game (even if just with leaderboards and achievements) and they expect their games to be more of a service than a product with ongoing, continual updates to keep the game fresh and interesting. You cannot serve the needs of these kinds of customers without a solid backend.
Some parts of the game industry have been getting themselves in a tizzy over how "freemium" is destroying the value proposition of games but they seem to be turning a blind eye to the sheer volume of gamers who want to consume (and pay for) games which are woven tightly into their online, always connected, always changing networks of friends.
#4
And don't forget to always include a link to your game so we can see a live, practical example of your theory. :)
- Dave
05/12/2011 (8:05 pm)
Sweet, Konrad. I'm looking forward to reading this series.And don't forget to always include a link to your game so we can see a live, practical example of your theory. :)
- Dave
#5
05/12/2011 (10:03 pm)
Exciting! Looking forward to the knowledge your experience has to offer. Fascination abounds :) 
#6
Looking forward to this Konrad ... nice article.
05/12/2011 (10:13 pm)
Woot ... between Konrad and Steve ... hell, you guys are my heroes. :)Looking forward to this Konrad ... nice article.
#7
05/13/2011 (7:12 am)
This will be great to hear. I've been using AWS for testing games for about three months now, and I have nothing but good things to say about it. You pay only the hours you use (my first month I paid all of $1.95 for a couple of hours of testing, lol), and the quality of the service is top-notch. 
#8
05/13/2011 (11:34 am)
Nice to see you still making headway. Nice write up.
#9
To be honest, I doubt I'll be saying new things for those of you who have been working with AWS for a while now, but I'll do my best. :) It too amazed me and I find myself thinking what else I could put up there. :)
I'll try to keep these blogs 5-7 days apart so they don't steal limelight. And to be honest, with all the grinding going on that would probably be a pace I could keep.
@Steve: I was thinking of including a pic of Catchick(tm)! But I have to work out the legal aspect of that first. :)
@Guimo: Thanks! I hope you will give AWS a try! The cool thing is I just realized that they created a free tier where anyone can use a micro instance and S3 too I believe free for a year to get a taste of things. So there's no reason not to give it a try. :)
@Matt: I plan to cover / make use of EC2, S3, CloudFront, RDS, SNS CloudWatch (and private metrics), SeS and related services such as EIP and so on.. I'll probably not go into VPC, ELB, Route 53, MapReduce, SimpleDB, BeanStalk, CloudFormation, etc.. because I haven't yet had a chance to try them. As you said, this is not just for the games I mentioned - the power to scale makes it the right choice for anything online.
@David: Thank you. :) Well, I'm a little worried about being too informative with this, since I'll be sharing information about Xenocell essentially, but I'll try to balance information and security somehow. :) Also, Xenocell is still not in an active public beta, but it's getting there fast! Hopefully by the time I get to the end of this series I'll be able to invite anyone to give it a try.
@Michael: Thank you very much. :) I hope you won't be disappointed.
@Quinton: Thanks. :) I've got so much from this site and its community that I'll be giving back my entire life. :) Hope you'll like the series!
@Ted: Yes, I agree, the prices make it reachable for anyone! That is the best part! You can take your idea and see how it does. You can create a prototype with blazing fast service for free or just a few bucks. Pretty cool!
@J.C.: Thanks! I've been silent on the site in the past few months due to the grind on Xenocell, and a surgery also kept me away for a while.
Truth be told, I was also frustrated by the many times the site went down at around 10am CET and didn't bother checking it knowing it would just be an error message. It's 10 am now - I hope I can post this.
PS: The site was down again, and had I not copied all the text, I - again - would have lost it all. Had to wait over 30 minutes for it to come back. Grr.. Not good, GG! Not good at all!
05/14/2011 (1:35 am)
Thanks guys! :)To be honest, I doubt I'll be saying new things for those of you who have been working with AWS for a while now, but I'll do my best. :) It too amazed me and I find myself thinking what else I could put up there. :)
I'll try to keep these blogs 5-7 days apart so they don't steal limelight. And to be honest, with all the grinding going on that would probably be a pace I could keep.
@Steve: I was thinking of including a pic of Catchick(tm)! But I have to work out the legal aspect of that first. :)
@Guimo: Thanks! I hope you will give AWS a try! The cool thing is I just realized that they created a free tier where anyone can use a micro instance and S3 too I believe free for a year to get a taste of things. So there's no reason not to give it a try. :)
@Matt: I plan to cover / make use of EC2, S3, CloudFront, RDS, SNS CloudWatch (and private metrics), SeS and related services such as EIP and so on.. I'll probably not go into VPC, ELB, Route 53, MapReduce, SimpleDB, BeanStalk, CloudFormation, etc.. because I haven't yet had a chance to try them. As you said, this is not just for the games I mentioned - the power to scale makes it the right choice for anything online.
@David: Thank you. :) Well, I'm a little worried about being too informative with this, since I'll be sharing information about Xenocell essentially, but I'll try to balance information and security somehow. :) Also, Xenocell is still not in an active public beta, but it's getting there fast! Hopefully by the time I get to the end of this series I'll be able to invite anyone to give it a try.
@Michael: Thank you very much. :) I hope you won't be disappointed.
@Quinton: Thanks. :) I've got so much from this site and its community that I'll be giving back my entire life. :) Hope you'll like the series!
@Ted: Yes, I agree, the prices make it reachable for anyone! That is the best part! You can take your idea and see how it does. You can create a prototype with blazing fast service for free or just a few bucks. Pretty cool!
@J.C.: Thanks! I've been silent on the site in the past few months due to the grind on Xenocell, and a surgery also kept me away for a while.
Truth be told, I was also frustrated by the many times the site went down at around 10am CET and didn't bother checking it knowing it would just be an error message. It's 10 am now - I hope I can post this.
PS: The site was down again, and had I not copied all the text, I - again - would have lost it all. Had to wait over 30 minutes for it to come back. Grr.. Not good, GG! Not good at all!
#11
05/15/2011 (4:54 pm)
Fascinating Konrad, really, really good read, and tottally got me looking forward to the next blog! 
#12
05/18/2011 (8:19 am)
This is the kind of idea I am looking for, from what I can this seems to be the choke point of most online games and hopefully these will help all of us in some way. 
Associate Steve Acaster
[YorkshireRifles.com]
I'll forgive the lack of pics ... this time ;)