Online Games Network Architecture
by Ted Southard · 09/22/2011 (11:18 am) · 15 comments
So much has changed since I last blogged about making Epic Frontiers, I decided to address something that many developers have a hard time with: Server Architectures. Now, server architecture is something that is more of an IT skill than a game design skill, but with the proliferation of social gaming and online-enabled gaming in general, it's something that designers need to be very well aware of. As a matter of discussion, I'll talk about server architecture as it relates to one of my own projects, an MMORPG demo named Epic Frontiers.
Now, Epic Frontiers was based on a T3D server which connected to a MySQL database directly, serving all of the information to the client machine through the game server. It's an old architecture that looks like so (I'm omitting Authentication and Master Servers because they are mainly involved only in the beginning of the game interaction cycle):
You can read more here, since Markuplite doesn't have the features I use on my site :P Feel free to respond here though!
Now, Epic Frontiers was based on a T3D server which connected to a MySQL database directly, serving all of the information to the client machine through the game server. It's an old architecture that looks like so (I'm omitting Authentication and Master Servers because they are mainly involved only in the beginning of the game interaction cycle):
You can read more here, since Markuplite doesn't have the features I use on my site :P Feel free to respond here though!
About the author
Started with indie games over a decade ago, and now creates tools and tech for games. Currently working as a contractor for startups and game studios.
#2
09/22/2011 (11:38 am)
Very enlightening, thanks Ted! 
#3
09/22/2011 (11:38 am)
A nice read indeed! 
#4
09/22/2011 (12:26 pm)
can some one help me plz how do i import my castle from torque constructer to torque 3D
#5
@Brian: best thing to do is post either in the Artist forums, or read the documentation that comes with the engine. This happens to be a comment area related to my blog. Thanks ;)
09/22/2011 (12:31 pm)
@All: Thanks. It's something I've been wanting to write up for a while, and I'll probably follow up with more.@Brian: best thing to do is post either in the Artist forums, or read the documentation that comes with the engine. This happens to be a comment area related to my blog. Thanks ;)
#6
09/22/2011 (12:33 pm)
Handy indeed. My online RPG aspirations have mostly been on the order of magnitude of something like Neverwinter Nights rather than massive worlds like World of Warcraft or Everquest. I would like to be able to handle up to 64 players in a persistent world and that would about be it. For me the key would be to make it as simple as possible to allow players to host their own game servers, which would be as easy as writing a simple manager app I suppose. 
#7
Constructor is not a good tool to make models for T3D. If you like a simple 3D modelling app, you can use MILKSHAPE, it have a Torque DTS exporter. You can also use other apps where generate collada. Both, collada and DTS are the standard formats for T3D. So, plz read the forums and the tutorials, it will help you a lot, and then go asking for things you dont find out in the tutorials or forum.
D'ont get this wrong!
Please read the tutorials about torque basics ! No one will help you here in blogs. And i saw your other posts, where others tell you the same, and tell you also to use the forum about such question. TY
09/22/2011 (12:45 pm)
@ Brian :Constructor is not a good tool to make models for T3D. If you like a simple 3D modelling app, you can use MILKSHAPE, it have a Torque DTS exporter. You can also use other apps where generate collada. Both, collada and DTS are the standard formats for T3D. So, plz read the forums and the tutorials, it will help you a lot, and then go asking for things you dont find out in the tutorials or forum.
D'ont get this wrong!
Please read the tutorials about torque basics ! No one will help you here in blogs. And i saw your other posts, where others tell you the same, and tell you also to use the forum about such question. TY
#8
Much thanks for taking the time to brew that one and post it.
09/22/2011 (12:52 pm)
Hey Ted, thats a really nice article.Much thanks for taking the time to brew that one and post it.
#9
It's actually not that hard to fit 64 players on a dedicated server. On an AWS server, I have thrown 50 players in a relatively tiny area for an action game I'm currently working on for a client. I trimmed out unused networking data to save on bandwidth, but even so, there was very little lag. Given a larger area with slower action and the necessary network adjustments, and it wouldn't be too hard. However, outside of that, I did find it pretty hard to do on my home server- partially because of upload speeds on my (at the time) cable connection, and also because I haven't perfected a headless client, so running 50 player clients on my dev laptop is a bit...difficult... ;)
In any event, I have kicked ideas like NWN around, especially because with the smaller world sizes, you can do so much more, and even more so if you deliver an outstanding set of campaign tools to the end-user, and simply offer a free master server.
Regarding apps... That should probably be my next blog. I've written 2-3 apps so far for different portions of the network (though one was for testing the other two, so maybe such a crappy app shouldn't count). But yeah, apps are king. You can collapse your network diagram to save money on servers (not too much, though, or else you start compromising network integrity), but those apps are still needed.
09/22/2011 (1:19 pm)
Quote:I would like to be able to handle up to 64 players in a persistent world and that would about be it.
It's actually not that hard to fit 64 players on a dedicated server. On an AWS server, I have thrown 50 players in a relatively tiny area for an action game I'm currently working on for a client. I trimmed out unused networking data to save on bandwidth, but even so, there was very little lag. Given a larger area with slower action and the necessary network adjustments, and it wouldn't be too hard. However, outside of that, I did find it pretty hard to do on my home server- partially because of upload speeds on my (at the time) cable connection, and also because I haven't perfected a headless client, so running 50 player clients on my dev laptop is a bit...difficult... ;)
In any event, I have kicked ideas like NWN around, especially because with the smaller world sizes, you can do so much more, and even more so if you deliver an outstanding set of campaign tools to the end-user, and simply offer a free master server.
Regarding apps... That should probably be my next blog. I've written 2-3 apps so far for different portions of the network (though one was for testing the other two, so maybe such a crappy app shouldn't count). But yeah, apps are king. You can collapse your network diagram to save money on servers (not too much, though, or else you start compromising network integrity), but those apps are still needed.
#11
I am using JSP/JEE for my Web Services with JSE for my normal services. Direct TCP/IP connection using the HTTP/Object on a secure port ... works pretty well for me ... just need to iron a few minor issues. :)
Your article has allowed me to see that I am on the right track with maybe a few minor tweaks (my Dialogue System was inside T3D and I am now going to move it out).
Thanks man.
09/22/2011 (11:07 pm)
Fantastic read. I am using JSP/JEE for my Web Services with JSE for my normal services. Direct TCP/IP connection using the HTTP/Object on a secure port ... works pretty well for me ... just need to iron a few minor issues. :)
Your article has allowed me to see that I am on the right track with maybe a few minor tweaks (my Dialogue System was inside T3D and I am now going to move it out).
Thanks man.
#12
That last piece took a little massaging, and if you want to wait on a web server to send you .mis content, you really need to implement timeouts for your callbacks, or you can sit there forever waiting on info that won't come.
During all that, I had started trying to build out flash libs for Interrogative (my NPC chat piece), and my procedural AI stuff, and it keeps coming back to me that maybe some of it is better left off-loaded to its own process, maybe on its own server. We did this in every single datacenter I ever worked at- servers dedicated to web, file, print, email, backup, security, etc. Why not break up the game's processes and have them scale as they're needed on the merit of how much they're being used, instead of scaling out a new server which processes chat, crafting, combat, etc, just because physics was bogging down and didn't have enough room due to sharing with everything else?
I'm just glad others are already doing this in the Indie space :)
09/23/2011 (7:45 am)
@QuintonD: I have to thank you too for your input on using JSP/JEE for some of your back-end stuff. So far, I've written up a zoning system in PHP, an AS3 Master Server, and a T3D instance manager (also AS3). I've also done a couple of authentication servers in PHP, as well as serving not just dynamic PHP pages to WebKit (HTML5 might be better for WebKit so you don't have to mess with plugins and slots/signals, though), but also serving mission file content from a web server which is assembled in T3D prior to launching the mission (all from the same web server). For the dynamic mission serving, I also whipped up a pair of (AS3 again) apps for the client to use to push content into the database.That last piece took a little massaging, and if you want to wait on a web server to send you .mis content, you really need to implement timeouts for your callbacks, or you can sit there forever waiting on info that won't come.
During all that, I had started trying to build out flash libs for Interrogative (my NPC chat piece), and my procedural AI stuff, and it keeps coming back to me that maybe some of it is better left off-loaded to its own process, maybe on its own server. We did this in every single datacenter I ever worked at- servers dedicated to web, file, print, email, backup, security, etc. Why not break up the game's processes and have them scale as they're needed on the merit of how much they're being used, instead of scaling out a new server which processes chat, crafting, combat, etc, just because physics was bogging down and didn't have enough room due to sharing with everything else?
I'm just glad others are already doing this in the Indie space :)
#13
Since they did a major change on Drupal 7, I ported my module across to Drupal 7 but wasn't happy with it. This prompted me to move to an environment where I had full control ... JEE/JSP and JSE. :)
09/23/2011 (8:13 am)
@Ted: One of my earlier authentication and news methods was served up by a Drupal 6 Module that I wrote (PHP). It also has a license key authentication routine. :)Since they did a major change on Drupal 7, I ported my module across to Drupal 7 but wasn't happy with it. This prompted me to move to an environment where I had full control ... JEE/JSP and JSE. :)
#14
09/23/2011 (8:59 am)
@Ted: Really interesting read. Thanks for sharing. 
#15
Even our fairly simple web-site now has many stages so that we can have a fluid push pipeline for updates. We have two dev environments, one QA environment, a staging environment and the live web site.
Live and stage flip-flop via the server vhosts file so that we can have an instantaneous upgrade.
Get a back-end up in running is the first challenge. Making so that you can update it is another. Making it scale is a science on it's own.
I met with several execs at Blizzard once when I was working on the IA platform. Their back-end is mind boggling.
09/23/2011 (12:13 pm)
Very important topic Ted. Thanks. I really appreciate the complexity. It gets even more complex when you get to the point of updating versions live. Even our fairly simple web-site now has many stages so that we can have a fluid push pipeline for updates. We have two dev environments, one QA environment, a staging environment and the live web site.
Live and stage flip-flop via the server vhosts file so that we can have an instantaneous upgrade.
Get a back-end up in running is the first challenge. Making so that you can update it is another. Making it scale is a science on it's own.
I met with several execs at Blizzard once when I was working on the IA platform. Their back-end is mind boggling.
Torque Owner D Fasel
Game 3D