Game Development Community

dev|Pro Game Development Curriculum

Xenocell: Server magic

by Konrad Kiss · 03/13/2009 (8:00 am) · 25 comments

I am creating a game that is a lot like MMOs out there. It runs several missions at the same time on many dedicated servers, and players are transmitted between these servers as needed.

The game revolves around players in teams fighting for resources on and beneath the face an alien planet. The main game field is a huge valley full of abandoned portals that lead to rare resources on the planet. The more portals a team controls, the more resources a team gets each hour. Read more on the website, or my previous blogs here and here.

I did not mean to dive into the game much, but this was needed for you to see why I designed the server architecture the way I did.

I refer to dedicated game servers as instances - a term probably familiar from World of Warcraft to many of you. Each of these instances run one game zone (or mission). My game distinguishes between three types of zones:

- persistent, singleton zones started at world start (such as the main game world)
- non-persistent, instanceable zones started on demand (such as cities in the world)
- non-persistent, singleton zones started on demand (such as resource sites)

One game world is basically a set of many servers. Just as if they would be different games, several worlds can be replicated by creating the same architecture. You can view a mockup of the architecture here:


www.xenocell.com/dev/xcservarch.jpg

The road that lead to this structure was pretty rough. I had my list of things I needed to get done, but I was not sure about how I'd go about doing that.

I've spent numerous nights to prepare an architecture for the servers that fulfills the following requirements:

It is self-balancing

New non-persistent server instances are started and stopped within seconds on demand. If a city is too crowded, another instance is started and the player is given the opportunity to switch over to another instance of the city. Also, if an instance times out because of a crash or hardware failure, new instances must be started in the place of existing ones while players are transmitted to other instances of the same zone.

It uses the database efficiently and not unnecessarily

Data caching is a very important aspect of my solution. Most of the data the game requires can be cached - item stats, skill and quest descriptions are good examples. These data should be cached once and then provided to the game servers without having to access the database. When any such data changes, the cache would be automatically cleared, so a new one can be created at the next time the data is queried for.

The server cluster can be easily expanded

When a new server is prepared to be added to the cluster, registering it as a resource is just adding it to the list of servers in the database. Once there, the master server will take care of managing instances running on the new hardware.

This is very important in preparing for a sudden rush of players that would otherwise make the game kneel down and crash in a matter of minutes.

Available server resources can be measured

It is possible to define ports where instances can be started on each hardware. The master server will take care of putting the available ports into a pool where the server with the least relative load will get to start the next instance when needed.

So by counting the number of overall and available ports and players on each server, the master server can guess the load percentage of each game server, and act accordingly.

Data retrieval runs as a separate thread

The game server must never wait for data. Whenever data becomes available to the game server, the data is processed. This way the database will not become a bottleneck when many queries, or even unoptimized queries are run. This gives me just a little extra to be able to further optimize the database without having effects that affect players too much and would stress the hell out of me.

...

Most of this architecture is already functional. I am now finishing it all up with the XML data gateway / relay and caching facility. My investor is preparing our servers as I write this, and I'm hoping some of you could help me with alpha testing starting next month. If you are interested, sign up on the Xenocell website.

I will be sure to take only small steps. Starting with only a few features, and adding more one by one.

As closing words, to anyone brave enough to plan an MMO and not be let down by the anti-MMO craze, here are a few tips:

1. Until about two weeks ago, I've worked on the game without having a dedicated server. If you are making a multiplayer game, make sure you start by separating client and server, otherwise you will have many problems later on. It is important to keep in mind that local client to a non-dedicated server will be able to access many server variables without having to scope them to the client. That causes a huge number of headaches if you don't pay attention.

2. Another tip is that when creating any object that you need each client to have one of, never name them upon creation.

Let's assume you name a clients camera Cam when the client joins the game. When the next client joins the game, his/her camera will also be called Cam on the server. This causes first client's cam to be invalid, and the client will get a nice pink screen. (This has happened to me. :)

If you are interested in more information about certain parts of the Xenocell architecture in detail, let me know. And as always, constructive criticism is more than welcome.

Thank you for reading through.
--Konrad

@konradkiss
KonradKiss @ GitHub
konradkiss.com
Page «Previous 1 2
#1
03/13/2009 (8:26 am)
@Konrad: That was an amazingly informative blog, and a very good point about the data caching that you're using (as well as the named client-side objects, which I had not yet thought about, so that's noted so I won't get pink-screens during testing, lol).
#2
03/13/2009 (9:44 am)
Really cool Konrad
#3
03/13/2009 (12:26 pm)
Thanks Ted and Surge! A month from now, I can tell if this is working in a live environment as I think it should.
#4
03/13/2009 (1:54 pm)
@"If you are interested in more information about certain parts of the Xenocell architecture in detail, let me know."

Yea, i want all!, waht is the solution you will use for chat, ingame browsing, etc...?, community?
#5
03/13/2009 (2:38 pm)
I notice, another free open source app for backend work. MOM use python (TGEpye), so does EVE online. You're using Curl-That's a new one for me, nice website and info.

Lightwave Dave got a article on using TNL for communicating between instances of TGE(A) Galactic

Nice behind the tech info :)


#6
03/13/2009 (3:58 pm)
@Javier: I am planning communications to rely on existing services that were created for real life communications.

Chat is going to be connected to an irc server where basically channels will be rooms. This will be a good background for future expansion as well. I can also have this placed outside of the server farm. I have not yet worked out the details though, so it's still all blurry. This is definitely not going to be there for the alpha - until its done, testers will use the built in solution. (Heh, even that's not switched on right now.)

Community is the key, isn't it. :) I am planning to create a firm and stable community through gameplay elements primarily. The game will reward both teams accepting a new player, and new players who do not want to join a team. Accepting new players will widen the range in which a team can conquer resource areas. Different resources are available at different levels, so low level players will also have their uses in a team, while high level players can be spared for the higher level resource sites.

Players who chose to play solo will be the only ones able to gain access cards. These cards are required when one team attacks another by raiding the enemy team's base.

Eventually, I am planning to let players create their profiles over the www site that would appear ingame when allowed. Same goes for team information. Profile images, team decals, etc will also be implemented, but I find these to have a lesser effect on raising a good community around the game, so this might not make it till launch.

I am hoping that these create quick connections which I am planning to back up using the aforementioned irc based chat, simple ingame messaging, automatic game to RL email notifications (ie. your base is being attacked), and optionally voice chat - although that's not high on the priority list right now. That is something that players can decide about for themselves.

@Johnny: Truth be told, my python is not as good as my php. Regardless, I don't think it makes a difference much.

cURL is being used by the game server instances to fetch data. I found it to be a better, faster and more scalable solution than HTTPObject or a direct db connection.

TNL for communicating between server instances is an interesting concept. My game will not need much communication between instances - the little player routing that's needed will be handled by the master server. That communication is also done through cURL and an only internally open web server on the master server side.

Thanks for the comments. The website is becoming updated soon, since the current design does not really fit the latest design of the game's ui, so that's in for an overhaul. Also, I really need to write blogs about the game more often - both here and there.
#7
03/13/2009 (4:23 pm)
Nod, MoM uses Python and in particular the twisted library.

http://twistedmatrix.com/projects/core/documentation/howto/pb-copyable.html

Here is a bit of a short howto on how some of it works. Most of the communication via python is done this way and the information is cached server side and only updates are sent to the client (or in some cases server to server).
#8
03/14/2009 (2:21 am)
@Joshua:

That looks really interesting, and it just gave me an idea.

In theory, it would be possible this way to make one game be run by multiple servers sharing the handling of objects amongst each other. (say based on object id) That'd be a pretty interesting project to take on.

I have just skimmed through the page you gave me, and it gave me the impression that the same could be done through serializing data on one end and unserializing on the other. I'm not sure though that I see all the pros of this method.
#9
03/14/2009 (1:10 pm)
For IRC, if your not using nickserv/chanserv yet, Anope has treated us very well in the past.
It uses MySQL as the back-end and it sounds like you have that part pretty much handled so having the game servers control your IRC experience is pretty easy.
Back when we were using the Mozilla integration resource (TGE 1.5.2) we were amazed at how well it worked.
We have an SMF portal (again MySQL back-end) that was tailored to the game (look up other players, auction in-game items, hack other players, etc.) and the Mozilla resource allowed the players access the portal from in-game.
All this stuff is pretty solid, scalable and AFAIK is cross-platform.
Hope it helps or at least gets some ideas going. :)
We purchased 1800 slots at Green-Ear, but I'm not sure if I'm willing to give up the power of Anope/MySql controlled IRC chat yet.
Also, Ruin was seriously dependent on the Event Driven Database resource.
Tony Richards has been working on porting it for us (and so have I, but with little luck) so I can get Ruin back on track with persistence.
But, I'm starting to get a bit nervous, most of our non-persistent features are pretty much complete.
I've been starting to look at XML also as a way to communicate to MySQL without tying up the server's processes.
Glad it's working out for ya.
I may be following the same path at some point.
Am I correct that you are using authentication on the master server?
Keep it up man, holler if you need us, we owe you a few favors. ;)
#10
03/14/2009 (1:23 pm)
Oh, one more thing, chanserv/nickserv is a REALLY REALLY REALLY good idea to prevent floods, bots, etc.
I'm an ex-DALNet scr1ptk1d and I hope that's a hint at why you should focus a bit of time on locking down your IRC.
Them k1dz have no respect or restraint sometimes. ;)
#11
03/14/2009 (3:09 pm)
:) Thanks Ari, I really appreciate all the info and help!

I will definitely ask for your advice with IRC. All I can show up in that dept is that I'm seriously security paranoid. Yet, without thorough knowledge of IRC, that doesn't worth much.

The mysql backend sounds great for communication, I'll definitely keep Anope in mind. I want to lock down everything that comm within the game doesn't need from an IRC server. Right now, I am not planning to have it open to the rest of the world, only the game servers, so it can probably be made pretty tight.

Funny you mentioned that Event Driven Database resource. Not being able to port it was my reason to go the XML way.. right now it seems to me that it's a pretty good choice, and right now, I would want to go that way even if someone had some luck porting that resource, because it's a solid platform for my data caching plans.

Embedding Mozilla is a really great way for those data to be presented to the players. I have tried to save as much time planning out the GUI as I could. This idea is a great addition to that. I enjoy working with XHTML and JavaScript a lot more than Torque's GUI, I will just have to figure out how to intercept JavaScript calls. Thanks for the idea anyway! Unfortunately I couldn't find the resource you were referring to. Can you give me a link maybe?

I've had my share of adventurous kids when I ran a browser game around 2001-2004. I know very well what you're talking about! :) Still, I am not prepared enough just yet to plan my IRC based communication, so I'll read up on it and let you know what I come up with if that's ok with you.

Thanks again!
#12
03/14/2009 (3:16 pm)
Oh, and I forgot to answer one question about authentication.

While there's no authentication in effect just yet (its easier to work on it this way right now), it's there.

I'm using a token based system with MD5 hashing and a double challenge (with one at encryption time and another random one at authentication time). Also, access to any server within the cluster needs a key that serves as a cluster secret - dynamically created and valid for only a very short amount of time. I hope this will do.
#13
03/14/2009 (3:34 pm)
Ah nevermind, I found the mozilla resource! Thanks, Javier!

I wonder if embedding flash and streaming in movies through this would work...
#14
03/14/2009 (5:00 pm)
I haven't actually hooked up a packet scanner to see if it actually serializes the data before it sends it over (rather the fact if it is even encrypting/compressing the data).

The issue with MoM is they never released any real metrics on how well things actually worked. I believe they had/have 4 servers total and I heard of upwards of 100 or so in a single zone (they had multiple zones and worlds), but no mention of CPU/disk/Bandwidth metrics to give one a good idea on what to expect. The reality of things now a days server hardware is cheap compared to 2-3 years ago and that never hurts. You just have to have somewhere to host it.

Also, don't use MD5, use SHA1. It is much more secure and available in most libraries along with MD5.

--Josh
#15
03/15/2009 (12:55 am)
Point taken about MD5. I will see how big a trouble creating SHA in C++ and PHP would be, but you're right about it being more secure. Thanks for the suggestion.
#16
03/15/2009 (1:01 pm)
The Mozilla resource won't do flash.
Such a pity.
There is a resource to embed Flash into Torque but I blew it off without even looking at it, assuming it wasn't as powerful as I needed.
I could have been dead wrong, I still don't know.
SHA1 is the way to go if you can pull it off.
SMF (simple machines forums) uses SHA1 for user challenge and I went down that road but never completed it, instead I had the user authenticate via the browser (redundant since they are already playing) but maybe it adds flavor to gameplay? lol.
And before I forget there is another resource can't remember the name off hand, but it opens a web port on Torque so that you can send the game server commands via web pages.
This opens up HUGE possibilities!
As far as IRC, we used Inspircd + Anope + MySQL, but there are other flavors.
An interesting part of IRC (started with the TGE Lobby resource) is that the game server can also talk to the IRC server.
So giving the zone server an account and setting the info in prefs allows the server to kick, ban, silence, create channels, etc.
Basically the game servers run as IRCOPs.
Also... there are TONS of IRC bots out there.
In a game like yours or mine, this could seriously add to gameplay.
Think HAL from 2001, "Daisy, daisy...." or a bot chalk full of information like in MoM that answers player questions.
Bots can be in multiple channels at a time, so a bot could be in EVERY channel if needed.
nickserv/chanserv can also be set to verify that the IP adress the chat is coming from is the IP the player is on, that should help keep out the riff-raff, floodbots, etc. to a degree.
Definitely gimme a jingle when you're hardening IRC.
And feel free to contact me or Jondo via IM (I know how much you love IM, lol).
#17
03/15/2009 (2:02 pm)
Our bot Sentinel was sooper cool. I could log in to dev mode Ruin locally and he'd be on there. I could tell him to give me OPs in #global, find out the last time Ari was logged in, etc. Also, our users who were on the webpage could talk in #global via a web-based chat, and we had a similar java chat integrated into our svn-based launcher/patcher (worth a blog of its own...Ari!). Its pretty cool when its all set up right.
#18
03/15/2009 (4:01 pm)
Hey guys, that sounds really really cool! I look forward to wasting a whole week on this! :) I really do. I'm not too proficient at this, so it's definitely gonna be a very interesting journey!

Quote:
Think HAL from 2001, "Daisy, daisy...." or a bot chalk full of information like in MoM that answers player questions.

That faq bot.. it's got my mind going... really sweet stuff!

Quote:
And feel free to contact me or Jondo via IM (I know how much you love IM, lol).

Lol. Yep, that's my cross to bare. ;) Thanks a lot! I will get my act together. :)
#19
03/15/2009 (7:13 pm)
Found one of the links I was trying to remember for you:
Web Chat for TGE, Communicate from the Outside
#20
03/24/2009 (12:51 pm)
XML RPC, or REST API over HTTP are very good ways to go.

When I wrote the Event Driven Database resource, I really wanted to go that route, but I didn't really have time to do things "right". Keeping things as services and out of the game client and server definitely limits your impact when things change.

This approach is more of a service oriented architecture using web services.
Page «Previous 1 2