dynamic dedicated servers
by julius · in Torque Game Engine · 10/07/2010 (3:36 am) · 11 replies
are there any working implementations for dynamic dedicated servers?
ideally...
1. client runs the game, choose a mission then run it.
2. in the backend, a dedicated server will be spawned.
3. the client will automatically join the spawned dedicated server.
4. if another client joins the same mission, it will be connected to the existing spawned dedicated server.
5. when there are no more players in the spawned dedicated server, the server will just exit by itself.
ideally...
1. client runs the game, choose a mission then run it.
2. in the backend, a dedicated server will be spawned.
3. the client will automatically join the spawned dedicated server.
4. if another client joins the same mission, it will be connected to the existing spawned dedicated server.
5. when there are no more players in the spawned dedicated server, the server will just exit by itself.
#2
Our solution can also manage many game instances over many servers. You judge how many ports you allow on one server to be opened for new servers, add those ports to a database as possible ports and the rest is taken care of by our automation.
I'm now working on taking all this a step further and have the automation start / stop Amazon EC2 instances on demand.
10/07/2010 (10:56 pm)
Yes, our game can do this. So if you were interested whether it would be possible - definitely. Of course this is not stock.Our solution can also manage many game instances over many servers. You judge how many ports you allow on one server to be opened for new servers, add those ports to a database as possible ports and the rest is taken care of by our automation.
I'm now working on taking all this a step further and have the automation start / stop Amazon EC2 instances on demand.
#3
10/08/2010 (12:03 am)
How game specific or generialized is your solution? I would be interested in hearing how this was accomplished in a generalized synopsis, as my above statements are kind of unfounded ideas really.
#4
10/08/2010 (12:13 am)
I wrote a blog about it some time ago. Granted, a lot has changed - new services were brought in, tasks were propagated elsewhere - but the concept stayed the same. 
#5
10/09/2010 (5:47 pm)
Very cool Konrad!
#6
10/09/2010 (6:53 pm)
Thanks Ryan
#7
I do have questions with your architecture.
1. How many players per TGEA game server did you able to manage to squeeze in (without any lag)?
2. If you do have player limitations with the each TGEA game server, does your Master server do the following:
a. Place the new player to another TGEA game server
b. that player that just joined in can see the players in the other game server
3. If #2.b answer is Yes, then did you just modified the Torque network code? or did you use some other concurrent programming language (Stackless Python, Twisted Python, Erlang, etc.)?
4. If #3 answer is "other programming language", then did you replace the Torque netcode with it? or did your own netcode sit on top of Torque netcode?
5. Are you doing this alone? or do you have a team? How long has it been from research to a running server architecture?
6. How many physical servers do you have? specs? bandwidth size (for TGEA game server and master server)?
7. I haven't tried Amazon EC2 but can you remotely control your servers from your home for example?
10/11/2010 (2:42 am)
Konrad, thanks for the blog. Very cool and informative.I do have questions with your architecture.
1. How many players per TGEA game server did you able to manage to squeeze in (without any lag)?
2. If you do have player limitations with the each TGEA game server, does your Master server do the following:
a. Place the new player to another TGEA game server
b. that player that just joined in can see the players in the other game server
3. If #2.b answer is Yes, then did you just modified the Torque network code? or did you use some other concurrent programming language (Stackless Python, Twisted Python, Erlang, etc.)?
4. If #3 answer is "other programming language", then did you replace the Torque netcode with it? or did your own netcode sit on top of Torque netcode?
5. Are you doing this alone? or do you have a team? How long has it been from research to a running server architecture?
6. How many physical servers do you have? specs? bandwidth size (for TGEA game server and master server)?
7. I haven't tried Amazon EC2 but can you remotely control your servers from your home for example?
#8
1. We are unsure yet. We were not able to group 128 players at the same time on the same level in our private beta yet.
2. We've moved to Torque 3D, but the basics are the same. The master server now actively works with another application called the overseer. They together work out player routing. Players are visible on a need to know basis. Our final communication solutions are decoupled from the game entirely if that is why you're asking. Other that that, the player receives information only about the players within the client scope.
3. Except disabling some features such as datablock change notification, there are no networking changes required to be able to route a player from one server to another. The languages we use for the entire game are C++, TorqueScript and PHP.
4. We have made changes and optimizations to the networking code of existing objects or created new objects similar to existing ones that are easier on the network. We are using the Torque networking layer. It's a great system, too.
5. We have an investor, two programmers and almost everyone else is contracted out for sfx, caracter dialogs, music and art. We also use some content pack art and code.
6. Since the game is not running live yet, the number of servers won't really give you any information. We don't know yet how we will be able to scale, this is why we prefer to go to Amazon instead. We now have enough servers to run web services, a dedicated database and the game instances that are able to serve our testers.
7. Yes, you can access your EC2 instances just like you would any physical server.
10/11/2010 (8:39 am)
Julius,1. We are unsure yet. We were not able to group 128 players at the same time on the same level in our private beta yet.
2. We've moved to Torque 3D, but the basics are the same. The master server now actively works with another application called the overseer. They together work out player routing. Players are visible on a need to know basis. Our final communication solutions are decoupled from the game entirely if that is why you're asking. Other that that, the player receives information only about the players within the client scope.
3. Except disabling some features such as datablock change notification, there are no networking changes required to be able to route a player from one server to another. The languages we use for the entire game are C++, TorqueScript and PHP.
4. We have made changes and optimizations to the networking code of existing objects or created new objects similar to existing ones that are easier on the network. We are using the Torque networking layer. It's a great system, too.
5. We have an investor, two programmers and almost everyone else is contracted out for sfx, caracter dialogs, music and art. We also use some content pack art and code.
6. Since the game is not running live yet, the number of servers won't really give you any information. We don't know yet how we will be able to scale, this is why we prefer to go to Amazon instead. We now have enough servers to run web services, a dedicated database and the game instances that are able to serve our testers.
7. Yes, you can access your EC2 instances just like you would any physical server.
#9
Thanks for the informative reply. I was busy with other stuff. I can come back to this later.
Anyways. May I know how much bandwidth your hosting environment provide and how much max players were you able to join in a game server?
We just did load testing but had a disappointing result.
We managed only 44 concurrent connections. The bandwidth of our hosting service is only 2Mbps and is shared with other services as well.
I am currently analyzing the PacketRateToClient and Server.
PacketRateToClient 2816 64 bytes * num players
PacketRateToServer 2816 64 bytes * num players
PacketSize 512 bytes
Players/Game Server 44
Players per Game Server 44 Players
Bandwidth per Game Server 1.89 Mbps
I'm not really sure if the calculations are correct but that pretty much what happened during our load testing. We capped out the 2Mbps bandwidth.
Am I missing something from the above calculation?
I am thinking of creating a separate thread for this.
Thanks so much.
10/20/2010 (9:57 am)
Konrad,Thanks for the informative reply. I was busy with other stuff. I can come back to this later.
Anyways. May I know how much bandwidth your hosting environment provide and how much max players were you able to join in a game server?
We just did load testing but had a disappointing result.
We managed only 44 concurrent connections. The bandwidth of our hosting service is only 2Mbps and is shared with other services as well.
I am currently analyzing the PacketRateToClient and Server.
PacketRateToClient 2816 64 bytes * num players
PacketRateToServer 2816 64 bytes * num players
PacketSize 512 bytes
Players/Game Server 44
Players per Game Server 44 Players
Bandwidth per Game Server 1.89 Mbps
I'm not really sure if the calculations are correct but that pretty much what happened during our load testing. We capped out the 2Mbps bandwidth.
Am I missing something from the above calculation?
I am thinking of creating a separate thread for this.
Thanks so much.
#10
Our beta server is on a 100Mbps line, we haven't been able to gather enough people to cap it. I think 44 players for a 2Mbps connection is great with stock Torque considering how unoptimized the network packets are here and there. Far from disappointing to me. Of course if this is your own game, then it depends on what you've changed in those packets.
Cheers,
Konrad
10/20/2010 (10:07 am)
Julius,Our beta server is on a 100Mbps line, we haven't been able to gather enough people to cap it. I think 44 players for a 2Mbps connection is great with stock Torque considering how unoptimized the network packets are here and there. Far from disappointing to me. Of course if this is your own game, then it depends on what you've changed in those packets.
Cheers,
Konrad
#11
We haven't changed anything regarding the network code and packet, except for the PacketRateToClient and PacketRateToServer which is now 64 each and PacketSize is 512.
We are considering to upgrade the bandwidth size of our hosting environment. But we really want to host at least 128 players per server. Many forums I have searched so far mentioned that they are able to host more than 128 players per game server but doesn't mention about bandwidth.
Anyways I will crunch bandwidth numbers for our boss to see. I would be interested to test your beta server if I have time. I just started another thread to discuss hard numbers on bandwidth, player counts and ghosts. Hope people with experience such as you could chip in to the discussion.
Thank you so much. Your inputs are of great help.
Regards,
Julius
10/20/2010 (10:33 am)
Konrad,We haven't changed anything regarding the network code and packet, except for the PacketRateToClient and PacketRateToServer which is now 64 each and PacketSize is 512.
We are considering to upgrade the bandwidth size of our hosting environment. But we really want to host at least 128 players per server. Many forums I have searched so far mentioned that they are able to host more than 128 players per game server but doesn't mention about bandwidth.
Anyways I will crunch bandwidth numbers for our boss to see. I would be interested to test your beta server if I have time. I just started another thread to discuss hard numbers on bandwidth, player counts and ghosts. Hope people with experience such as you could chip in to the discussion.
Thank you so much. Your inputs are of great help.
Regards,
Julius
Torque Owner Richard Preziosi
WinterLeaf Entertainment
The hard part would be having the proper setup on the master server to create a new mission directory, launch the mission, assign it a port, send the information to the client, have the client connect, and then load the mission.
There may be an easier way, however even with my limited server experience, this is the thought I have had on the subject for a while now. I would love this functionality as well, as I do not like the "if the host leaves the mission is exited" functionality of the current network code, but we have to remember that this is an FPS engine, and for the longest time that is how FPS multiplayer worked.
But maybe someone else will know of an easier solution and will chime in.