Optimizing Torque for hundreds of players?
by Pauliver · in Game Design and Creative Issues · 01/26/2005 (10:14 am) · 20 replies
I'm sorry for posting this before reading everything, but there are thousands of threads here. If what i am saying is already bieng done, or if there is another resource out there for this, please leave me a link and then excuse my ignorance.
I am thinking of (and about to try to)
1) optimize the torque engine for over 100 [i'm thinking 200 is a nice number] players
2) multiple teams [read more than 2]
3) zone transfers [moveing a charactor from 1 server to another]
Some of these are going to be easy [We made zone transfers work in Tribes 2 for all of 15 min till we were like "well that was cool, but now i have to load a new map..], others are going to be harder [I'm a student who has over 6 months of C socket coding under his belt, and i have 4 tcp/ip networking books on my desk right now, messing with the network code has me alittle worried]. And as of this moment i have no idea how going from a team vs. team, to Team Vs Team Vs Team Vs ect.. is going to work at all (although i have only been looking into that for 15 min, and i do know there was a tribes 3 map with three teams.. although this did crowd your display).
I'm not one of these MMP people. I just really enjoy playing large strategy games. I understand some of the problems [i have a dl590/64 -quad 64bit itanium proc capable (currently 2 800mhz proc) -upto 64gb of ram (currenlty only 4gb of ram)] that i intend to run the server on when i have finished, and its going to be co-located to a dedicated t1.
So i want to take the torque engine and tweak it alittle more than your average joe. I'm going to try to cut down the network code (i have not yet opened up the existing network code) [you only need to know where the people you can see are, not reason to track someone who is miles away], and maximize the map size.
I really dont want a persistant world, and i'm not going to be doing anything RPG with this. But i figured w/ all the people interested in the MMORPG i could always work with you. We do have some common goals. I'm not sure what the current server sizes for mmo's are.
sorry for rambling, please tell me what you think. I'll probably be starting a project page if i get any support at all, or there is any interest.
to sum it up 3ish goals
1) larger teams (and therefor larger maps)
2) more teams (may already be in..)
3) zone transfers (already capable, but im thinking some kind of master-server implementation, so that someone knows where everyone is and you dont get people just showing up in the middle of no where.)
These things could be usefull to all you mmo programmers, I just want them so i can write a FPS game that supports 600ish people.
For those of you actually curious about what i want to do...
i want to take the expanson on torque that i hope will be created, and run it on 3 servers [with a 4th server somewhere trackign everything to make it run smooth]. I hope to set them up so that its basically a straight line, with one server between the others. Then i will mod in a fps game, so that its 1 server for each team, with battle grounds in the middle, and the ability to take the other clan's server [in an effort to bring a new aspect to competative fps games]. Then if there was support add on 2 more servers [to make hte basic layout look like a + (1 server in teh middle, an a different teams server attached to each side)] That is my end goal.
Thanks for reading. Please leave your oponion.
Paul
I am thinking of (and about to try to)
1) optimize the torque engine for over 100 [i'm thinking 200 is a nice number] players
2) multiple teams [read more than 2]
3) zone transfers [moveing a charactor from 1 server to another]
Some of these are going to be easy [We made zone transfers work in Tribes 2 for all of 15 min till we were like "well that was cool, but now i have to load a new map..], others are going to be harder [I'm a student who has over 6 months of C socket coding under his belt, and i have 4 tcp/ip networking books on my desk right now, messing with the network code has me alittle worried]. And as of this moment i have no idea how going from a team vs. team, to Team Vs Team Vs Team Vs ect.. is going to work at all (although i have only been looking into that for 15 min, and i do know there was a tribes 3 map with three teams.. although this did crowd your display).
I'm not one of these MMP people. I just really enjoy playing large strategy games. I understand some of the problems [i have a dl590/64 -quad 64bit itanium proc capable (currently 2 800mhz proc) -upto 64gb of ram (currenlty only 4gb of ram)] that i intend to run the server on when i have finished, and its going to be co-located to a dedicated t1.
So i want to take the torque engine and tweak it alittle more than your average joe. I'm going to try to cut down the network code (i have not yet opened up the existing network code) [you only need to know where the people you can see are, not reason to track someone who is miles away], and maximize the map size.
I really dont want a persistant world, and i'm not going to be doing anything RPG with this. But i figured w/ all the people interested in the MMORPG i could always work with you. We do have some common goals. I'm not sure what the current server sizes for mmo's are.
sorry for rambling, please tell me what you think. I'll probably be starting a project page if i get any support at all, or there is any interest.
to sum it up 3ish goals
1) larger teams (and therefor larger maps)
2) more teams (may already be in..)
3) zone transfers (already capable, but im thinking some kind of master-server implementation, so that someone knows where everyone is and you dont get people just showing up in the middle of no where.)
These things could be usefull to all you mmo programmers, I just want them so i can write a FPS game that supports 600ish people.
For those of you actually curious about what i want to do...
i want to take the expanson on torque that i hope will be created, and run it on 3 servers [with a 4th server somewhere trackign everything to make it run smooth]. I hope to set them up so that its basically a straight line, with one server between the others. Then i will mod in a fps game, so that its 1 server for each team, with battle grounds in the middle, and the ability to take the other clan's server [in an effort to bring a new aspect to competative fps games]. Then if there was support add on 2 more servers [to make hte basic layout look like a + (1 server in teh middle, an a different teams server attached to each side)] That is my end goal.
Thanks for reading. Please leave your oponion.
Paul
#2
Torque's networking is already highly optimized you wont need anything like a t1 even with hundreds of people. (well eventually youll need the t1)
just completed a test recently with the TGE, made some minor tweaks to tick rate and bandwidth
and was able to easy run 200-300 players on a 3ghz desktop machine (memory was even a factor at all, as a torque server in dedicated mode hardly uses anything).
could run around just fine (all 200~300 were moving). ran out of test machines to run more clients against the server, meaning it could hold more.
basically with default torque you could make a massive battlefield 1942 or any of the myriad of similar games but with a couple hundred people battling it out, easily.
good luck with whatever you do but its definately possible to do it, now you just need a kick butt game design :)
02/21/2005 (12:47 pm)
Um this is not as hard as you think.Torque's networking is already highly optimized you wont need anything like a t1 even with hundreds of people. (well eventually youll need the t1)
just completed a test recently with the TGE, made some minor tweaks to tick rate and bandwidth
and was able to easy run 200-300 players on a 3ghz desktop machine (memory was even a factor at all, as a torque server in dedicated mode hardly uses anything).
could run around just fine (all 200~300 were moving). ran out of test machines to run more clients against the server, meaning it could hold more.
basically with default torque you could make a massive battlefield 1942 or any of the myriad of similar games but with a couple hundred people battling it out, easily.
good luck with whatever you do but its definately possible to do it, now you just need a kick butt game design :)
#3
02/21/2005 (2:27 pm)
@TheMartian: I haven't peeked inside the network compartement yet... do you have any quick descriptions on those changes you made?
#4
02/21/2005 (4:30 pm)
I would also be interested in what you changed. or if you could just point me at hte correct files /classes i have no experience working with this number of files nad truth be told its a little intimidating...
#5
primary tick rate control is on gameBase.h about 1/3 the way down youll see something like
1 << 5 this is of course the default tick rate of 32ms per tick (search for tickShift to find it).
(side effect is that since the camera is networked if you slow it way down the camera will get spoongie so youll have to correct that)
in netConnection.cc is where you can alter the settings for how many messages and the size of the messages. The overriding values are set in prefs.cs (packetrate to client, packet rate to server, etc).
about 1/3 to 1/2 way down is the code for limiting the number of messages and the size of each of those messages. increasing the size of the packets will also speed up connecting to the server and starting a game (more bandwidth to send all those datablocks).
couple things of note, slowing the tick rate to get more time for processing (like ai stuff) also mucks with prediction, youll run faster and have a tendency to penetrate into objects, so it takes some more tuning to correct that, or slow down the players movement speed can also correct it as you dont move so far per tick (which is what seems to cause the problem).
I think with some decent tuning and tweaking this engine could do 300-500 players. with a bit more work and some additional threading I think it can do 500 - 800 players on a map (multi processor for this one)
this is based on the fact I can do 200~300 out of the box with only minor changes.
(ps dont forget to change your server settings, the default configuration limits players to 64
and the bandwidth is set to 32 messages and a packet size of 450 so youll need to tweak that too).
that should get you started, with a bit of work you could probably remove the whole ticking thing altogether and just let it run as fast as it can. good luck :)
02/22/2005 (10:07 am)
Sure npprimary tick rate control is on gameBase.h about 1/3 the way down youll see something like
1 << 5 this is of course the default tick rate of 32ms per tick (search for tickShift to find it).
(side effect is that since the camera is networked if you slow it way down the camera will get spoongie so youll have to correct that)
in netConnection.cc is where you can alter the settings for how many messages and the size of the messages. The overriding values are set in prefs.cs (packetrate to client, packet rate to server, etc).
about 1/3 to 1/2 way down is the code for limiting the number of messages and the size of each of those messages. increasing the size of the packets will also speed up connecting to the server and starting a game (more bandwidth to send all those datablocks).
couple things of note, slowing the tick rate to get more time for processing (like ai stuff) also mucks with prediction, youll run faster and have a tendency to penetrate into objects, so it takes some more tuning to correct that, or slow down the players movement speed can also correct it as you dont move so far per tick (which is what seems to cause the problem).
I think with some decent tuning and tweaking this engine could do 300-500 players. with a bit more work and some additional threading I think it can do 500 - 800 players on a map (multi processor for this one)
this is based on the fact I can do 200~300 out of the box with only minor changes.
(ps dont forget to change your server settings, the default configuration limits players to 64
and the bandwidth is set to 32 messages and a packet size of 450 so youll need to tweak that too).
that should get you started, with a bit of work you could probably remove the whole ticking thing altogether and just let it run as fast as it can. good luck :)
#6
02/22/2005 (10:31 am)
Thanx for those tips TheMartian, time to go play around ... :)
#7
Someone please correct me if I am wrong, but TGE is not threadsafe (even though it does give you support for threading, the codebase itself is not configured to use it), and runs in one process, so multi-cpu won't actually increase your performance capabilities. At least in my experience in the past, the smallest unit of processing that can be handed to a different cpu is a process, so the extra cpu's would be unused with stock TGE.
It is of course very possible that my knowledge of multi-cpu scheduling is outdated, and current kernels do allow for shopping off smaller units (below the process level) to additional cpus...so if someone is more current on the technology, please jump in!
02/22/2005 (10:33 am)
Quote:I think with some decent tuning and tweaking this engine could do 300-500 players. with a bit more work and some additional threading I think it can do 500 - 800 players on a map (multi processor for this one)
Someone please correct me if I am wrong, but TGE is not threadsafe (even though it does give you support for threading, the codebase itself is not configured to use it), and runs in one process, so multi-cpu won't actually increase your performance capabilities. At least in my experience in the past, the smallest unit of processing that can be handed to a different cpu is a process, so the extra cpu's would be unused with stock TGE.
It is of course very possible that my knowledge of multi-cpu scheduling is outdated, and current kernels do allow for shopping off smaller units (below the process level) to additional cpus...so if someone is more current on the technology, please jump in!
#8
But NT derived kernels (NT4, Win2k, XP) actually do schedule by thread, so a multiprocessor machine could run two threads at once.
Edit: Oh, and you're right. Torque isn't threadsafe.
02/22/2005 (11:13 am)
Linux/Unixes tend to schedule by process, so you're right there.But NT derived kernels (NT4, Win2k, XP) actually do schedule by thread, so a multiprocessor machine could run two threads at once.
Edit: Oh, and you're right. Torque isn't threadsafe.
#9
02/22/2005 (11:18 am)
At least Torque runs on multiprocessor machines. Halo for the Mac slows to about .3 frames per second on dual processor Macs. Disable a processor and it runs wonderfully.
#10
02/22/2005 (7:03 pm)
Thanks i found the places that these changes were to be made. Now for tweaking. (*on a side note how do you go about getting 300 people to help you test this?*)
#11
But yes you would need to make the pieces your going to break out thread-safe, then of course minimize the shared data between those threads.
for the networking part it isnt that bad actually, but for the rest of the code it will take a fair amount of work.
sorry I didnt explicitly state that, i was implying when I said "multi-processor" that you would have had to make it multi-threaded and thread-safe (at least the pieces dealing with shared data). of course stock TGE would not benefit from a multi processor machine (thought that was obvious lol )
anyway, getting enough test clients was also our problem, I wound up making stripped down tge clients (no sound, no rendering) so that I could run 30 to 50 of them on one box.
02/23/2005 (1:53 pm)
Your right torque is not thread safe.But yes you would need to make the pieces your going to break out thread-safe, then of course minimize the shared data between those threads.
for the networking part it isnt that bad actually, but for the rest of the code it will take a fair amount of work.
sorry I didnt explicitly state that, i was implying when I said "multi-processor" that you would have had to make it multi-threaded and thread-safe (at least the pieces dealing with shared data). of course stock TGE would not benefit from a multi processor machine (thought that was obvious lol )
anyway, getting enough test clients was also our problem, I wound up making stripped down tge clients (no sound, no rendering) so that I could run 30 to 50 of them on one box.
#12
If you dont mind me asking, what did you thread?
From my understanding of threads.. They share global variables, and thats it. So what can you use a thread on a server for? I mean if you spool a seperate thread for every tcp client.. then great you have threads [they could maintain references to every socket connectd before them but they would have no way to communicate w/ sockets connected after they are created] that do little..
I guess im confused as to what you would use a thread for [i bet im missing something obvious].
02/23/2005 (7:02 pm)
I'm going to be interning at a data center this summer. during my spring break I will be there [march 21st - 26th]. While there i would like to conduct a test of the actual number of clients we can do. I'll advertise closer to then and try and draft as many people as i can.If you dont mind me asking, what did you thread?
From my understanding of threads.. They share global variables, and thats it. So what can you use a thread on a server for? I mean if you spool a seperate thread for every tcp client.. then great you have threads [they could maintain references to every socket connectd before them but they would have no way to communicate w/ sockets connected after they are created] that do little..
I guess im confused as to what you would use a thread for [i bet im missing something obvious].
#13
If you dont mind me asking, what did you thread?
From my understanding of threads.. They share global variables, and thats it. So what can you use a thread on a server for? I mean if you spool a seperate thread for every tcp client.. then great you have threads [they could maintain references to every socket connectd before them but they would have no way to communicate w/ sockets connected after they are created] that do little..
I guess im confused as to what you would use a thread for [i bet im missing something obvious].
02/23/2005 (8:30 pm)
I'm going to be interning at a data center this summer. during my spring break I will be there [march 21st - 26th]. While there i would like to conduct a test of the actual number of clients we can do. I'll advertise closer to then and try and draft as many people as i can.If you dont mind me asking, what did you thread?
From my understanding of threads.. They share global variables, and thats it. So what can you use a thread on a server for? I mean if you spool a seperate thread for every tcp client.. then great you have threads [they could maintain references to every socket connectd before them but they would have no way to communicate w/ sockets connected after they are created] that do little..
I guess im confused as to what you would use a thread for [i bet im missing something obvious].
#14
02/23/2005 (11:35 pm)
Great, those numbers will come in handy =) Atleast as some sort of ballpark figure.
#15
For torque you spend quite a bit of cpu in networking(compression, replication, etc)
you can also easily spend a lot of cycles in ai.
so to improve scalability and give more cycles to other important things like game simulation, collision, physics you break out those expensive pieces into there own threads so that they can be managed on seperate cpus. (although obvious ill mention that you only gain from this if you run on a multiple processor machine).
so now for example, one processor could be handling networking/replication, one processor handling AI and other processors handling collision/physics/game simulation (can divide this up as well).
so dont think so low level with threads (like a thread per socket or that type of thing) look at it from a higher level perspective, where do you spend your time and put that part into its own thread and onto its own processor.
a side comment, this type of architecture is much more suited for windows servers, as windows threads are relatively light weight compared to processes. For a linux environment you can get gains by going the multi-processes route instead of multi-threading, as processes in linux are resoanbly light and you get improved resources management with seperate processes then seperate threads (will still work with threads in linux but you have other options).
anyway, hopefully thats clear enough to get you started in the right direction :)
02/24/2005 (12:13 pm)
For threading something like a game engine such as torque you first need to profile it to see, during the course of a tick, where are you spending most of your time.For torque you spend quite a bit of cpu in networking(compression, replication, etc)
you can also easily spend a lot of cycles in ai.
so to improve scalability and give more cycles to other important things like game simulation, collision, physics you break out those expensive pieces into there own threads so that they can be managed on seperate cpus. (although obvious ill mention that you only gain from this if you run on a multiple processor machine).
so now for example, one processor could be handling networking/replication, one processor handling AI and other processors handling collision/physics/game simulation (can divide this up as well).
so dont think so low level with threads (like a thread per socket or that type of thing) look at it from a higher level perspective, where do you spend your time and put that part into its own thread and onto its own processor.
a side comment, this type of architecture is much more suited for windows servers, as windows threads are relatively light weight compared to processes. For a linux environment you can get gains by going the multi-processes route instead of multi-threading, as processes in linux are resoanbly light and you get improved resources management with seperate processes then seperate threads (will still work with threads in linux but you have other options).
anyway, hopefully thats clear enough to get you started in the right direction :)
#16
02/25/2005 (8:09 am)
I was thinking thousdands if you just zone it.
#17
~Paul
02/25/2005 (10:36 am)
Um. thats no good if each zone only hods 5 people though Eradvongizmo. Were talking about maxing each servers size. how you link server is not what were discussing here. What were discussing is how many people we can get a single EXE to hold~Paul
#18
Windows Threads
and
Windows Events
I have learned alot about Posix threads, does the same apply to windows?
thanks for your help.
03/08/2005 (9:44 am)
I have been looking into this and i have gotten pretty far, but what i really need is a good explination of Windows Threads
and
Windows Events
I have learned alot about Posix threads, does the same apply to windows?
thanks for your help.
#19
The windows posix implementation is fairly complete (only a few eccentric things you cant do).
check out posix threads for windows here
http://sources.redhat.com/pthreads-win32/
03/09/2005 (7:34 am)
No, windows uses its own crazy'ness to implement threads, however, you can download a posix wrapper for windows so that you can write to posix and it will work on windows and linux (for example).The windows posix implementation is fairly complete (only a few eccentric things you cant do).
check out posix threads for windows here
http://sources.redhat.com/pthreads-win32/
#20
I made some changes i found around
[mainly some torque script changes, and in gameConnection.h]
Now the master server is reporting back a max number of players of 244 [it also changes my version number every time i refresh].
Is this a problem other people have had? Does anyone know if this is a real limit, or some sort of wierd math error on the part of the master server. [If it was a multiple of 2 i would know it as a master server error but its this weird number.]
03/09/2005 (12:19 pm)
Thanks Martian.I made some changes i found around
[mainly some torque script changes, and in gameConnection.h]
Now the master server is reporting back a max number of players of 244 [it also changes my version number every time i refresh].
Is this a problem other people have had? Does anyone know if this is a real limit, or some sort of wierd math error on the part of the master server. [If it was a multiple of 2 i would know it as a master server error but its this weird number.]
Torque Owner Anders Jacobsen
Torque should be able to handle hundreds of players out of the box with processor, memory and bandwidth being the limiters.. My guess is you're planning on running more than one server process to handle the players? Or you're going to have problems utilizing more than one processor I hear. Something about Torque not being multithreaded...