Lets talk, Thread-Per-Client for MMO's
by Dallin Wellington · in Technical Issues · 10/04/2008 (8:53 am) · 3 replies
This question has been quite a popular one. But no matter how much I search the net, i still cant find conclusive information as to if its right, or wrong.
So... lets talk.
From my experience, these would be the pro's.
Pro's:
All clients get about the same amount of Priority for packet handling. Since everyone has their own thread, and are using a blocking receive in that thread, there packet gets handled right on receive.
A 'Seemingly' synchronized world (read why i say this below).
Con's:
Extreme overhead. Once you get over 100 clients/threads running, you will have EXTREME overhead with the OS scheduling those threads. This is where the 'Seemingly' synchronized world comes in. Since everyone is getting packets handled in a equal priority manor, it seems in sync, but, can seem quite slow pace, for doing simple actions, like opening a crafting tool or bazaar terminal.
Resource Race Complexity. No doubt about it, your going to have to use locks, or some form of thread-safe messaging system. (The latter, being the fastest, but yet most complex to Assemble)
So... lets talk.
From my experience, these would be the pro's.
Pro's:
All clients get about the same amount of Priority for packet handling. Since everyone has their own thread, and are using a blocking receive in that thread, there packet gets handled right on receive.
A 'Seemingly' synchronized world (read why i say this below).
Con's:
Extreme overhead. Once you get over 100 clients/threads running, you will have EXTREME overhead with the OS scheduling those threads. This is where the 'Seemingly' synchronized world comes in. Since everyone is getting packets handled in a equal priority manor, it seems in sync, but, can seem quite slow pace, for doing simple actions, like opening a crafting tool or bazaar terminal.
Resource Race Complexity. No doubt about it, your going to have to use locks, or some form of thread-safe messaging system. (The latter, being the fastest, but yet most complex to Assemble)
About the author
#2
Thread-Safeing everything is going to be a pain in the arse, but it will pay off in the end. Im more worried about deadlock then the actual data resource racing. >_<
10/04/2008 (6:18 pm)
I guess what i will do is use a thread pool, that has X (user defined, configurable) amount of threads for processing.Thread-Safeing everything is going to be a pain in the arse, but it will pay off in the end. Im more worried about deadlock then the actual data resource racing. >_<
#3
threading is like that, its good for specific uses and really bad for other things.
multi-threading everything, especially your network code can result in a VERY complicated system
that is extremely SLOW.
if your not careful with your data structures you will wind up with locks all over the place because of simultaneous access to the same data structures which inevitably slow everything down.
you also wind up with some really nasty bugs related to thread synchronization, starvation, race conditions, memory utilization, etc.
I dont have the links handy but there are some great resources that do some serious benchmarks with networking models showing the performance of single thread, versus multi-thread polling and sending/receiving of network packets.
in the right architecture (where you not wasting resource copying the network message all over the place)
and not doing too much work on the message in the network handling the single threaded architecture
was far superior.
now what I mean by that is, instead of threading everything all over the place, you put larger portions of the functionality into a thread.
for example, have a single thread that does just the network polling/receiving/sending and nothing else, when it gets a message it hands it off to a queue that has threads that listen for messages on the queue
and an available thread pulls the message(do it without copies) and processes it.
this has remarkable throughput and far less complexity then trying to multi-thread the polling/receiving/sending. almost think of it as horizontal threading)grouping similar functionality together) instead of vertical threading (spitting up similar functionality into multiple threads).
of course there are significant number of ways to do it, but one important point is that if your going to go multi threaded, design it that way from the beginning, dont retro-fit it later on, it can be excessively complicated and nasty to do it after the fact.
so design the architecture first, then place in your implementation taking care to understand how your data structures are going to be used.
anyway thats a quick broad high level overview but its something that requires you to dig into different designs and understand the pros'/con's of them and how your requirements match up with the various solutions. youll probably wind up threading different systems different ways depending on your goals.
10/30/2008 (8:38 am)
So its like a lot of things you find a good solution for one thing and begin to think its a good solution for everything.threading is like that, its good for specific uses and really bad for other things.
multi-threading everything, especially your network code can result in a VERY complicated system
that is extremely SLOW.
if your not careful with your data structures you will wind up with locks all over the place because of simultaneous access to the same data structures which inevitably slow everything down.
you also wind up with some really nasty bugs related to thread synchronization, starvation, race conditions, memory utilization, etc.
I dont have the links handy but there are some great resources that do some serious benchmarks with networking models showing the performance of single thread, versus multi-thread polling and sending/receiving of network packets.
in the right architecture (where you not wasting resource copying the network message all over the place)
and not doing too much work on the message in the network handling the single threaded architecture
was far superior.
now what I mean by that is, instead of threading everything all over the place, you put larger portions of the functionality into a thread.
for example, have a single thread that does just the network polling/receiving/sending and nothing else, when it gets a message it hands it off to a queue that has threads that listen for messages on the queue
and an available thread pulls the message(do it without copies) and processes it.
this has remarkable throughput and far less complexity then trying to multi-thread the polling/receiving/sending. almost think of it as horizontal threading)grouping similar functionality together) instead of vertical threading (spitting up similar functionality into multiple threads).
of course there are significant number of ways to do it, but one important point is that if your going to go multi threaded, design it that way from the beginning, dont retro-fit it later on, it can be excessively complicated and nasty to do it after the fact.
so design the architecture first, then place in your implementation taking care to understand how your data structures are going to be used.
anyway thats a quick broad high level overview but its something that requires you to dig into different designs and understand the pros'/con's of them and how your requirements match up with the various solutions. youll probably wind up threading different systems different ways depending on your goals.
Associate James Ford
Sickhead Games
Definitely the ability to process clients in parallel when you have a large number of them is going to be good, but perhaps having several worker threads that get handed jobs for "process this client" would be better than having a separate thread for each?
Or possibly just identify certain parts of a client's processing that would be particularly good to thread and breaking that part out into a "job". Maybe some things just need to be done in order and are more trouble to thread than its worth, while others lend themselves well to threading?
Its kindof the same problem as trying process AI agents in parallel, usually they (and clients) have to query and modify lots of world data, which means you need to make all of that thread safe. Possibly with mutexs or making local copies of the relevant world state and passing it to each client-thread/job. But both of those have their own disadvantages.
I have no experience actually trying to do this, most likely you'd have to just try several methods and do performance tests...