MMO Infrastructure Revisited
by Nate "Nateholio" Watson · in General Discussion · 12/17/2004 (9:56 pm) · 7 replies
Its my understanding that the main thing hindering MMOs using Torque is that the engine isnt designed to handle such a thing. Is this true?
I was wondering because I just discovered the new (new to me at least) source code documentation and started sniffing around in the netcode docs. What I want to do is try to kill several birds with one stone, that is, learn C++ while trying to make a netcode base for Torque that makes it easy for those who wish to have a shot at an MMO to do so, and EARN a little respect in the GG community. Before I continue, let me say that I already know about GORPE, the expense of the hardware needed to support a MMO, and the number of MMOs on GG that never made it past the planning stage, etc. So, what I have planned is the following:
My post is at the bottom
Coming from a hardware background this seems like the most logical solution to the networking infrastructure problem. Multiple servers with each dedicated to a specific area(s) or task(s). I want to know if anyone is interested in trying to set up this type of system. Would anyone be willing to test the system on their comps if I can get something like this working, even if only simple at first? Or am I just chasing my tail here with thoughts of giving this a try?
I was wondering because I just discovered the new (new to me at least) source code documentation and started sniffing around in the netcode docs. What I want to do is try to kill several birds with one stone, that is, learn C++ while trying to make a netcode base for Torque that makes it easy for those who wish to have a shot at an MMO to do so, and EARN a little respect in the GG community. Before I continue, let me say that I already know about GORPE, the expense of the hardware needed to support a MMO, and the number of MMOs on GG that never made it past the planning stage, etc. So, what I have planned is the following:
My post is at the bottom
Coming from a hardware background this seems like the most logical solution to the networking infrastructure problem. Multiple servers with each dedicated to a specific area(s) or task(s). I want to know if anyone is interested in trying to set up this type of system. Would anyone be willing to test the system on their comps if I can get something like this working, even if only simple at first? Or am I just chasing my tail here with thoughts of giving this a try?
#2
"Critical Data is data that is of the utmost importance to the MMO infrastructure. When a packet of critical data is lost it MUST be resent. Most of the data in this category deals with synchronizing servers, updating world object data blocks, load balancing, and server outage/backup handling. Examples of critical data are:
Server synchronization (load balancing, load forecasting)
Interserver configuration messages (timehacks)
World geometry updates (weather, terrain deformation, etc.)
Server status requests/replies
Primary Data is data such as player, NPC, and object positional information and status. Examples of primary data are:
Player position, facing, and status
NPC position, facing, and status
Object position, facing, and status (objects are pretty much everything that moves or interacts with the envoirnment/player)"
Im afraid that for the next month or more I'll be limited to writing technical things about how things are organized and such as I try to learn C++. So dont cross your fingers for anything like this soon.
12/18/2004 (2:17 am)
I've already started to list the requirements for data priority, and load balancing is one of the things that takes priority."Critical Data is data that is of the utmost importance to the MMO infrastructure. When a packet of critical data is lost it MUST be resent. Most of the data in this category deals with synchronizing servers, updating world object data blocks, load balancing, and server outage/backup handling. Examples of critical data are:
Server synchronization (load balancing, load forecasting)
Interserver configuration messages (timehacks)
World geometry updates (weather, terrain deformation, etc.)
Server status requests/replies
Primary Data is data such as player, NPC, and object positional information and status. Examples of primary data are:
Player position, facing, and status
NPC position, facing, and status
Object position, facing, and status (objects are pretty much everything that moves or interacts with the envoirnment/player)"
Im afraid that for the next month or more I'll be limited to writing technical things about how things are organized and such as I try to learn C++. So dont cross your fingers for anything like this soon.
#4
It's extremely efficient at doing so.
As has been said here and in a lot of other places, and as you seem to be aware of, it's the backend and server support that can be too much too handle for an indie, on an "real"indie budget :)
I strongly recommend you approach the problem from another angle : get a good book on network programming (Andrew Tannenbaum's book is still a staple of universities all over the world) and another one on the whole TCP/IP stack, explore UDP, etc. Ditto with database programming, etc.
Know the tools before tackling the masterpiece :)
Understand all of that, and then you'll be in a much better position to work on the higher level stuff you're presently pursuing : otherwise, you incur the risk of having to redo a lot of it, because of misplaced assumptions.
You'll also be in a better position to realistically lead a team of programmers and designers that way, as well as make a better choice at what kind of MMO you want to do, massively multiplayer wise : how many players online at a time, and how many accounts are two distinct problems, even though closely tied, since one is a server load problem, the other a database management and admnistration problem.
Seems to me you're putting the cart before the horse by doing some quite detailed high level technical requirements first, before even knowing the tech and how you're going to use it.
I'm not telling you to give up : I'm trying to point out some shortcomings in your approach to a very, very intricate and massive (hehe) problem that has solutions that vary according to what you want to do.
Without a better understanding of networking and database backends, I don't see how you can do anything but compiling a list of buzzwords and features.
Do your homework :)
Have fun
12/18/2004 (7:56 am)
Erh, TGE doesn't have a problem (or TNL/OpenTNL) with sending the data.It's extremely efficient at doing so.
As has been said here and in a lot of other places, and as you seem to be aware of, it's the backend and server support that can be too much too handle for an indie, on an "real"indie budget :)
I strongly recommend you approach the problem from another angle : get a good book on network programming (Andrew Tannenbaum's book is still a staple of universities all over the world) and another one on the whole TCP/IP stack, explore UDP, etc. Ditto with database programming, etc.
Know the tools before tackling the masterpiece :)
Understand all of that, and then you'll be in a much better position to work on the higher level stuff you're presently pursuing : otherwise, you incur the risk of having to redo a lot of it, because of misplaced assumptions.
You'll also be in a better position to realistically lead a team of programmers and designers that way, as well as make a better choice at what kind of MMO you want to do, massively multiplayer wise : how many players online at a time, and how many accounts are two distinct problems, even though closely tied, since one is a server load problem, the other a database management and admnistration problem.
Seems to me you're putting the cart before the horse by doing some quite detailed high level technical requirements first, before even knowing the tech and how you're going to use it.
I'm not telling you to give up : I'm trying to point out some shortcomings in your approach to a very, very intricate and massive (hehe) problem that has solutions that vary according to what you want to do.
Without a better understanding of networking and database backends, I don't see how you can do anything but compiling a list of buzzwords and features.
Do your homework :)
Have fun
#5
There's a lot of high hopes in there, and in order to get it done as an indie you'll have to scale it back. Reason I say this is because as someone who made an extremely similar technical doc in 2001 and has since been working on a persistent world, I can say from hindsight and knowledge and experience what it takes to even start developing one. As to what it takes to launch and maintain one, you'll have to ask me in another year ;)
The database stuff is way off, only because you need to figure out what data you want to put into it and how you want to organize it rather than how many objects you want to stuff into it. IOW, that one id field won't do. I have like 20 tables in my test DB, and it's not getting any smaller...
Splitting up servers like you're looking to do to make huge AI applets to control weather, nature, military stuff... Script it all as events, and save yourself the time and money. On the client-side, it'll look the same, and they won't miss it. This design takes a complicated project and adds a few layers of extra complexity to it, so what you want to do is read every book and article you can about MMO's- especially the ones that are cautionary tales and talk about all the bad things, because that's where you learn how to do things better than the others.
Read up on the TGE networking code, it answers your thinking on critical and prioritized packets even more detailed than you wrote, and it's already in the engine and TNL.
Read, Read, Read... Knowledge is power :)
12/18/2004 (4:47 pm)
Ack, the forums swallowed a very long-winded post where I discussed several of the shortcomings of your design, and since I have a date with St. Paulie's Girl in a half hour, here's the 2-minute condensed version:There's a lot of high hopes in there, and in order to get it done as an indie you'll have to scale it back. Reason I say this is because as someone who made an extremely similar technical doc in 2001 and has since been working on a persistent world, I can say from hindsight and knowledge and experience what it takes to even start developing one. As to what it takes to launch and maintain one, you'll have to ask me in another year ;)
The database stuff is way off, only because you need to figure out what data you want to put into it and how you want to organize it rather than how many objects you want to stuff into it. IOW, that one id field won't do. I have like 20 tables in my test DB, and it's not getting any smaller...
Splitting up servers like you're looking to do to make huge AI applets to control weather, nature, military stuff... Script it all as events, and save yourself the time and money. On the client-side, it'll look the same, and they won't miss it. This design takes a complicated project and adds a few layers of extra complexity to it, so what you want to do is read every book and article you can about MMO's- especially the ones that are cautionary tales and talk about all the bad things, because that's where you learn how to do things better than the others.
Read up on the TGE networking code, it answers your thinking on critical and prioritized packets even more detailed than you wrote, and it's already in the engine and TNL.
Read, Read, Read... Knowledge is power :)
#6
I already know what type of MMO I'd want to do, but its not just about what I would like to do its about providing a 'customizable' MMO support code. I also understand that you cant just up and write netcode that supports any type of MMO. Datbase management is relatively easy for me, the harder part would be server coordination.
Well, this is a rare instance where I actually write ideas down before I begin work on something. I usually just go off whats in my head and it hasnt failed so far (check profile for list of experience). But to keep in line with the community and to document things for future users (if I can get something put together) I decided to actually write things down.
The 32 bit ID field is only to identify a particular object to the system. It doesnt do anything more than that. This allows administrators and servers to track objects. Sure 32 bits may be low, but it seems like enough for objects the player can interact with (then again there was a time when "nobody will ever write a program that needs more than 2k of RAM"). As to the rest of your post about DBs I'm not really seeing your problem. Here is a ver simple example of how things could be organized, its an old abandoned idea, but I did have a program working using this system that efficiently managed several hundred megs of crap. Just a fraction of what a MMO would take, but a start.
Lastly, i agree with you about AI, the server codes would be simple ones, not complex supercomputer algorithms. The idea isnt to be complex here but to make a believeable approximation of a persistant universe. Basically the programs or scripts would take several inputs and produce an output according to some simple rules. Like if theres clouds over an area and the temperature and air pressure are right you get rain. Things could be "firm-wired" to occur in some places at a higher frequency and/or seasonal basis, with some randomness added for fun. Some things however need to be programmed well, especially planetary movements. If one is traveling between planets it would suck to have nothing where a planet was supposed to be and little fuel left to find it.
But anyways.....
12/21/2004 (12:19 pm)
@Nicolas & Ted: Im not trying to argue with either you, just relaying some thoughts about your posts....so here it goesI already know what type of MMO I'd want to do, but its not just about what I would like to do its about providing a 'customizable' MMO support code. I also understand that you cant just up and write netcode that supports any type of MMO. Datbase management is relatively easy for me, the harder part would be server coordination.
Quote:Seems to me you're putting the cart before the horse by doing some quite detailed high level technical requirements first, before even knowing the tech and how you're going to use it.
Well, this is a rare instance where I actually write ideas down before I begin work on something. I usually just go off whats in my head and it hasnt failed so far (check profile for list of experience). But to keep in line with the community and to document things for future users (if I can get something put together) I decided to actually write things down.
The 32 bit ID field is only to identify a particular object to the system. It doesnt do anything more than that. This allows administrators and servers to track objects. Sure 32 bits may be low, but it seems like enough for objects the player can interact with (then again there was a time when "nobody will ever write a program that needs more than 2k of RAM"). As to the rest of your post about DBs I'm not really seeing your problem. Here is a ver simple example of how things could be organized, its an old abandoned idea, but I did have a program working using this system that efficiently managed several hundred megs of crap. Just a fraction of what a MMO would take, but a start.
Lastly, i agree with you about AI, the server codes would be simple ones, not complex supercomputer algorithms. The idea isnt to be complex here but to make a believeable approximation of a persistant universe. Basically the programs or scripts would take several inputs and produce an output according to some simple rules. Like if theres clouds over an area and the temperature and air pressure are right you get rain. Things could be "firm-wired" to occur in some places at a higher frequency and/or seasonal basis, with some randomness added for fun. Some things however need to be programmed well, especially planetary movements. If one is traveling between planets it would suck to have nothing where a planet was supposed to be and little fuel left to find it.
But anyways.....
#7
ive made several mmo's and its no picnic, mmos need a lot more then just good networking. they need really good tools and content pipeline so you can continue to grow and maintain the game after ship.
customer support tools, noc maintenance tools, patching system, billing system, database, database middle tier server for translating to and from the database and possible for caching data, since getting data from the database in real time can be way too slow in a mmo.
the servers need a dirty bit system for tracking what data changed and what needs to be updated to the db.
you need other server processes as well, such as a spawner manager (for managing the to other server processes)
you need a group manager(guild system), a chat server (sorry torque chat is not global chat), you need a session manager so that the loginserver (oh yeah need one of those too) can tell where the player is and if they are logged in already.
what your proposing also sounds like a zoneless world were you migrate from server to server, although this works well and has been used in several mmo's its not a very scalable design at all.
32bit ids will definatley not work, but that depends on one how many unique objects will be in the world (for example if you craft something and you want to put the players name on it, now you have to store it and need an id).
if you dont reuse the id (say someone deletes an inventory item, do you resuse the id or grab a new one when someone creates another item in the world? if you keep grabbing ids then you will run out.)
i could go on for pages, but so thats just the tip of the iceberg, MMO's are a lot of work, just getting the DB stuff right is out of most peoples grasp (even WoW is having DB problems). so its a huge project, requiring a lot of functional experts in many areas to get it right.
if youve never made a game, I wouldnt suggest an MMO as your first project. getting the technical part right is hard enough but getting the game design right seems almost impossible for most inexperienced game designers(even very experienced game designers) get some games under your belt before attemping an MMO.
thats not to discourage you, its just the realities of trying to make a ferrari when youve never made go kart before.
If you do have experience, then make sure to find people who can fill in the pieces you dont know well, dont think you can do it all, no matter how smart you think you are its unlikely you are an expert in every area.
so if your all alright with all of that and understand the existing MMO designs well, both gameplay and technically then go for it.
torque is definately a good step forward for any game, but be warned, even torque is only a small piece of what you need for a true MMO/PSW type game. (of course if your just building a small MMO not made to handle many people, etc then you can get away without building a lot of stuff, but if you want to really go for it and get up there with the big boys your gonna need to do it right and get all that other stuff in there).
been there done that just trying to save you some pain. think long and hard before trying to build an MMO, maybe build a multiplayer game first then if the game design pans out expand it to a MMO/PSW. good luck.
12/28/2004 (7:27 am)
Ugh this has been brought up a few times, should do some searches there was a really good discussion about this not too long ago.ive made several mmo's and its no picnic, mmos need a lot more then just good networking. they need really good tools and content pipeline so you can continue to grow and maintain the game after ship.
customer support tools, noc maintenance tools, patching system, billing system, database, database middle tier server for translating to and from the database and possible for caching data, since getting data from the database in real time can be way too slow in a mmo.
the servers need a dirty bit system for tracking what data changed and what needs to be updated to the db.
you need other server processes as well, such as a spawner manager (for managing the to other server processes)
you need a group manager(guild system), a chat server (sorry torque chat is not global chat), you need a session manager so that the loginserver (oh yeah need one of those too) can tell where the player is and if they are logged in already.
what your proposing also sounds like a zoneless world were you migrate from server to server, although this works well and has been used in several mmo's its not a very scalable design at all.
32bit ids will definatley not work, but that depends on one how many unique objects will be in the world (for example if you craft something and you want to put the players name on it, now you have to store it and need an id).
if you dont reuse the id (say someone deletes an inventory item, do you resuse the id or grab a new one when someone creates another item in the world? if you keep grabbing ids then you will run out.)
i could go on for pages, but so thats just the tip of the iceberg, MMO's are a lot of work, just getting the DB stuff right is out of most peoples grasp (even WoW is having DB problems). so its a huge project, requiring a lot of functional experts in many areas to get it right.
if youve never made a game, I wouldnt suggest an MMO as your first project. getting the technical part right is hard enough but getting the game design right seems almost impossible for most inexperienced game designers(even very experienced game designers) get some games under your belt before attemping an MMO.
thats not to discourage you, its just the realities of trying to make a ferrari when youve never made go kart before.
If you do have experience, then make sure to find people who can fill in the pieces you dont know well, dont think you can do it all, no matter how smart you think you are its unlikely you are an expert in every area.
so if your all alright with all of that and understand the existing MMO designs well, both gameplay and technically then go for it.
torque is definately a good step forward for any game, but be warned, even torque is only a small piece of what you need for a true MMO/PSW type game. (of course if your just building a small MMO not made to handle many people, etc then you can get away without building a lot of stuff, but if you want to really go for it and get up there with the big boys your gonna need to do it right and get all that other stuff in there).
been there done that just trying to save you some pain. think long and hard before trying to build an MMO, maybe build a multiplayer game first then if the game design pans out expand it to a MMO/PSW. good luck.
Torque 3D Owner Stephen Zepp
One "server" may actually encompass 3 relatively low activity countries, while in one of the "very active" countries, it may be down to one server per village.