Game Development Community

Peer-peer or Distributed network- Scalable online worlds

by Edward Gardner · in General Discussion · 04/26/2001 (8:04 am) · 61 replies

So I got to thinking about how to scale a massively multiplayer world. Something that Phil said in the RPG thread about "if you are willing to live without total control."

You have a massive world. Let's take a shooter for example. Thousands of servers exist for games like Quake 3, let's start there. Imagine, firing up your game, and instead of picking a server to play on from the list, you pick a launch point and explore the world, you "see" fighting in the distance, and not wanting to miss the fun, you head towards it. Suddenly your in the thick of the fight. Ok this is great, you pick up some powerups, but you get bored, and go look for a different fight, head down a corridor, hear fighting, head towards it, and voila.

Now, what is my point? In my minds eye, what it happening above, is the character is moving from server to server in a large distributed network. The first server may be his home machine, the fighting in the distance would be a local server on his ISP (perhaps even geographically close) and yet a third server hosting the last fight.

So, how do you effect communication between the distributed nodes? And how do you track a player as they move from server to server in this world? If it's an RPG, you want to have some control over the player stat tracking so you don't turn into Diablo.

Maybe do it with a master server list, geographically close servers are "close" to the character, you have snapshots of the game on the close servers, maybe brodcast in a peer-peer model, not reporting to a central server, and a centrally managed database for character information?

Now I am thinking outloud, but there is something here. HOW do you have snapshots of the game available to "nearby" servers and players without dragging the server itself or the network to a grinding halt? How to transition the player form one server to the next and make it "seemless." Servers that are "far" wouldn't need to track information on your local node, nor would you need information on them, but the "closer" the server, the more information, until you reached some kind of threshold, then you are ON that server. How do you best define "close" and "far" in this context and track it?

What kind of world would you launch in an environment like this? A shooter, an RPG, some kind of RTS, some kind of hybrid?

The real beauty of this, is it's scalability. Use all those clients out there to power the network and you don't need a huge server farm to run the world at the application level, as clients come on and drop off, new areas come into existence and dissapear. You WOULD need some kind of master tracking server (or would you), and an industrial strength database.
Page «Previous 1 2 3 4 Last »
#1
04/26/2001 (8:51 am)
Peer 2 peer proablly would not work for a massivly distrbuted game such you described. Something more akin to a trusted distrbuted server network would proablly work. Each server node would negoatiate to be connected into the secure servers network. Some servers would act the way DNS act for the interent. They would store routing information so you connect to the server closest or your destination server. To take into the account for possible server failure, a fair amount of redundancy has to be built into the network, in terms of data storage and perhaps even back up servers.

It's a cool idea, a small scale version could be created fairly easily.

-ddn
#2
04/26/2001 (9:10 am)
That's kinda where I am going with it in my head. DNS is a good analogy.
#3
04/26/2001 (10:30 am)
Edward,

This is essentially what I'm researching. Here's a few posts on GG that go along with it:

MMOG Networking
Game Networking
Big Terrain (scalable networking near bottom)

I believe a peer2peer model could work for a lot of different game types. Perhaps not an MMORPG, due to security of the character data. But definitely adventure-type games, shooters, MM-Tribes, etc.

With some additional work, adding trusted-server authentication to a master server, a peer2peer model could include character data, etc. In this case, you'd just have to trust 'the system' to keep rogue servers in check.

I see the challenge of this model to be roughly similar to that of clustering. How do you propagate state updates between nodes in this mega-cluster, assuming the only thing they share is a network connection? How do you implement fail-safes, redundancy, etc. to ensure the world survives if a power outage takes out some of the nodes? I have some ideas on how to solve these problems, but I'd also like to hear fresh insights.

Or you could simplify it: each node is the whole owner of a particular piece of the world. Then how do you deal with overcrowding? What happens if you logoff from that server, but it's not there when you want to play again? If that server goes down, what happens to the avatars on it? Who stores my character data?

--Bryan
#4
04/26/2001 (10:52 am)
I think you have to have SOME kind of master database server if you were storing character statistics and items in. So, certain things you would pick to track, and that "state" would be what followed you around, perhaps a local cached copy, updated async periodically thru the game?

But, (again in my mind's eye) I see it as the "simpler" model, where that server owns that part of the world, wholly owns. Leave it to the node admin to set character limits for his node, just like in Quake. If the server is laggy, leave, that kind of thing.

I'm still struggling with the what happens when you logoff, and where you start up again.
Intuitively, I like starting where or near where you left, but I could see starting in a "lobby area" or on your own node.

The whole how much control do you give the server op is a neat mind puzzle as well. Do you make like the FPS, where they have total control over maps and mods, but still register with the trusting authority? Or do you "disperse" environments to the distributed nodes, and limit the choices for the server op?

The basic commincation between the client/nodes and the trusting authority, the database, and the general scheme is the challenge, some of the details are just that, details. I think some of them would fall out, once you picked a genre for you MM world, no? Shooters wouldn't matter for some of it, but RPG's would be very sensitive to certain aspects.

Ok, how do you effect this kind of communication, and what base characteristics would you track about the servers and the characters?
#5
04/26/2001 (11:12 am)
I am going to pull off on the database side here.

I would think that do what you want you would have a spider setup so to speak. Each server would have a dedicated database server setup to associate with those clients that connect to it. Each server would then do periodic updates to a intermediary server which would store index information for how to access that characters stats. This would then report to a central index server containing contact info to the server that holds each characters data.

While complex in nature it would allow you to have a character connect anywhere in your world and immediately recieve their information without extreme effort. A full server dump from base server to master server with this would take forever and would tie down resources incredibly, but an indexed setup would allow for quick access between base database server to base database server. It would be unlikely that all players will jump server to server, but there is always that possibility.

To simplify replication of data it would make sense to have a main db server that supports possibly 10 base db servers that gets a dump fo the database on an hourly database. However this does make the one assumption that the main information for each client is stored on the server that they signed up on. However their information will be moved to whatever base server they login into for the duration of their time there.

The best part is there would essentially be no limitation to the amount of information you collect from the clients, short of hd space which would have to be taken into account from the beginning.

One last thing. If you wanted to protype this setup it is very easy to duplicate and stress test.
#6
04/27/2001 (1:11 am)
There is an alternative to getting the technology to implement this type of idea.

You could modify the idea slightly to approach the technology, without loosing the vision.

I'll illustrate this with two points:

1) Massively multiplayer is in many respects about community, and continuity. None of this requires graphics. A simple example being automatic online FPS ladder systems (stats), which track how well a person does in a given game.

2) The vast majority of people already get into a server via a central database, using a tool such as GameSpy.

I have my own ideas on how to simply and easily implement massive multiplayer with V12 type technology, but it's the approach that's important. By slightly altering the current framework, you could easily create a massive online community in a persistant universe, without players being able to walk from server to server, or see the beyond the current server.

We've been fooling people for years that the 3D graphics they see represents a reality, so why not do the same with this?

Doug EnkiSoftware Limited
#7
04/27/2001 (6:04 am)
Ok, the biggest issue for all this is that of security and authentication (both clients and servers).

Assuming you have this covered, you then look at the structure of your worlds. Here is an idea I had for my space/genetics game:

Each player hosts thier own "homespace". That is, they have a grid coordinate (lets say that its a tranlation of thier IP address into a 3d cube). From that homespace,they can enter "jump coordinates", that is they can warp to anywhere.

The jump coordinates would basically be a url type address, which could be resolved by name servers to the actual machine hosting that space.

One thing to point out here, that I think is nice, is that permanent (i.e. hosted) servers would tend to become a "hub" of the game. People would start in thier homespace, "jump" to a hub, then communicate with others and find out about new "hubs" etc.

The point of the game was space exploration, trading and genetics (with some small fighting as well).

Imagine that the "space" you are in, is essentially a "namespace" as well as a 3D representation of that.

Phil.
#8
04/27/2001 (6:42 am)
It's the same kind of of model that VRML wanted to be a few years ago. Except, hardware accelerated and with weapons.

But, that's pretty much what I was thinking.

Issues: Security, authentication, transition between "nodes", what does each node know about the other nodes, how much control doe node operators have over the local system (see the first 2 in the list). What else?

Sounds like we're thinking along the same lines, Phil, hope I'm not stealing any thunder from you :)
#9
04/27/2001 (7:00 am)
Not at all Ed, its all just thought until its implemented :))

Another few things:

Resource control (if the game design has resources, then you must have a way to control its dispersal and concentration).

Versioning

Of course, a distributed model means that different versions could be active, you either deal with this at the "master authentication" level, or at a lower level.

One thought I had was to implement this whole thing as a "virtual file system", that is, each peice of data is a named "file" (think of file as collections of data).

The purpose of this was that you could write a network layer that pretty much worked like a file system (file descriptors, filenames etc), then say, you have a set heirarchy of different data types.. heres an example:

models
->world
->ships
-->scorpion
-->sabre

say, another player enters my "homespace" and doesnt have a "sabre" craft, they could trade with me for it, if I accept the trade, I do what is essentially a filecopy of the -->sabre directory onto his filesystem, and hence he has that available.

The reason I thought of a filesystem, is that it seems like a problem that will have been thoroughly researched.

That and I dont really like the www protocol.

Phil.
#10
04/27/2001 (7:52 am)
I think, if you were to get this done quickly, you would have to go the "Master Server" way. Use the Quake analogy again, only same version clients can connect to same version servers, and those servers may or may not support autodownload of things like models. Make it so the servers HAVE to be the same version (some sort of check when it heartbeats to the master), and hotfix the servers with an autodownlod from the master.

Keep it simple to start :)

I think I need to break out Visio, start drawing this stuff out...

I think you are right, the dedicated servers would become the hub, but, if it were flexible enough for the peer servers to play a role, that would be a good draw.

Maps/Environments could be a challenge. Maybe something autogenerated, like Diablo II, for the peer servers (environment, items/power ups, etc), and maybe push content to the dedicated servers (cities, towns, encampments in an RPG game, for example).

Definitely need to draw this out...
#11
04/27/2001 (8:53 am)
Yeah, I agree, do a typical FPS client/server model as persistant "entry points", have some naming system that will know about each active server and provide name/ip resolution for them.

Basically a kerberos like authentication should take care of a lot of issues.

I saw the "personal" servers (i.e each and every player hosts a small server) as a sort of homepage that allows them to start at a known point. It would also be the place where they store gathered goods and information (but still in the persistant world while they are online in the game).

I wonder how scalable a game such as this could actually get?, I guess the biggest issue is to control all protocol transactions such that you can set a "maximum" data transfer rate on any single part of the system (i.e. analyse all parts of the system and model thier individual transactions etc.

Phil.
#12
04/27/2001 (9:08 am)
I like Phils' idea on the namespace becoming physical space, however the only major issue with that i think is that you would have to be able to distingiush which areas are not able to handle traffic such as someone playing on a 56k modem. While you address this by having hubs, you are back to the only really playable space is the hub since it's purpose is to be a server to the world.

Perhaps it should be more dynamic to provide the ability to for those that have good connections to setup servers but not require the creation of space just becuase a person logs in? This again brings up the need for version control. But I think this has been handled rather well to date in games like HL. There you have to login to a WON server before you can even enter a server. If you fail the version check, you don't play. If we maintained one or two master servers whose life was to check versions we could version control the servers peoples are putting up.

On to my next point. Your idea of the file system makes sense, but I personally believe that we could better maintain the "world" environment if we used a database style connection. Your schema would still hold value and would be accurate and easy to maintain. But the transfer speed to the database would most likely be faster than using standard I/O. It would also give us a final control over the naming conventions used on the said object, preventing multiple objects of the same name. This would be of particular interest to players that go hunting for a specific object and don't want to get swindled by someone who just name something the same name so he could sell it.

I think that a SSL database connection could be implemented in the said project's client making the communication secure and help prevent possible hacking of the data or client information. To even go a step further we could lock the communication server down to the ports necessary for communication to help prevent unnecessary access to the database. However how do we handle this in the case of someone setting up their own server?
#13
04/27/2001 (9:26 am)
You can get around the problem between Namespace and physical space issues by having a naming convention that allows heirarchy. For instance, top level names in the heirarchy would be "permanent" parts of the gamespace.

Then you get less and less reliable as you go down in the namespace (as an example, a class C or D ip address is generally going to be more likely a non-permanent machine than a Class A).

I think it really wouldnt be massively difficult to extend the tribes 2 networking model beyond just adding master server authentication (which would obviously be the first step, and is something thats required if I read the posts about this issue rightly).

Perhaps thats something I'll take a look at, adding the whole WON.NET type authentication service to V12 (it would make sense because Ive done it before, and I'm probably going to have to do it again).

Once you have an authentication backend, then its on to the whole "game world" authentication and organisation stuff.

So here's a good question, what sort of configurability do you think would be needed by an authentication service to allow it to handle multiple games and game organisations?

Phil.
#14
04/27/2001 (10:53 am)
Hvae you looked at something like FSGS? Last I checked, they had some authentication built into that and can support multiple games, I bet it could be strenghtened to support what we are talking about without having to write from scratch.
#15
04/27/2001 (6:41 pm)
Just a note -- to solve the problem of rogue servers -- for instance, a mod that requires custom 3D models -- have the game limit the extent of the customization and control the admin has! It's got vastly limited Super Admin powers -- you can't change the map/edit the terrain, you can't kill players, etc. And as for the object system -- you need a way to solve the problem of the objects! What if I create a Gauntlet of L337N355 that makes the wearer invincible? What happens when he leaves my server? Oops.

Instead, the game has built-in limitations on the amount and power of objects that can be created. Otherwise, server admins will give members of their clans/guilds/tribes super-powerful weapons and spells, etc.

What I'd really like to see solved is some way of 'going shares' on hosting a shard of the world. For instance, a group of 5-10 people all want to host a world, but they all have just basic DSL and couldn't handle too many people. So some sort of gnutella/SETI-at-home distributed computing takes place between the 5 lower-bandwidth servers, allowing them to host a very large and expansive world.
#16
04/27/2001 (6:52 pm)
Hey, guys, I think that there might actually be some interesting ways of making the Tribes 2 method of networking work pretty well.

Obviously, it would be easier to use a 'single server' method, rather than distributing the computing across thousands of machines and creating one immense world.

So instead, let's just try to find a way of making the T2 model work for us! Let's find some way of making the 'single server, single world' approach work, instead of taking this immense amount of coding upon ourselves.

For instance, instead of having a world where you can go anywhere, what about using something like those teleporter thingies on Asheron's call? Just write up the story to explain it -- for instance, the portals are actually portals into other worlds. If you go into a portal, you can transfer from server to server. And there is no master server list! You HAVE to use this system to switch servers.

Of course, this would require a number of large dedicated servers to begin with; there has to be a point of entry, and a place that the player can come back to if a shard of the world goes offline. However, limiting the usefulness and interest of those worlds -- making them 'safe' zones, providing no powerful objects, disallowing magic use, etc -- should help to move people through quickly. Also, the privilege of being a 'gateway' server would be granted by the master authentication server.

There are a lot of very creative ways of using the Tribes 2 multiplayer model for a MMORPG-type game. We just have to be really creative. ;)
#17
04/28/2001 (3:40 am)
One thing I'd be quite happy to do for the V12 engine (and subsequent "credit") would be to write a Won.net style backend service (I've done it before), this is also something I'd be doing anyway (for a new employer), so I'm prepared to put in the work and really make a cracking setup.

But:

What do we see as the immediate requirement for this type of thing? basically copy the Won.net model?

How do we see it hosting a number of different game types?

How do people think it would be best structured?, I tend towards modifying available solutions rather than starting from scratch, prefer protocols that are tried and tested, but I know from experience that security is a huge huge issue to services of this type.

Basically, if anyone wants to give input, comments, or help out, that would be great, After a few weeks of interviews and such, I'll be free for a while waiting for visa stuff to get sorted, so it would be an ideal time to work on this.

Feedback would be a great help, as its something everyone would get access to.

Phil.
#18
04/28/2001 (3:48 am)
As a side note, I checked out the FSGS stuff, Its kind of the type of thing I am proposing, however I couldnt use that for any of my own games, because I would lose control of the network too much.

I would also want source code to anything like that, in case a security hole appeared, after all, you are responsible for youre own services.

Phil.
#19
04/28/2001 (7:03 am)
Good questions that I don't have answers for ;)

I've looked at the FSGS licensing stuff, my tendency would be to license something like that and get the source, or use the system as a template for features, etc...

I'm heads down for a couple of weeks starting Tuesday, have no time till mid to late May (getting married, going on honeymoon, etc ;)). I would be interested in seeing if this could go anywhere on my return :)
#20
04/29/2001 (10:56 am)
Well, in reviewing the license model they have here:
http://developer.net-games.com/products/sdk/licensing/
I realized that it's not very clear.

Nothing like a direct question to get your answers, we'll see what they say about licensing :)

http://developer.net-games.com/forum/showthread.php?threadid=54

We'll see. I am from the buy rather than build school if you can get it :)
Page «Previous 1 2 3 4 Last »