Game Development Community

Peer-peer or Distributed network- Scalable online worlds

by Edward Gardner · in General Discussion · 04/26/2001 (8:04 am) · 61 replies

So I got to thinking about how to scale a massively multiplayer world. Something that Phil said in the RPG thread about "if you are willing to live without total control."

You have a massive world. Let's take a shooter for example. Thousands of servers exist for games like Quake 3, let's start there. Imagine, firing up your game, and instead of picking a server to play on from the list, you pick a launch point and explore the world, you "see" fighting in the distance, and not wanting to miss the fun, you head towards it. Suddenly your in the thick of the fight. Ok this is great, you pick up some powerups, but you get bored, and go look for a different fight, head down a corridor, hear fighting, head towards it, and voila.

Now, what is my point? In my minds eye, what it happening above, is the character is moving from server to server in a large distributed network. The first server may be his home machine, the fighting in the distance would be a local server on his ISP (perhaps even geographically close) and yet a third server hosting the last fight.

So, how do you effect communication between the distributed nodes? And how do you track a player as they move from server to server in this world? If it's an RPG, you want to have some control over the player stat tracking so you don't turn into Diablo.

Maybe do it with a master server list, geographically close servers are "close" to the character, you have snapshots of the game on the close servers, maybe brodcast in a peer-peer model, not reporting to a central server, and a centrally managed database for character information?

Now I am thinking outloud, but there is something here. HOW do you have snapshots of the game available to "nearby" servers and players without dragging the server itself or the network to a grinding halt? How to transition the player form one server to the next and make it "seemless." Servers that are "far" wouldn't need to track information on your local node, nor would you need information on them, but the "closer" the server, the more information, until you reached some kind of threshold, then you are ON that server. How do you best define "close" and "far" in this context and track it?

What kind of world would you launch in an environment like this? A shooter, an RPG, some kind of RTS, some kind of hybrid?

The real beauty of this, is it's scalability. Use all those clients out there to power the network and you don't need a huge server farm to run the world at the application level, as clients come on and drop off, new areas come into existence and dissapear. You WOULD need some kind of master tracking server (or would you), and an industrial strength database.
#41
01/18/2002 (1:46 pm)
Ed,

I checked some old notes I had on a system like this I worked up for a a game that never got built.

After our little chat at the Brewery I got to thinking about some of the issues and I believe that there may be a very simple solution to the problem of scalability and security at the same time.

The gates are the key. In order for a character to move from one server to another they must go thru a gate. The color of the gate determines the security level of the world you are visiting.

A world that is full has a white gate. A world that is off-line has a black gate. Worlds which are player developed and new are one color - say red. No items on a red world can leave a red world.

As the site grows in popularity and builds trust in the community it can be promoted thru a rainbow series. As it moves up in trust the items in that world become available in all lower colored worlds.

Using a rainbow scheme (red orange yellow green blue indigo violet) you have seven stages of development for a world. At the top stage the world is fully recognized by all others - it has proven popularity and playability. It rewards fairly and the objects acquired can be taken to any other world.

Whenever a character moves up in a game world all objects of a lower color are lost (or warehoused at the players machine). The system would become self-regulating over time.

Each person interested in participating buys a copy of the "editor" version of the game and builds a world. They link it to a master server which sets it's color level based on IP address key or the like. The developer of the world sets a limit as to the number of people who can play in it at one time.

Trading items would have to be done at locations where the value of the items is recognized. Master servers at each of the color levels handles the character's items for that level. There are a few details to work out but the system should be viable and testable with very little hardware beyond what the developer's have. Perhaps one machine to act as a master server for the highest level and a second to act as a master level at some lower level to test out the exchanges and storage of items. Everything else would work on the developers systems.
#42
01/18/2002 (2:32 pm)
that sounds like a good idea =)
perhaps you could put some line breaks in there next time... that sure hurts my eyes!
#43
01/18/2002 (2:40 pm)
Yeah, let me dress that up a bit...
#44
01/18/2002 (9:36 pm)
My question is this:

How do you do this and still keep new area interesting and new?

You'll need structures to diversify everything, and downloading a level every switch doesn't make things very appealing.
#45
01/20/2002 (6:51 am)
That's a really good question. I think that for a FPS the answers would differ significantly than for an RPG.

For a shooter the are issues of balance to keep one type of player from gaining an overwhelming advantage. Each player needs to be able to play the style of game they want to play without being totally secure from other players. They must have access to the type of weapons and ammo that they want to use, but they should only be acquired at some risk. The list of balancing issues could go on for a very long time but those seem to be key.

For an RPG I would say that there are several things you need to do to keep a level interesting. It should be challenging enough to keep people coming back, the rewards should vary intelligently so that each time you return you get something that is useful to your character or team. The area itself should be visually interesting. There should be an area where people can congregate to stage adventures, discuss progress, and trade. RPG's are more about community, so opportunities for community have to be present. Monster AI must be intelligent and non-repetitive. When the door opens the troll might not be in the same place, position, or state of readiness as the last time. He might even have a friend!
For major items that don't change you need to make the paths to the item an obstacle. Leap chasms, a special challenge to get a key, or a long walk through a trapped room? Make your choice.

Finally, you need to take into account those players who would spoil it for others. Preventing the high level character who camps a great item to try and accumulate a few thousand to resell/trade later comes to mind.

Also, you need to pay a lot of attention to your testers. If they don't like an area - change it - as long as it still fits within the theme of the area you are trying to make. If they have suggestions for improvements try them. If they all think it is wonderful the first time they play it - get rid of them and get a group who will tell you the truth ;^}
#46
01/21/2002 (9:44 am)
I had given this some thought in terms of battle tracking with Red Baron II/3D (M)MP and Tribes(1)

For those that don't know , Red Baron 3D server software is similar to Tribes , in that it was given with the game software. The capacity , with sufficient bandwidth , is around 50 players , although 75+ has been pulled off for a few hours.Much though has been given as how to run a long term war event across the entire western front.

(Based on some things Tim Gift had said in the past, I suspect the network code of Torque might handle 200+ players with the kind of activity found in a WWI flight sim)

The 4 historical western front maps are huge in RB3D.Lets remember the "world" is on the clients machine , and the server only transmits the positional , damage state , and other object state data in relatively small packets.Client bandwidth is not that critical , controlling data packet loss is.

So in this rather simple example , "tying" together 4 maps/servers for a full western front battle , would be a matter of parsing the "score" logs for each player from each server into a database , and that database being able to update the "score" logs of each server.

The live gameplay positional and damage data only runs on the particular server btween it and the clients ,only "score"(including character change) values need to be stored-exchanged-updated between "live" servers

Transition between servers needs to be addressed , as each map/server would have boundries in this setup .. however space seperation can be cleverly/creatively handled in almost any game scenario where the initial maps are large enough for the number of players to start with , and provide the necessary variety of gameplay within each's contained space.

Off the top of my head , such a transition between servers in an historic flight sim might be handled by the player sitting in a train car in WWI , or as a passenger on a transport flight in WWII. In a game like Tribes , there could be impassable mountains between servers , and the transition a tunnel/cave. Crossing waters between landmasses or having to walk thru a windstorm valley. being "blinded" and unable to shoot anyone for a short time :)

Basicly you create a few minutes(at most) of believable transition time where the servers can update the player state between them, while the player is "static" in relation to any gameplay score or state change.

Of course , I'm not looking at this from a programmers point of view , so there may well be some huge problem with this admittedly simplified outline !:)
#47
01/22/2002 (5:38 pm)
I'd just like to jump in here and say that I think that even without a revolutionary scalable server architecture of some sort, you sould still make a pretty damn good 'backyard MMORPG'.

In Tribes 2, 120+ person games have and I believe still are being successfully hosted. I have a crappy DSL connection (I get about 240k-sec) but I can still play on those servers.

What if we were to host a 500-person game? Impossible!

What about a game like Age of Empires, though? Look at all of those little guys running around! Big 8-person games can have close to 1000 little guys in the whole map, man! 1000 people wouldn't be so bad!

Well, hold up. When one of those little guys moves, the only information that has to be updated are his stats, current location, target location, and what he's attacking. That's simplifying it a bit, but really the only important stats are those ones. And his location is tile-based. With Tribes 2, we're sending complex and precise location, heading, and velocity information in addition to other less instantly noticeable things. When there are 120 players on a server, and our ping gets around the high 300's, you really notice people start jerking. Add to this the burden of pixel-perfect aiming and all of the bullets and things that are zipping around the world, and, well, you get the idea. In the Tribes 2 model, we need to be able to update the client on EVERYTHING, and we need to do it in about 1/10 of a second or things start to get ugly.

So comparing our backyard MMORPG and the characters on the server to Age of Empires and those little dudes really isn't a good analogy because our networking model isn't anything LIKE . . . oh.

An MMORPG doesn't require us to send the clients the same sort of highly detailed information about all of the server's other inhabitants, like a FPS does. Look at Ultima, or even a closer analogy like Diablo. A great way to speed things up even more would be to use a point-and-click system of movement instead of a walk forward/backward/turn method. Why? Well, because that will make client-side prediction way smoother. If my computer knows that Mikey is on his way to point B from point A, we might go another second without Mikey's position being updated, but there won't be any apparent lag.

I'll leave any kind of server-linking code to others to figure out, but see, if we could host 500 people on our own server, it suddenly becomes apparent that we are hosting a server that holds more people than an Everquest Zone can handle easily at once! Heck, 100 people would be incredibly sweet; want to have your three friends over for D&D? Or how about 100? But we can handle 100 in Tribes 2, it just isn't even playable by people with dial-ups.

What do you guys think? An essentially grid-based system of movement, with a location-to-target structure(maybe we could work out something different in cities or buildings) would work wonders for the number of people we could fit on our server. And if we threw in some very simple location-based netcode optimizations (kind of like netcode minizones, where we only update the client on people within their square and adjoining ones) we could easily be pushing 1000. Of course, the server would need to have a buff connection, because it's still going to be outputting information to more total clients, but if we can keep the amount of data that it sends to each person down, it might be just as affordable as having a good Tribes 2 server. The biggest servers would have to be really good, but a 300-person server -- pish tosh, no problem at all.
#48
09/30/2004 (2:56 pm)
Ok this may be stuipd, but you could essentially have a p2p with a central server.

First have the player files temporarily stored on the client machines kind of like the windows temp directory or temporary internet files.

When a person decides they wish to play they login and the client synchronizes with a central server the temporary files are downloaded from a central server or "cluster" if need be. When the player picks up an item, loses an item, gains experience it is changed in the temporary file on the client machine. During game play the server the player is on looks for the temporary files and synchronizes per server or "world" the player is on. Then that server synchronizes with the central server at a given time interval. After the player decides to logoff the files are synchronized with the server they are on and deleted from the client machine. In this way you could run your own server and have people connect to it and play like in tribes 2.

Like this

Events:
Initial login
Client computer <-sync-> central server or cluster

Game Play
Client computer <-sync-> client server

Log off
Client computer upload-> client server

Time interval
Client server <-sync-> central server or cluster

New server
Client computer <-sync-> central server or cluster

This would obviously not be a true p2p, but it is close. Otherwise you would have to store the client machine files on the client machine and hope that no one tampers with them.

Just a thought...
#49
09/30/2004 (3:54 pm)
@Barry Mesa: Old thread. But, what if the client modifies this information on the fly, then? I can't see how that would work at all. Never trust a client.
#50
09/30/2004 (4:33 pm)
You could in some way lock the files so that editing them knocks you out of the game and the client would lose any un-synced inforamtion or lock the interface so you cannot jump out of the game while playing it.
#51
09/30/2004 (4:46 pm)
I love this thread though, amd very interested in hearing new ideas.
#52
09/30/2004 (11:33 pm)
I found a really cool mmorpg and it's free to play. www.project-entropia.com

It has an interesting concept if you choose to use it.
You can use real cash to buy in game money, and you can exchange in game money for real cash.
#53
10/01/2004 (1:53 am)
@Barry Mesa: That wouldn't work with a Peer to Peer approach as the clients are responsible for what's been sent and recieved, the master server only being their for auth and security purposes.

The client can just be modified to not send anything further (or the distributable serverpackage, whatever).
#54
01/14/2005 (4:09 pm)
Apparently, the "how" I wonder about in my initial post here is with TNL, as evidenced by Dave Wyland's recent plan :)

Funny old world...
#55
01/16/2005 (12:12 am)
Interesting thread. I just finished coding my initial peer to peer system which works as follows. I have a master server consisting of a mysql DB on the internet and some associated C cgi apps. My game "world" is devided into many villages. Each village can only be hosted by one server at any one time, allocated by the master server.

The player decides which village he want to visit. When he joins a game (village) the master checks if its being hosted already, if so the player joins as a client (if there is enough player space). If the game is not being hosted the player gets to be the host, provided his cpu is fast enough.

All mission filea are kept on the master server and downloaded to the host who distributes it to the clients in the usual manner. With every heartbeat event, the mission file is uploaded in chunks and crc checked etc to ensure that the integrity of the master server missions files are kept safe from broken uploads and so on. Thus any in game location changes and additions of game objects are made persistant. Simple in concept, not so simple to code, but seems to work as desribed in practice.

So thats my version of a very low cost persistant universe world.

But I tend to loose interest because there are so many games out there that it seems impossible to ever be competitive and make a buck along the way with something like this. I guess thats just me because my ideal activity is something thats fun and profitable, not just fun, so I tend to drift onto other activities when I can't see realistic profit projections.
#56
03/22/2005 (1:16 pm)
How does this compare to terrain "paging", is paging just a part of a "large scalable" game world?

Is it really so hard to load a different map once you reach the edge of the current one?


i imagine a farily simple X-Y grid for terrain tiles... 0,0 would be the origin and where new players would always start..
img90.exs.cx/img90/1134/grid3ez.png
#57
03/22/2005 (3:48 pm)
@Ed: While the concept is simple, the implementation is not. TGE is designed to only have one terrain block, period, so right off the bat you are looking at probably 20+ different systems that use the terrain block that must be wrappered to handle terrain block paging.

Then, you have to deal with collision across boundaries, rendering across boundaries, the boundary crossing itself, plus a host of other "minor" issues that all add up to a very huge modification.

I spent about 2 and a half months on this exact concept, using a 5,500 line patch file against TGE 1.1-1.2 in an attempt to bring fully working terrain paging into TGE 1.3..and once I found out that TSE will support nearly infinite terrain, I dropped the entire working copy like a hot potato. I'd MUCH rather spend the money for TSE, and the time to port our current project to TSE when the time is right than deal with the effort to get it working in TGE.

That is NOT in any way a slight on TGE, but reality when it comes to my project! It's a lot of work, but it absolutely can be done.
#58
05/14/2005 (10:26 am)
*IDEA* How could you set the boundaries on each map to bounce you to a new server when you crossed the line?

Client [on current server] --> |Play Area Boundary| -->Client [New Server]

Also, How can you increase the size of each map so that the terrain repeats outside of the boundary?

Thanks
Barry
#59
05/14/2005 (10:37 am)
Use a trigger, there's a resource :)
#60
05/22/2005 (10:01 pm)
Can you post the link I can't find it. I am not sure if it is the new format or what. Thanks