General Networking Discussion--Game Genres

#1 Good piece of information.

MMO's, as popular as they might be, still cheat alot when it comes to networking. Torque doesn't contain a whole lot of that, and is more accurate out of the box. For instance:

* 1. There's a reason movement is so smooth in WoW even with 1k ping. It's because their design allows the client to have a say in how much you've moved, and they only do marginal checking of it to make sure you're not terribly out of sync. This has the side-effect that if a client lags, other players will see the lagging clients player as warping a bit.

* 2. Scoping isn't checked as much as in Tribes/Torque, and if you move fast enough you'll end up outside your current scoping area and entitys will start to pop up. If you profile a Torque game with many many players in near vicinity of eachother, you'll notice that this takes up quite a few cycles.

There are tons more "cheats" but I'll leave that to others.

#2 First of all, I love Torque!

Quote:
With about 7 changes to the underlying code and a few adjustments to previously programmed parameters, you can easily serve 200 players with a decent network (not even a good one)--and that's more than Everquest serves even today per zone.

Stephen, I think it would be in this case extremely illustrative to pinpoint the areas of code which would facilitate that particular statement. The capabilities of Torque are often examined in decision-making on which engine to pick, so having this type of change readily documented or at least described better would probably go a long way towards improving credibility and street swagger.

#3

Quote:With about 7 changes to the underlying code and a few adjustments to previously programmed parameters, you can easily serve 200 players with a decent network (not even a good one)--and that's more than Everquest serves even today per zone.

I will want know wich are the adjustments for a MMOG (RPG - STRATEGY, ok i understand that no mmofps...).

Thanks a lot!.

#4 Not that I want to get into an argument or anything, but ...

Quote:you can easily serve 200 players with a decent network (not even a good one)"

I would say that is misleading. Yes, you can easily serve 200 players per server ... but the pipe to the 'net is only so big, and you'd have to account for the bandwidth used by every server using that pipe. For example, if your pipe is only big enough to support 200 players, but you have two servers each capable of hosting 200 you're still only going to be able to support 200 players total even though the servers can handle 400.

For efficient use of bandwidth in an MMO, there are a lot of other things that you would have to do. For example, Torque mission load is seriously heavy on the bandwidth. You can cut a lot of that out by loading everything that's static (datablocks, buildings, etc) directly on the client and not transmitting them over the network. Further, things like the camera, projectiles, etc should be client side in an MMO and Torque does those server side.

In summary, to make an MMO on the scale of e.g. WoW or Everquest in Torque does require significant changes to the codebase. However, it is not anything like a complete rewrite. It's doable for a knowledgable indie and Torque's netcode is actually pretty well suited to it. The way you need to use the netcode is just different to the default, because the default is setup for a twitch FPS.

It's also worth pointing out that you don't need to do anything particularly major right off the bat with Torque. An MMO has a lot to it that is not related to networking. You can get everything built in a one zone "MMO" without touching the networking at all if you're careful how you go about it. You only need to worry about networking once the game is sufficiently progressed that you either need more than one zone or actually have the people playing it to require more efficient networking.

As with anything in game dev, it's all about priorities - who cares if you can support thousands of players if you have no game for them to play ? :)

T.

#5 I'm definitely planning on jumping back in to this conversation as soon as I can, but I'm super task-saturated today and tomorrow. I'll try to throw up a more complete analysis here in the next day or so.

#6 Stephen, really looking forward to your identification of those 7 places... Hope you will include suggestions for the changes to make as well for different MMO types..

Bookmarked so it doesn't get away...

#7 1. how many concurrent users we can host with this server:

* IBM X-series 366
* CPU: two dual core 2.8GHz Intel Xeon processors
* 4 GB RAM
* 10,000 RPM SCSI hard disk
* 1000 Mbps Ethernet network interface card

2. is very hard or is possible with others tools convert the tge server in a load balance servers (cluster)?
en.wikipedia.org/wiki/Load_balancing_(computing)
www.onjava.com/pub/a/onjava/2001/09/26/load.html

#8 With the stock torque (engine and scripts) such server can hold (at peak times) 10 torque instances with each serving upto 100 players (assuming you run dedicated servers under correctly configured GNU/Linux or FreeBSD) with the game latency not more than 50ms.
The more scripts and logic is added - the more latency will be == less players in total.

But it dependents on the internet/network connection too.

P.S.
This is assumption and based on my own experience and can't not be taken as 100% confirmed figures.

#9 1. How many concurrent players you can host on any server depends entirely on your game. The only way you can find that out is to make your game and test it. As bank said, it also depends on the internet connection.

2. Load balancing with Torque would be pretty difficult and would be a lot of work to implement. If you really have so many players that you need load balancing, you'd be better off with multiple independant servers and allowing the players to choose which they play on.

#10

Quote:With about 7 changes to the underlying code and a few adjustments to previously programmed parameters, you can easily serve 200 players with a decent network (not even a good one)--and that's more than Everquest serves even today per zone.

mmm, i am waiting for the tips for optimizing the network connection... thanks... whatever...

i readed this article: www.garagegames.com/articles/networking1/index.html
www.garagegames.com/docs/tgea/official/content/documentation/Engine%20Reference/...

edit:
thanks, read this article
tdn.garagegames.com/wiki/TorqueNetworking
answer my questions...

#11 Ok, this post has been a long time coming, and I apologize for the delay. I'm going to break it out into a couple of different perspectives over the next couple of posts (probably next couple of days as well).

The first, and absolutely most important decision to make is: what is my genre, and therefore, what are my networking constraints?. I'm going to make a few very broad statements here, and of course you can pick apart any particular set of goals or point at inconsistencies with your particular needs, but this is intended as a "generally speaking" type of discussion.

FPS: Historically, two of the biggest lessons learned with multi-player FPS games from a networking perspective have been:

Authority Matters: Early on in the history of FPS development, networking bandwidth and CPU load distribution was at a premium--the Internet was very limited, and computers just weren't powerful enough to do what game developers wanted. This lead to many experiments regarding who can tell the simulation as a whole what was going on--client side hit detection, authoritative client input without server side validation, low frequency re-synchronization (or no forced synchronization at all) have all been seen in games (and are sometimes still seen today).

Update frequency is critical: We all hate lag (more properly termed latency when speaking specifically of networking). With today's (fps) games and today's (fps) gamers, 200 ms is too high of a one way trip time, and I've routinely seen people refuse to play with > 50ms ping. When it comes to user control of a particular avatar, zero "lag" is accepted--your player has to do what you tell them to do, when you tell them to do it.

FPS games generally aim at between 8 and 32 players in a given game instance (again, broad, general statement here), and this target is actually most of the time defined by the very points listed above (as well as many others).

Racing: I don't want to spend a ton of time discussing this one since it is generally a much smaller market, but there are a couple of networking related things that take focus when speaking about a generic racing game.

Physics response and accuracy: Racing games normally require a much tighter physics coupling between all participants. Game design can preclude this in some ways, but generally speaking particular attention must be paid to make the networked physics more responsive to all player's inputs.

Synchronization timing: Clark Fagot has put this much more eloquently than I ever could in some internal email discussions here at GG, but to paraphrase briefly, he indicates that racing physics require smoothly grained updates much more strongly than other genres. Specifically, Torque networking by design allows for update "clumping", where you commonly handle multiple physics updates to a particular object in a very short period of time, while a racing game model should not allow this type of updating clumping--updates should instead of spread out over the actual amount of real time they are approximating within the simulation.

RPG/Adventure/Story games: Huge, huge artificial grouping here, but generally speaking the attribute that defines this group is "non-twitch game play". Games are probably either single player, small group play, or possibly larger scale but still within the 8-32 player realm.

This category of games doesn't particularly care too strongly about frequency or synchronization as much as the others listed above (they are important, but not necessarily critical), but they do replace them with some other considerations:

Update sizes: Commonly, this type of game includes a large amount of information transfer as strings. Storyline, Room descriptions, player chat all tend to take a lot of bandwidth.

Large grained scene changes: Commonly implemented as zones/zoning, it is pretty expected that players in this type of genre will perform large granularity scene changes as they move from one locale to another. This results in a high frequency (human scale: 1-5 minutes for example) requirement to perform a large amount of synchronization updates, and especially new object instantiations (which require a lot more bandwidth than a typical interval based object update).

RTS Games: Commonly grouped based on one player (of several within a game instance) controlling many objects instead of just one--i.e. armies/squads instead of an avatar.

Large number of updates: While FPS games based on projectiles can also peak with a very large amount of updated objects, RTS' commonly have a higher total number of objects to maintain between client/server/client.

This is further extended by actions per object, since a very common game mechanic is that players give units or groups of units commands, and the game performs those commands automatically until no longer able to do so. This sets up a scenario where players can issue relatively few user inputs that result in hundreds or thousands of game state changes.

MMOXX: Huge category of course, since "XX" can be "FPS", "Racing", "RTS", "RPG", or any combination of the list. There is no way we can possibly cover all the combinations, but generally speaking, any MMO is defined by the following:

Extremely large amount of concurrent users: Let's aim at a reasonable indie number of 1,000 concurrent users as our example case, although obviously 10,000, 50,000, even hundreds of thousands or millions need to be at least assessed.

The rest of the follow on posts will rely heavily on a good understanding of the general Torque Networking implementation, especially the general challenges and strategies for resolving those challenges in a generic multi-player game, so I strongly advise studying the (old, but not outdated) Torque Networking Overview on TDN.
(end of post one)

#12 Great post and very informative... a few comments:

RPG/Adventure/Story games:

Quote:Update sizes: Commonly, this type of game includes a large amount of information transfer as strings. Storyline, Room descriptions, player chat all tend to take a lot of bandwidth.

If you cache this data on the client, you would be surprised as to how little bandwidth this requires.

Chatting is insignificant. If you have only 32 players, I seriously doubt "players chatting" will consume much bandwidth. Most people can't type much faster than 50 words per minute, which ends up being less than 2kbps total, even if you have 32 players typing as fast as possible.

Large grained scene changes are a bit outdated. If you have terrain paging and even "zone" paging and caching, your bandwidth utilization will be much lower and your players will love you for it.... without the dreaded "loading" screen happening every "1 - 5 minutes", your players can play more and wait less.

(In case you were wondering, happy customers continue subscriptions and buy the upgrades. Disgruntled customers are a pain in the arse.) :-P

For MMO games and even a lot of other games that have any form of in-game community, there's a whole lot more than just the bandwidth being used by the "zone" server or whatever you want to call the portion that Torque provides.

Beyond the zone server, you'll need a good architecture for the rest of the system. Service oriented architectures have proven to be the best architecture you can use for any online game.

First, you'll need an efficient way to have your zones and other content downloaded. There are a whole lot of solutions here but I've found a REST API over HTTP(s) is awesome for downloading zone and other game data. For the huge artwork updates, let your clients do most of the work and use a type of Torrent. Login servers, AI servers, etc should fit in the architecture as services. Even the zone server should be written as a service.

Torque worked very well with the standard client / server architecture used for last generation games... got anything more appropriate for next generation games?

Indies need to innovate more, and if we continue being one or two generations behind the commercial guys, how can we expect to compete seriously?

#13 I had definitely intended to follow up more rapidly (I've got about 3 more pages of stuff to post), but I will definitely respond to some of your points:

--absolutely, client side caching of static data is extremely beneficial. Torque doesn't do this out of the box because it has a different design requirement: full modability. Torque servers assume that the client does not have the correct data, and therefore transmits every single datablock, and will even attempt to transmit missing resources (to a point). In a controlled release environment game such as an MMO, this design criteria isn't a good default at all--you should have some mechanism of course for verifying the client has the right data, but you shouldn't brute force push it every time they connect.

--I don't necessarily agree Chatting is insignificant--after all, two of your original blog's assumptions were 1kb/sec, and 10,000 simultaneous supported users. Within Torque especially, the stock example chat system (and that's all it is) gives chat higher priority in the update packets than actual updates (since chat is implemented as NetEvents, and updates are bitstream based pack/unpackUpdates). As Tony says, it works pretty well for 32 players (which of course is why it's the example implementation in the Starter.FPS kit!), but it definitely causes Torque to break down when you have exposure to 100's of other players that all are socializing to some extent or another.

The simple solution here is simply to move chat off to a different network connection, and have a chat server in your back end architecture. This allows cool stuff like cross-shard chat, irc integration if you so desire, or whatever you want to do.

--large grained scene changes: you're both right, and also glossing over the issue in some ways--more modern games do implement better large scale data transmissions than Torque does by default, and that is definitely an area that a Torque MMO will want to look at much more closely. However, unless you have absolutely no travel mechanisms in your game (no teleportation, summoning, city to city transportation mechanisms, no large distance mass transit systems), even with completely seamless loading you will still run into large grained scene changes, and need to handle them.

Background loading from a content server (as opposed to a play server) is of course the best solution, no challenge there at all from me. As you mention, you also want login servers, AI servers, support ticket management servers, load balancing architecture, and literally dozens of other back end systems--but nothing in Torque's architecture whatsoever precludes any of these.

I still reiterate my point in response to your comment:

Quote:
Torque worked very well with the standard client / server architecture used for last generation games... got anything more appropriate for next generation games?

My answer is Torque...under the hands of a smart and experienced development team. Nothing you've listed at all suggests Torque's networking doesn't apply.

#14 One last note to address the last statement you made to me:

Quote:but it does not in any way require a "significant re-write".

In order to get Torque to fit nicely into a service oriented architecture without treating it as "legacy" code... yes, the networking does require a significant rewrite.

I've been there... I've done that. Have you?

#15 Yes, I have. Thanks for asking. I've also provided commercial support to 3 different companies, and am acting as an adviser right now to a fourth that are not having/did not have any need for substantial re-writes at all.

To me at least, that suggests you may be taking a very unique position.

#16 Wow, you posted faster than I expected... I'll respond to a portion of it where I think you are incorrect.

Quote:
two of your original blog's assumptions were 1kb/sec, and 10,000 simultaneous supported users.

...

The simple solution here is simply to move chat off to a different network connection, and have a chat server in your back end architecture. This allows cool stuff like cross-shard chat, irc integration if you so desire, or whatever you want to do.

My simple solution was to make chatting be peer to peer... it requires zero bandwidth from the server, even if you're supporting a "kabillion" players.

#17 We're leapfrogging posts... sorry.

Quote:To me at least, that suggests you may be taking a very unique position.

I'm innovating and slaughtering sacred cows... as Indies we should all be doing this, but if my position is unique in that regard then it is indeed a sad day.

#18 Peer to peer anything out in the wild is a risky solution, one that many commercial companies aren't too excited about--and many knowledgeable customers are afraid of as well.

For example, let's look at peer to peer chat in an FPS. Peer to peer (by definition) requires exposing IP addresses to each player for those that they are going to chat with, or, if you're using a distributed system within your client systems, exposure to any neighbor nodes.

With that information available to a hacker/griefer , they now can easily push Denial of Service/Flood packets at will (ever try fighting in an FPS or even an RPG/PvP game like Shadowbane when you suddenly are not getting any packets to or from the update server?), and if you happen to make a mistake in your game and open up a vulnerability, you're handing the hackers a list of everyone that's vulnerable (at least a partial one).

I'm not in any way saying that it wouldn't work, or can't be implemented--but it has built in design risks that have to be considered.

#19

Quote:Yes, I have. Thanks for asking. I've also provided commercial support to 3 different companies, and am acting as an adviser right now to a fourth that are not having/did not have any need for substantial re-writes at all.

Going back for a second... just curious, but how many of those companies are or were Indies? I'd be pleasantly surprised if you said anything greater than zero.

#20 Sounds like a brush-off question, but how do you define indie?

My team was pure indie--hell, my team was myself (before I joined GG, I was building my own game) and a linux sys-admin.

Do you really mean beginner, as in never published a game before, or do you mean a development team working without external funding at all, or do you mean a small team with external funding but performing all work on their own?

Those of course are all biased from my point of view--what do you mean by indie?

To answer your question:

--1 was a small, unfunded, completely new to 3D game development team (my team)
--1 was a AAA MMO that realized their networking was terrible, and purchased TNL and integrated it (no, I can't release the company's name specifically, and no, they aren't still around. Was one of the many MMO's that couldn't generate a user base large enough to stay alive).
--1 was a non-gaming company under contract from the US Gov't.
--1 was a fully funded but small development team that uses Torque (vSide).
--1 is an unannounced MMO that is also using Torque, small but funded team.