V12 Technology and Massively Multiplayer Online Game

#1 From what Tribes 2 info I've seen in the media, it seems they support 64 players per server. While I'm sure server operators can set the player maximum to whatever value they want, 64 is the supported maximum. Going above it will probably overburden the average game server.

To make a truly massively multiplayer experience using the V12 engine, you will probably need to explore adding some sort of multiserver load balancing... Definately not a trivial task.

#2 Much noted, and appreciated feedback, I guess my main question is, will my programmers be able to go into the engine code itself and either modify or replace the current networking code fairly easily.

I assume when the faq says you get the engine code, that doesnt mean just the API's but the full source. Am I correct in assuming this? and if so any other thoughts on load balancing :)

Thanks again for the feedback.

#3 You get the source, so your programmers have access to and can change anything.

Jeff GG

#4 So would the above networking capabilties be possible using the v12 engine?

thanks

Adrian

#5 You do get the source so the sky is the limit. So how massive is your massive? :) Tribes2 has some pretty tough requirements for accurately tracking player actions, physics interactions and having very little lag. I am not sure what kind of game you are working on but if it was acceptable to give up a bit of accuracy I bet you could use a single server model and get upwards of 200 or more people. If you're looking at massive 10,000 people? ...you are looking at a load balancing server farm. Much of the fundamentals in the way that object information is transmitted would be the same but there would be a lot of work to do on server-server communication and database management.

--Rick

#6 Thats what I was looking for :)

And for our demo, that we are currently working on, using the Lithtech SDK, we were hoping to scale the demo up to 100 players for initial testing and attracting a publisher.

For the final version though, yes we are looking at a much more complex network code that would be able to handle upwards of 10000+ players, and thats where my concern was.

Right now with lithtech, since we only have the SDK, with the code API's we are limited in what we can do, and until we find a publisher we would be unable to obtain the source for lithtech. But it sounds like v12 may be a much bettter route for us to go, and I am excited at the prospect.

Who at GarageGames do I contact for more info? Myself and our coders have alot of questions :)

Adrian Wright
MGO.NETwork
Max Gaming Technologies, LLC.
www.maxgaming.net
www.mgonetwork.com
www.critical-subterfuge.com

#7 Adrian,

I've been researching Massively Multiplayer Networking and the problems related to it (server farms, etc). After coming across the GDC article on the Tribes networking model, I was hooked. It integrated everything I needed and had some great potential for hooks like trickle-streaming updates, new content, etc.. I was impressed.

I threw away my old project and implemented a Tribes-like model including the Connection, Stream, Event, and Ghost managers as described in their documentation. So, from experience, I can say you'd have no problem using the Tribes model for an MMORPG.

As for server farms, etc. I would be interested in corresponding with you (or anyone else) who is interested in developing these technologies.
--Bryan

#8 John Ratcliff's speech (at the GDC2001 I think) on the Planetside engine would be a useful reference for developing an MMORPG.

#9 Could clustering technologies be useful for Large Massive Online Games?

There is after-all things like Beowulf for Linux. And OSCAR.

Not only that there are some of the inbuilt technologies available in Windows. Especially the Windows 2K Server Edition.

Just a thought that the tecnologies applicable to building supercomputing clusters could be used to build the game server. Such as MPI (Message Passing Interface).

Anyway its certainly something I want to look into to see how it could be used.

#10 Yes if writing correctly it would be possible in theory to use clustering. I have some experience with clustering Linux servers, and it works rather well.

Matter of fact a word to the wise, if you are designing a MMORPG, one the server side I would recommend you develop the servers using Linux rather then a windows enviroment, this will give ou added stability, and faster processing in my experience.

For example running on the same machine, I had a dedicated Counter-strike server, on windows it used 15 percent of the cpu at all times, on linux it ran between 5-10 percent depending on load.

A word of caution, if you are new to linux, find someone with experience to help you setup your test server for optimal performance, if anyone needs any help they can contact me via email.

#11 Dont want to be negative or anything, but supporting 1000 or so players is going to require a lot more than a linux cluster.

Most of the commercial enterprises will be using heavy load balanced clusters with distributed databases (normally oracle), world management (i.e. zoning or other load management).

I'm not saying that its impossible, but I would just say that if you want to do MMOG, then think realistically about what you are trying to do. You cant just take the tribes code and scale it up.

The biggest issue is that of distrubuted database speed and integrity. Frankly the comms coming between the client and the game is nothing compared to the comms occurring between the different backend servers.

The cost of the oracle solution and hardware alone should pretty much rule out MMOG with a central server system for any smaller teams or companies.

Think in terms of distributed servers, more like a "team" of servers is a much more scalable bet, of course that means that you havent got a single point to charge players and opens up issues of security.

Phil.

#12 Well not sure you are in tune with the "linux" availablilty of enterprise solutions, if you would like to know more you can visit the following companies website, who support enterprise platforms (including hardware and software).

www.redhat.com
www.ibm.com
www.oracle.com
www.informix.com
www.dell.com
www.valinux.com

Now you may notice the companies like IBM, Oracle, Informix and Dell, I guess these ppl only make crappy solutions for Linux? and there are many more developers out there with Enterprise platforms, but these are the ones I have worked with, and in every case I used them to switch from a NT enviroment to a Linux enviroment, with markable increases in performance, and Stability.

This post is no where near as long and involved as my original post but I accidently pressed back and lost the post.

Hope for those of you looking to develop a server solution with linux find the above companies can help you out.

I will warn you that developing a enterprise system using linux is still expensive, though no where near as costly as using Windows or Unix.

#13 Here are some more specific resources for linux both for enterprise solutions and clustering technology. I am a bit cranked up on this subject if you cant tell, but I want to make sure people know what a flexible and capable platform Linux can be. Beleive me I start in a Windows enviroment, and I still on a daily basis deal with Windows, AS/400, Linux, and Unix flavors, all working together even.Linux for IBM

http://www-4.ibm.com/software/is/mp/linux/software/index.html

Redhat Linux Enterprise

http://www.redhat.com/products/enterprise.html
http://www.redhat.com/products/software/linux/haserver/

Oracle for Linux

http://technet.oracle.com/tech/linux/

Informix

http://www.informix.com/informix/products/pa/lnxannc.html

mySAP

http://www.sap.com/solutions/technology/

Linux Clustering (note: some of these are out of date but gives you a sense of what can be done)

http://hp-linux.cern.ch/
http://www.cacr.caltech.edu/research/beowulf/
http://www-hermes.desy.de/ww/cluster.html
http://www.ibiblio.org/pub/Linux/ALPHA/linux-ha/High-Availability-HOWTO.html
http://www.zdnet.com/enterprise/stories/linux/0%2C12249%2C2644808%2C00.html

#14 MMORPG Server architecture and platform choice really depends on how many people you want to support.

For a few thousand people, a linux cluster won't be a problem.

For a few ten or hundred thousand people, a linux cluster would fall over and die.

At the upper end of the scale, you're going to be stuck with Sun or HP to get any reasonable sort of performance and scalability. Free/OpenBSD on Intel would scale better than linux, but I think the real problem is CPU architecture.

It pains me to say this, because I love linux, but I'm just being realistic here. Professional experience tends to show you what works and what doesn't.

I do, however, plan on doing some tests in a few months to see just how far Linux and FreeBSD on Intel will scale.

#15 Agree and disagree :) I also am speaking from experience, and as you know or may not, IBM new server platform is much more advanced then a "Intel" server structure, and I beleive that combined with linux could support 10,000 players plus in the world.

On the other hand, where I agree with you, If you have the money a nice farm of Solaris machines would be the best bet for a hugh MMORPG.

#16 Jeremy,

I'm not sure I agree with what you're saying. Pick any (relatively stable) OS and a high-end machine. Plug in 5 ethernet cards and connect them up to a nice T3+.

Now people start logging on and using up bandwidth 2000-10,000 bytes per client per second.

At some point, you go beyond the current machine's capabilities, so you add another machine. And another and another... Perhaps Win2K can only handle 400 clients, while Linux might handle 800.

After adding enough machines and enough T3, OC3, OC-192s, etc.. you'll have capacity for the number of players you need (ISP issues asside...).

This requires some back-end load balancing to ensure that each machine only has to serve a part of the world, etc.. This is the major work of the whole system (in my view).

Not sure how a beowulf cluster or Sun cluster changes the basic facts of networking. You're still just plugging in wires, whether the systems are the most efficient for the task at hand, that's a different question.

I don't see why Win2K Server running on 50 machines can't support a world just as well as Solaris running on only 25. The efficiency is higher, but the results are the same (and who knows, 50 Win2K macines might even cost less!).

Please enlighten me, I'm interested in learning more about these issues.
--Bryan

#17 I would rather have 25 Solaris machines then 50 Win2k machines any day. There are many reasons. Solaris is more relyable. 25 servers is easier to maintain then 50. I hate Microsoft. Being the realistic type, I'd go with a server farm of as many Linux machines I could scrap together and do some fancy work on balancing loads for areas of the world. Software I write is free, hardware is expensive.

#18 There's really a couple issues involved here...network throughput, system architecture (as in hardware architecture) and server design (as in the server code that handles your backend)

Also, remember we're talking about very large games here, not the 25-100 player games that can be handled on a single server.

First, network throughput: We can for now assume that we have a pipe of unlimited size (say multiple OC192s) and take that part out of the equation. We can also remove the LAN throughput problem from the equation, and assume everyone has 10GBit links to the servers from the border (border being the OC192's previously mentioned) ... so now we're down to the servers themselves. Now, I don't know how many of you have any real experience with various platforms and very high traffic, but I've done tests at up to OC48 speeds. With Intel based systems, especially linux, you get serious problems with interrupts from the network card. Much more than about 150-200mbit/sec into the box as a whole using multiple network cards, and your cards interrupt your CPU to death and the box keels over. Supposedly the new 2.4 kernel manages it's interrupts better, but I haven't gotten a chance to test it. I'm trying to get my hands on one of Lucent's oc48 cards, but they're not cooperating ;) Due to the difference in hardware architecture, sparc based systems don't have these massive interrupt problems, and I've pushed 8 gigabit/s through E420s with no problem.

Aside from interrupt issues, you also have a problem with core CPU architecture. Without going into too much detail, the Sparc architecture is significantly faster at lower clock speeds. Whether or not this can be made up by the higher clock speeds of the latest Intel and AMD chipsets is something I am planning on testing sometime in the next few months. The last tests I did were with 700mhz PIII's vs UltraSparc 400mhz, and the US ran rings around the PIIIs.

Server design itself tends to depend highly on your application, but it will affect your hardware infrastructure significantly for reasons other people have already mentioned (DB's, load balancing, etc)

Hope this is enough detail, I'm in a rush at work right now ;)

J

#19 Oh, one more thing, I'm also going to eval how Beowulf and that other clustering product from Israel (I forget the name offhand) compare to other methods of cluster design ... I'll share my results when I have them ;)

#20 Jeremy,

Thanks, that's the kind of information I'm looking for. So the Windows PCs become bogged down by interrupt overhead trying to field all these network requests. While the Sparcs are built to limit this overhead.

I assume then, that a Linux box on Intel/AMD hardware would essentially have the same problems as a Windows box, as the hardware/interrupts are the same?

And this is why you feel the Beowulf would not be able to handle a game of 10K+ users?

Adrian, any comments on hardware architecture related to high-end Linux servers which aleviate this issue?

Also, is the programming model vastly different between Solaris/Linux/Windows networking? If I wanted to write a distributed server app for a project and wanted it to be portable - but also efficent, what thread/socket/IO model should I use? (Currently I'm using non-blocking sockets on a single thread with Win2K Server for both the server and client apps)

--Bryan