Question, Server / Client seperation, can you ?

#21 So Pat, you are trying to tell me, that normal engines starts up a client (internally) and connects to a server (internally) to send object information (internally) ?

I got 4 engines on this computer alone (beside Torque3D) and none of them even remotely does that, nor need to.

Maybe its just me, who got no clue, i can live with that, im fairly new to Torque, but all the engines i have, some wich dont even contain network code, runs fine without that.

The idea that client simulation code is separable from server simulation code, which is separable from game-code is ludicrous. What data objects use to determine their position from frame-to-frame, and tick-to-tick needs to be the same on both the client, and the server or the results won't line up, and you will burn all your network time resolving the deltas between client-land and server-land. Furthermore, the code which processes that data (which must be the same) also needs to be 100% deterministic, across platforms (and in your case, programming languages). The data needs to be processed the same way on the server as it is on the client, otherwise the server is operating blind...it sends data to the client, and simply assumes the client is properly representing it.

We did a mmo emulator at a point (C#) (more as a test surface for some tech) and we didnt even make the client (wich was made in C++, for a Linux server), yet, our mmo server worked fine with it, all was right, char positions and everything, and we used MAX 5 KB/s pr player (at players highest use/large crowds).

Game networking is not the same as application networking...

Wow, would you look at that ... I guess "game networking" is the new fancy name for what used to be onUpdate() or Render() where all the things that needed to be sync became sync, with out networking :o

like I don't even understand how you think you are going to communicate network scope of objects. Do you plan on simply updating every object every tick and sending all that info down the pipe to the clients every network tick? All that info; even the stuff that didn't change? Do you need to communicate animation poses? Do you need to synchronize animations and particle effects? Do you need to smoothly interpolate data which is updated at varying frequencies? Do you need to communicate with the Torque object-type system to instantiate Torque objects on a client machine?

That should be handled in the ontick/onupdate event or what ever is used, and not in a client/server internal networking, its frikkin confusing tbh and un-nessesary, but perhaps your networking means something else then mine, im talking about the UDP server (i belive it is right?) the client starts when you start the game, and then it connects to it self, i have absolutely no idea why that is needed for a non networked game (or other networked server).

What if the end user has NO network card, no networking installed, and no network setup at all (no drivers or anything).

Then he cant play any games, because all games in the world apparenly require a internally run network to work or it wont be in sync.. (yes that was sarcastic, its was my turn).

But, i guess its just me, again, and im the donkey hole, because everything i give something out im apprently okay, but when i question some things in the engine, im just someone who knows absolutely nothing, even after 17 years working with computers/programming/ect (started when commodore 64 got out), but wow, by all means, i got no idea what im talking about, or, im just REALLY bad at explaining what it is i mean (wich isnt a surprice if it is).

#22
Bo, seriously...

What Pat is talking about is the data that is transmitted between the client and the server (wherever both are located; in-process, out-of-process, remote). For both to understand each other, they both must use the same protocol. In the case of Torque (and all other major game engines) this isn't a simple protocol defined in clear-text messages or somesuch but rather defined implicitly by the packUpdate and unpackUpdate methods in the C++ code using bitpacking and clever logic to detect what needs to be transmitted and what doesn't in order to maximize the use of available bandwidth.

So, this is basically an inseparable union. Both server and client use the same classes and thus share the same data structures.

Again, this isn't specific to Torque. In Unreal, the classes are defined in UnrealScript and the built-in replication code sync's up the instances between the client and the server (this picture is simplified of course).

All this has nothing to do with how the data is transmitted--over the wire or as in-memory streams, it doesn't matter. UDP or TCP, that's just the transport layer used by the networking code when the data goes over the wire. It doesn't tie into the game protocol itself in any way.

#23 Ok...been lurking about T3D forums since they came out without commenting much (I don't work on the Tech and Tools side of GG anymore), but since I used to teach Networking and Torque Architecture to a lot of our commercial customers, I figured I should step in here and throw out some thoughts:

First and foremost, to answer the original question, re-stated to be accurate and much less confusing:

Can Torque be used in a "client only" mode, with a different application performing the world simulation and Torque simply acting as input/output?

The short answer to that question is "yes, not only can it be done, but it has been done!", as Tom alluded to. I myself spent 3 months working with L3 Communications Internal R&D group and the Distributed Interaction Simulation Protocol (DIS Wiki) to link real world military training simulators up to Torque, and have them cross-network their simulation objects. In other words, we hooked up a UH-60 helicopter simulator (pilot and gunner stations), a Predator UAV simulator, a Command and Control simulator, and a 3 person "Seal Team Simulator" (Torque server + 3 Torque clients) into one composite simulation.

Eric Priesz has also done extensive work in this area with NASA by taking the output of their hyper-accurate billion dollar simulators and used Torque to visualize that data (although their work was only in a single direction--super computer sim to Torque).

Now, here's why you're getting such a negative response from everyone:

You are approaching the problem from a very focused perspective (networking), without fully understanding why a vast majority of game engines utilize the authoritative server/replicated client model. That's fully understandable--when I taught the Torque Boot Camps, more than 70% of the 3 day class was centered around the client/server model and Torque Networking, both of which are integral to how the vast majority of game engines work (obviously, different networking implementations for most engines).

The authoritative server/replicated client model evolved for many reasons, but the key ones are:

--maintain a synchronized world state across latency driven distributed environments.
--optimize user experience across highly latent/untrustworthy communications media (think 28.8k modems, and now Internet Backbone across global distances)
--minimize the ability for players to "cheat" by intentionally de-synchronizing world state between their local and other player's world states.

Torque accomplishes these strategies by a combination of a server world state that is always "boss" (de-synchronization issues are resolved by the authoritative server forcing the world state on the clients), a client side replication of that world state that is non-authoritative (but immediately reactive to local input), and a networking implementation that is optimized to keep these two parallel world states synchronized.

So, back to your original question:

Can you use Torque as a client only, and use a different application as your world state simulator, and network the two together?

The short answer is yes, you can--but you are throwing away resource-decades of research, design, and implementation that accomplishes the exact same task, so you can "save" a month or two of study.

I can just about guarantee three things if you choose to go the route you are asking:

--a year from now when you finally get it working in a reasonable manner, you will have solved a complex series of tasks that are already solved by the existing Torque server/client model.

--if you do a rational post-mortem of your team's work, you will have found that you would have accomplished the same net effect in less than 1/3 of the time by porting your existing C# logic/implementation to c++, inside the Torque architecture (assuming you use Torque best practices instead of just jamming everything in).

--you will not be completely happy with your work, and will wish that you had the user experience of Torque's already implemented and experience/bandwidth optimized implementations.

I'd be very happy to continue answering specific questions on the topic here in the thread as my work duties permit, but unfortunately I don't have time to brain dump the entire Torque Bootcamp into the thread here.

I'd also make one other suggestion: there is definitely a solution of some form for you--even if you ultimately decide to go with your original plan, it can be done. My suggestion however is that you should really balance the experienced opinions of the people in the forums here--for the record, Pat Wilson for example was one the key resources I used to learn how the Torque simulation and networking models actually work--the guy knows his stuff ;) Tom Spilman, Matt Fairfax, Rene and the rest of the associates--all of these guys really know their stuff in and out of Torque, and aren't just being fan boys when their initial reaction is "why in the world would you want to".

It's easy to get defensive, and there isn't a need to be--but I'd suggest that if you want to succeed, there is a need to take a step back and address the root reasons why it will be a lot more work to go the route you are thinking about than to use one of Torque's core strengths in the manner it was designed.

#24
Nothing like learning from someone who really knows. Thanks for this great post, Stephen (and for stepping out of the shadows :)

#25 I figured that I might get questions (from Bo, or others) regarding how we linked up military simulators with Torque using DIS, and since the design would be very much in parallel to what Bo would have to do if he sticks with his original plan, I figured it made sense to provide an outline.

Please note that this work was completed under contract, and that I cannot go into implementation details, nor can I provide example code or a resource.

DIS explanation (top level)

DIS is a text based protocol that is designed to allow different simulators to "speak the same information", and has several built in design constraints that make it work:

--no simulator is aware of other simulators. All information is broadcast via multicast UDP packets. This can be done over an open network, but every single real world implementation of DIS that I am aware of has been performed on closed, extremely high performance, absolutely trusted nets.

--every simulator is authoritative for world state changes they broadcast. Since these are trusted agents, if the F-16 simulator says it kills a tank, the tank simulator can't say "no, you didn't!"--the F-16 simulator is authoritative since it simulated and broadcast the Protocol Data Unit (PDU) that describes the world state change event. This has a side effect that if the tank simulator is greatly desynchronized, the tank could be blown up by an F-16 that isn't even in it's local world state.

--there is no mechanism for re-synchronization of world state, or late joining of a simulator to the composite world state. Effectively, the protocol is built around the concept of zero packet loss (which is not realistic in the real world). Simulators participating in the composite simulation are responsible for correct implementation of 1st through 3rd order interpolations based on information received within the PDU's, but they must receive the PDU's.

What We Did

--First and foremost, due to the very reasons Bo is getting such a negative response to the idea of using a different world state simulator for an authoritative source, I "cheated", and only connected a Torque Authoritative Server to the DIS composite simulation. This allowed me to completely isolate the clients behind the authoritative server, and the fact that they were contained within the composite simulation was completely transparent to the Torque clients.

--since the clients were isolated via the Torque Server, all I had left to do was to handle the changes necessary to participate in the DIS world space. These included:

--create a new series of classes (we did one for wheeled vehicle, npc player, and flying vehicle. we attempted to tie in the BraveTree Tank pack as a tracked vehicle, but due to lack of scope, we wound up scrapping that for the convention). These classes were responsible for reacting properly within the Torque simulation based on information provided by DIS via PDU's.

In effect, we ripped out 80% of the code in processTick() for these classes, and replaced it with algorithms to support PDU supplied interpolation information.

--using a string based protocol to receive and transmit PDU's. Since L-3 Communications had a highly experienced engineer very familiar with Java and UDP protocols, we had a Java App that listened to the UDP multicast net, and "pick off" PDU messages that the Torque simulation needed to know about, and transmit them across a TCP/IP socket to the Torque Server. We parsed the PDU's Torque side via standard (script based no less!) getWord, and generated "events" such as CreateNewDISObject, UpdateDISObject, DestroyDISObject, etc.

We then implemented routines server side to perform the appropriate Torque simulation steps to replicate the world state changes--i.e. for a CreateNewDISObject (new F-16, for example) we would:

--create a new Torque Object, and populate it with appropriate data (datablock, initial position/velocity, etc.)
--set this as Ghostable
--set it's interpolation parameters to be based on the PDU's interpolation order (mostly dead reckoning)

Since this was now a valid, Torque networked object, any clients participating in the Torque side of the composite simulation would treat it just like any other networked object.

"Seal Team Simulator"

Finally, we had to capture world state changes based on player inputs--since the purpose of including Torque into the composite DIS simulation was to demonstrate the ability of a game engine to participate in a DIS simulation, player actions had to have observable composite simulation impact--in other words, the Torque server had to generate the appropriate PDU's so that the other simulators could react to the Torque player's actions.

This part wasn't really as hard as it sounds--we mapped standard Torque world change events (movement, firing a weapon, mounting a vehicle, etc.) to their corresponding PDU's, and then communicated the data needed to populate the PDU to the Java Bridge. The Java Bridge then generated a fully formed PDU of the appropriate type, and broadcast it to the other participants in the composite simulation.

#26 Bo, the Torque MMO Kit uses Python (instead of C#) to run most of the logic and clustering aspects of the MMO servers. It then uses the networking code in Torque for single servers. A bit of the network was stripped down to make it friendly for MMOs since you don't need the same kind of speed and tick rate that you need with a twitch FPS style game.

That being said, it is still tightly driven into the engine aspect as most of the world position updates are handled via the torque servers and most of the RPG logic is handled via the Python servers.

Also, if they ripped the Networking out as you have seen by the other threads 200 people would complain that it was missing or they had to install it. Cannot win sometimes.

#27 I did a little bit of searching on TDN, and found some pretty good reference docs. None cover every single question you can ask, but they are pretty good information. Unfortunately, I'm not sure if everyone in this forum can access each article, but if you send an email to David Montgomery-Blake, I'm reasonably sure he should be able to hook you up:

Torque Server and Client Model--Overview

I wrote the following article as supporting information for teaching the Boot camps. It's not complete at all, but does drill down into the specific challenges and strategies that help explain why Torque's model works the way it does:

Torque Networking

Reference document used to create the page above. Goes into more detail, and less: OpenTnL Introduction to Networking Concepts. Note that this document does not address the need for an authoritative server model explicitly, but it is implicit as a design assumption.

#28 Take a look at Source. Source is a pretty standard commercial game engine. I'm not a big fan of the Unreal product lineup, though I know it's also considered a modern standard.

Source does exactly the same thing as Torque. It loads up a client and server on one box to handle singleplayer. Why? Because the engine was built for multiplayer.

I'll admit I didn't read this whole thread. This is like the 25th time I've responded to this question during my time working with Torque.

So, what's the problem with this setup? The only issue it adds is that you have to logically understand that all dynamic data the client needs to display/predict based on needs to be synced to the client. There are two very simple functions for this in every object type. Since this is singleplayer, and the client->server connection is simulated, you can crank up the packet size and rate to keep things quick.

If you're interested in sending data to your own secondary server for, say, MMO cross-server syncing purposes, that's easy enough. I've written a mySQL interface into Torque before that was pretty effective.

What you want to do in this case is use the Torque interface for your primary (zone) servers, where its built interpolation and prediction will keep things smooth at a low level. Use your secondary world servers to sync up switches between Torque servers.

I can't think of what else you'd need to do that this really would complicate.

#29 Oh MY! Stephen is back!

You should write a book man, the force has always been strong in you.

#30 Henry is of course correct, but I think that what Bo is trying to do here is to use C# to control his world state, and simply have Torque render that world state and accept input (which would in turn change the world state, so would need an uplink to the C# application to pass that information along).

Can it be done that way? Yes, as I've outlined above. Should it be done that way, given what little information we have about his project? Most of us are saying "absolutely not"--at least not without an in-depth study of the way synchronization, authoritative world state management, and the other challenges of networked simulations should be approached, and why Torque implements them the way it does.

And the thing is--if Bo does all of that I describe, he'll almost certainly come to the conclusion that it's less work to simply implement his server logic within the existing Torque best practices and architecture.

One final thought, and I don't want to turn this into a transport layer protocol war, but TCP is generally considered a bad protocol from a user experience level when things break down. Sure, when everything is 100% working properly (no packet loss, basically), then TCP and UDP achieve the same result--but as soon as you have players with network issues of any sort, they will have packet loss, which will trigger the need to re-deliver data.

The core reason why UDP is normally considered the better choice here is that TCP has, by definition, complete control of data retransmission at the protocol layer, while UDP does not. It sounds like that is a win for the developer (gee, I don't have to handle packet loss and packet re-transmission--TCP does it for me!), but the problem here is that TCP can only re-transmit the data it originally tried to send.

Since UDP doesn't have an implementation for packet loss/re-transmit, it allows you to implement those features at the application layer (which is what Torque does). This in turn gives you much finer control over the data that is retransmitted--for example, in Torque, if a client doesn't acknowledge that it received an object's position update in a reasonable amount of time, the Torque server will transmit a new position update, taking into account any changes that have happened to the world state since the original update was transmitted. This is a Very Good Thing(tm).

#31 Henrys suggestion sounds pretty much like this nice little series on the IBM website which I would recommend to anyone to read if he/she is interested in MMO development: www.ibm.com/developerworks/library/ar-powerup1/
Its TGEA not T3D but the networking hasn't changed since TGE I think

Wireing in a distinct server backend and network technology potentially can work as well. I would actually in this case benefit from what is present and start with seeing how far you can go by redirecting the packed / unpacked stream to your own network solution.


Thanks for all those informations and considerations Stephen :)

#32 Damn Stephen... dropping knowledge on us fools. :)

#33

Quote:
Damn Stephen... dropping knowledge on us fools. :)

Bah--I'm just regurgitating knowledge from years ago--you guys are the ones making such amazing strides with the new tech!

I have to admit though I do miss the "applying Torque to things you never thought possible" aspect of the job I used to have--InstantAction work is extremely fun and interesting, but on the "omg that is just freaking amazing" level, the high point of my GG career was watching 3 Torque players mount a UH-60 helicopter and get evacuated from a hot zone--and then glancing over across the booth and seeing the UH-60 pilot pull back on the stick to climb away from the rooftop, while the 50 cal machine gunner sitting next to him laid down suppressing fire--then glancing back to see the truck the gunner was firing at explode on the Torque screen.

You can do anything with Torque. That project alone proved it to me...you just have to be willing to dig down and understand the problems and challenges, and do what needs to be done.

#34 Stephen.. thoroughly enjoyed reading your posts.
Thank-you so much for the insite :)

#35 @Stephen

Thanks, for being awesome, i was expecting waking up to alot of flaming :P

After reading it all (yes, i actually did that :D) i think i understand a bit clearer.

Because you took your time explaining so awesome, im gonna drop some info on how we usually do the mmo network implementation and syncronization (for ortho/iso games as that is our current game project).

-------
Its all run over TCP, client maps contains all static objects (anything that does not move or get deleted).

If a player logs in, a single packet is send to client with player data, location, hp, mp, items.

All players in view (all players with that mapid is in a array so easy to locate who is where) get a similar, but smaller packet (alot of information is not needed for them, like other players items, stats, hp, mp, as they arent in party ect).

This is done in 1 small packet.

Now, everyone in view knows where everyone is, so when 1 "click" on ground somewhere, we need to move him, from point A to B, so all who currently see him, and new person who gets into view see's him, and at the correct spot.

This is once again done in 1 simple packet, as this system calculate movement from point A to B internally inside the client, and all clients use the same system, so path will always be the same.

Packet would look like

[1 byte opcode][2 byte packetsize][8 byte loc A][8 byte loc B][8 byte current location (for new players)][4 byte objectId (playerid)]

That way, the clients (new and old) will always know start location (A) end location (B), and current location for new clients (calculate point A, B like all others, but move char to (C) location and run from there.

As we already know the speed of character, movement can be done inside each client, and should be in sync (all our tests, other project they where).

Ofcouse, during movement, should he click somewhere else, it will send a cancel movement packet (small 7 byte) and/or a new movement packet.

Thats everything for movement, 31 byte only (excample) when changing location.
-------

Anyways, thats how our network structure is normally here :D

Again, thank you Stephen, you gave me alot of idea's on how to possible proceed with this, and i think the problem -> for me <- is, you guys have already build in a totally different kind of system that runs continiously "inside" every client, and for me the logic on that is lost, however, if you plan on building a Torque network driven network game, i can see why its build in :)

For someone who dont plan on using Torque networking (at all or halfway), its confusing you are using the networking internally to predict and keep everything together, as for me atleast, im used to that is done inside a onRender() or onUpdate() command and not over network.

But as you said Stephen, it can be done, and you came with some good exsamples on how to possibly do it, so now i can be a little more rested without team killing me (and im sure they would have, im the one convincing them to go with Torque xD).

#36 Bo,
How do you resolve conflicts if everything is calculated on the client?

Things may be in sync when you control the clients, but if I control one, maybe they won't be ;)

Right now Im using a 3G card to connect to the internet, packets often get dropped, and can regularly arrive up to 1 minute late. This causes network stuttering in most games, but because of the techniques Stephen has described above, nothing gets too out of sync.

In the system you describe above, the use of TCP and the lack of an authoritative server means that everything will fall down in a heap.

#37 @Matt

Its called "catch up" (or atleast thats what i call it), if you loose packets, and a player moved (eg) and you didnt receive the packet, you will just receive the new location, this will prevent you from moving if server does not receive your request move packet, and if you didnt receive a player movement packet he will not move, should you then receive one, it might look like the player is "lagging to" or "teleporting" to his current location and move.

Basicly , your bad connection will only effect you, and not the others.

Then again, IF your network is THAT bad, i would probably suggest you not to play, as loosing XP for lag is bad thing, but if you really loose tons of packets, you might consider getting some other internet.

There is plenty of MMO and multiplayer games using the same technique as we plan on, it works, we already testet our libs/networking plenty (and with "faked" packet loss, delay ect).

I might be bad at descripting it though, but as i said, isnt our first dance :), but its a good question, one we worked long on aswell when making our network systems - for PVP reason ect (thats why we want to use it aswell).

#38 Bo,

The best way to achieve what you described is create a custom player class that combines trimmed down aspects of Player and AIPlayer. Implement your pathfinding in it and instead of sending velocities and whatnot from the server, you send move destinations and let the clients move the characters there (actually that's how many click-based MMOs operate).

You're willing replacing the entire networking system, when a couple custom classes, database connection code, and some well-planned scripting would do.

Quote:Its called "catch up" (or atleast thats what i call it), if you loose packets, and a player moved (eg) and you didnt receive the packet, you will just receive the new location, this will prevent you from moving if server does not receive your request move packet, and if you didnt receive a player movement packet he will not move, should you then receive one, it might look like the player is "lagging to" or "teleporting" to his current location and move.

Not if you use TCP. The TCP layer will keep trying to re-send the lost packet until the client confirms it received it. You have no solid way to determine the packet was lost. You need your own protocol layer made on top of UDP (which Torque does).

#39 Bo,
but how do you suppress the underlying OS protocol stack's desire to reliably transmit every packet for an OS defined timeout (as it must do to comply with the TCP spec). What you need to do is send out the packets and control the retransmission yourself, which needs UDP.

I haven't looked at the networking stacks of many games (Professionally I work in another area of software that has big requirements on networking) but those I have seen use UDP for data and TCP for control (joining, leaving, clustering etc) ( the systems I work on do the same ).

Network latency is not necessarily 2 way. I can be quite happily be sending 'move' packets for the server without receiving them from the client.

Or I could modify my client to ignore (or slow down) enemy movement. Then I just need to move in to range and transmit "shoot" packets. Even easier I could just send the shoot packets and 'I killed them' packets to the server as soon as I receive the data regarding other clients. That way I dont even need to bother with all that difficult 'playing' stuff.

#40

Quote:
Then again, IF your network is THAT bad, i would probably suggest you not to play, as loosing XP for lag is bad thing, but if you really loose tons of packets, you might consider getting some other internet.

I have fast internet at home, I was just using 3G/mobile broadband as an example of an unpredictable networking layer, to demonstrate the point that what Torque networking does is a Good Thing!(tm).

Quote:
There is plenty of MMO and multiplayer games using the same technique as we plan on, it works, we already testet our libs/networking plenty (and with "faked" packet loss, delay ect).

Out of interest, how were you faking the packet loss? Its a pretty difficult thing to do, the best way I have found is to unplug network cables, but then its not fake.