Networking Physics
by Tom Spilman · in Torque 3D Professional · 12/06/2009 (9:31 pm) · 17 replies
Hey everyone.
So first note this IS NOT a feature announcement... its only some experiments on my part. This all may or may not be in Torque 3D 1.1 beta.
Also this is only one of 2 other methods i hope to support for physics networking including client side only physics for debris and server side only physics with ghosted positional updates to the client.
This is an experiment in networking the same physics simulation between client and server. The black wireframe is the collision hull for the server object.
My reasoning for trying to simulate on the client as well as the server is to provide higher quality prediction/extrapolation to fill in the gaps when you have more physics objects than the network bandwidth will allow.
As you can see my current implementation isn't dealing very well with lots of contacts or stacking. You can see stacks trying to correct themselves several times over until they finally stick. I think the problem here is that i should be smoothing the jump in rendering when a correction occurs instead of smoothing the physics correction itself.
Also a note on determinisim. Right now i see very different results between the client and server without corrections. I may still have some other bugs, but i did some investigation and found in the forums that Bullet is deterministic only if...
1. You have the same amount of objects.
2. The objects are added to the simulation in the same order.
This is because a different count or order will change the way contacts are resolved and potentially result in the object in a different position. From what i read it can even be true if the objects are not in contact with each other. Not sure if this is the same in PhysX and/or Havock.
How Torque's ghosting works objects will appear on the client based on a priority which has nothing to do with their processing order in the physics engine. So any time you blow up a wall and new passive physics bodies are added to the simulation, a new player or vehicle appears, etc. Seems like even worse if you had an open world game where new physics elements are added/remove all the time.
Anyway... i'm still working on Bullet for 1.1 beta and networking.
So first note this IS NOT a feature announcement... its only some experiments on my part. This all may or may not be in Torque 3D 1.1 beta.
Also this is only one of 2 other methods i hope to support for physics networking including client side only physics for debris and server side only physics with ghosted positional updates to the client.
This is an experiment in networking the same physics simulation between client and server. The black wireframe is the collision hull for the server object.
My reasoning for trying to simulate on the client as well as the server is to provide higher quality prediction/extrapolation to fill in the gaps when you have more physics objects than the network bandwidth will allow.
As you can see my current implementation isn't dealing very well with lots of contacts or stacking. You can see stacks trying to correct themselves several times over until they finally stick. I think the problem here is that i should be smoothing the jump in rendering when a correction occurs instead of smoothing the physics correction itself.
Also a note on determinisim. Right now i see very different results between the client and server without corrections. I may still have some other bugs, but i did some investigation and found in the forums that Bullet is deterministic only if...
1. You have the same amount of objects.
2. The objects are added to the simulation in the same order.
This is because a different count or order will change the way contacts are resolved and potentially result in the object in a different position. From what i read it can even be true if the objects are not in contact with each other. Not sure if this is the same in PhysX and/or Havock.
How Torque's ghosting works objects will appear on the client based on a priority which has nothing to do with their processing order in the physics engine. So any time you blow up a wall and new passive physics bodies are added to the simulation, a new player or vehicle appears, etc. Seems like even worse if you had an open world game where new physics elements are added/remove all the time.
Anyway... i'm still working on Bullet for 1.1 beta and networking.
About the author
Tom is a programmer and co-owner of Sickhead Games, LLC.
#2
12/06/2009 (10:09 pm)
I really hope that networking physics could be the next kicking ass feature for future T3D release. It's awesome for such a hard work!
#3
Yea... it wasn't until i wrote that sentence that i realized i should be smoothing the rendering and not the simulation correction. Sometimes you gotta write it out for the bug to jump out. ;)
As far as physics islands my thinking has been to adjust the ghosting prioritization process. I was thinking of maybe adding a 'group' in addition to the priority so that groups ways tend to be updated together. I was gonna look at the "hifi" stuff too as i think it did something like that. Worst case i can try doing NetEvents to update groups, but that seems just as tricky to get working within the system.
I've gone thru Glenn's stuff alot. He has some great stuff there, but his example is an easier case... one player controlled physics object. At his GDC 2009 talk he demonstrated multiple bodies networking, but he was using his "cheat" method where the client is allowed to be athoritative.
12/06/2009 (10:10 pm)
@BenYea... it wasn't until i wrote that sentence that i realized i should be smoothing the rendering and not the simulation correction. Sometimes you gotta write it out for the bug to jump out. ;)
As far as physics islands my thinking has been to adjust the ghosting prioritization process. I was thinking of maybe adding a 'group' in addition to the priority so that groups ways tend to be updated together. I was gonna look at the "hifi" stuff too as i think it did something like that. Worst case i can try doing NetEvents to update groups, but that seems just as tricky to get working within the system.
I've gone thru Glenn's stuff alot. He has some great stuff there, but his example is an easier case... one player controlled physics object. At his GDC 2009 talk he demonstrated multiple bodies networking, but he was using his "cheat" method where the client is allowed to be athoritative.
#4
He feels that determinisim isn't generally achievable in networked physics and the focus should be on sending more frequent updates and smoothing the corrections.
The good news is i think Torque's networking is architeched well for this sort of focus. Plus if you throw out determinisim as a requirement you can use lossy compression techniques like dropping some precision when sending physics state updates.
And all this prepares us for the future as physics in multiplayer will only be solved thru improvements to network latency and not trying to "mind read" the server.
@Ben - So any ideas on how to get islands/grouping into NetObject::getUpdatePriority()?
12/07/2009 (12:29 am)
I sent Glenn an email and he confirmed the determinism issues.He feels that determinisim isn't generally achievable in networked physics and the focus should be on sending more frequent updates and smoothing the corrections.
The good news is i think Torque's networking is architeched well for this sort of focus. Plus if you throw out determinisim as a requirement you can use lossy compression techniques like dropping some precision when sending physics state updates.
And all this prepares us for the future as physics in multiplayer will only be solved thru improvements to network latency and not trying to "mind read" the server.
@Ben - So any ideas on how to get islands/grouping into NetObject::getUpdatePriority()?
#5
I think there are two important issues, one is eventual consistency and the other is agreement with server. For a vehicle you are on you want more server-agreement. For physics props that don't really affect gameplay, eventual consistency is fine. In some areas continuous agreement is achievable and necessary, in others it's too costly. It's all what you're willing to pay. For your rocks example, it's more important it look believable and less important it reflect server state. For the player, consistency with server overrides even simulation consistency (ie it's better for player to smoothly walk through a wall if that's what the server thinks should happen, even though that violates the client's physical constraints).
Good to hear you are in touch with Glenn, he's a good guy and very passionate (and knowledgeable) about game networking. His peered physics stuff is great for fixed physics props.
12/07/2009 (1:54 am)
Choose island with highest priority item in it, send all items in that island. Cap island size to some max, or send only highest priority items in island. Or you could infer priority by dependency graph (ie if a high priority object is on top of a stack, work down the support graph to a static object).I think there are two important issues, one is eventual consistency and the other is agreement with server. For a vehicle you are on you want more server-agreement. For physics props that don't really affect gameplay, eventual consistency is fine. In some areas continuous agreement is achievable and necessary, in others it's too costly. It's all what you're willing to pay. For your rocks example, it's more important it look believable and less important it reflect server state. For the player, consistency with server overrides even simulation consistency (ie it's better for player to smoothly walk through a wall if that's what the server thinks should happen, even though that violates the client's physical constraints).
Good to hear you are in touch with Glenn, he's a good guy and very passionate (and knowledgeable) about game networking. His peered physics stuff is great for fixed physics props.
#6
Crysis shows that kind of:
gameplay relevant physics in MP only exist on the DX10 client, not the DX9 one as it is done through shaders.
12/07/2009 (5:24 am)
Determinism in physics simulation is at very best achieable through shaders, but not through cpus due to the pretty hefty differences on the floating point behavior and IEEE float error differences between the different generations and intel / amd in general.Crysis shows that kind of:
gameplay relevant physics in MP only exist on the DX10 client, not the DX9 one as it is done through shaders.
#7
1. If you don't have the same amount of objects
2. or the objects are added to the simulation in the same order
you don't have the same simulated result, because the physic system is not the same.
The client and server physic system initial state must be the same if you want the 2 simulations to be identical.
Perhaps you can give timestamp from time creation on the server to your physic objects to sort them, and an objects count in each physic packet sent from server.
With this, the client would now, when it receives a packet from the server, what objects are inserted or deleted from the whole physic system, and now when to insert or remove them in its own replication.
That would have the client system to keep a recent history of the physic state because it would have to rewind to the time of the oldest received event to modify the system from there and then let run the simulation with the updated server data.
If the back point is not in its history (too far away in time), it have to ask a complete update to the server.
When the client has the right data, it can continue to physic simulation with interpolation in rendering, etc… (classical Torque stuff)
Nicolas Buquet
www.buquet-net.com/cv/
PS : I'm absolutely not knowledgeable in physic nor in network, but sharing ideas is always a good thing. Thanks Tom.
12/07/2009 (5:31 am)
From my point of view, it seems normal that :1. If you don't have the same amount of objects
2. or the objects are added to the simulation in the same order
you don't have the same simulated result, because the physic system is not the same.
The client and server physic system initial state must be the same if you want the 2 simulations to be identical.
Perhaps you can give timestamp from time creation on the server to your physic objects to sort them, and an objects count in each physic packet sent from server.
With this, the client would now, when it receives a packet from the server, what objects are inserted or deleted from the whole physic system, and now when to insert or remove them in its own replication.
That would have the client system to keep a recent history of the physic state because it would have to rewind to the time of the oldest received event to modify the system from there and then let run the simulation with the updated server data.
If the back point is not in its history (too far away in time), it have to ask a complete update to the server.
When the client has the right data, it can continue to physic simulation with interpolation in rendering, etc… (classical Torque stuff)
Nicolas Buquet
www.buquet-net.com/cv/
PS : I'm absolutely not knowledgeable in physic nor in network, but sharing ideas is always a good thing. Thanks Tom.
#8
@Nicolas - Actually i think any time you have to rewind and replay physics your sort of screwed. Physics can be slow even in normal cases... having to resimulate 4-5 frames immediately and render will probably put a big dent in your framerate.
In reality it doesn't matter... the client sees the past server state anyway you slice it. So some buffering is smoothing is almost the only tool you have and even then you sometimes just gotta snap.
12/07/2009 (5:42 am)
@Marc - Actually i think the physics in Crysis multiplayer is still limited to client side effects and don't impact the gameplay simulation. I never ran it in DX10, but i don't know what graphics feature there would help network physics.@Nicolas - Actually i think any time you have to rewind and replay physics your sort of screwed. Physics can be slow even in normal cases... having to resimulate 4-5 frames immediately and render will probably put a big dent in your framerate.
In reality it doesn't matter... the client sees the past server state anyway you slice it. So some buffering is smoothing is almost the only tool you have and even then you sometimes just gotta snap.
#9
But a GPU at latest since DX10 is a lot more than just a thing to create graphics, its after all a super scalar stream processor.
For that reason, the cpu independent GPGPU through DX10 and DX10 hardware changes a lot.
I should say that I am at least under the impression that this is the reason why the DX10 client has physics and the DX9 hasn't as the DX10 one uses shaders for it
Once bullet is able to unleash OpenCL, it should potentially be (much more) deterministic too
12/07/2009 (6:14 am)
@Tom: graphic features would not change anything, that is right.But a GPU at latest since DX10 is a lot more than just a thing to create graphics, its after all a super scalar stream processor.
For that reason, the cpu independent GPGPU through DX10 and DX10 hardware changes a lot.
I should say that I am at least under the impression that this is the reason why the DX10 client has physics and the DX9 hasn't as the DX10 one uses shaders for it
Once bullet is able to unleash OpenCL, it should potentially be (much more) deterministic too
#10
I'm really looking forward to seeing more updates to the physics. Thanks for all of your hard work!
12/07/2009 (9:11 am)
What about cases where the physics are important to gameplay but having a networked game isn't? The game I am working on may eventually be multiplayer but is being designed as mainly single player. It's more of a puzzle game where blocks are stacked and can fall if stacked incorrectly. The physics simulation needs to be there and it would be nice to know when things stop moving so that their position can be recorded but they don't have to update the server (or client depending on how it all works out) that often. It would be nice to let the client simulate the blocks falling and then notify the server when the objects are at rest. Is that something that can be easily achieved? Or would be server side simulation with the client interpolating a few frames behind?I'm really looking forward to seeing more updates to the physics. Thanks for all of your hard work!
#11
It is something we're working on.
12/08/2009 (3:23 am)
Quote:What about cases where the physics are important to gameplay but having a networked game isn't?Are you talking about a true single player mode vs the fake multiplayer that Torque has done in the past?
It is something we're working on.
#12
12/09/2009 (9:45 pm)
When working with ODE<->TGE, I found that messing with the threshold for disabling the physics objects at low velocities helped greatly in keeping down network transmission and definitely seemed to be better on matching client/server simulations. With a good setting the objects stacked funny at times, but I got good results with enough objects moving in multiplayer to impress.
#13
Most of the improvement that you see here is a better correction implementation. When the correction arrives from the server i apply it immediately to the client physics object, but the rendering state converges on the new position/rotation over several ticks.
I also compressed the correction state pretty heavily. It was about 52 bytes per correction before and 19 bytes now in most cases. I could probably shave more off if i had two methods... one for high precision low velocity and another for low precision high velocity.
My next bit of work has to be on adusting the update priorities to heavily favor corrections nearest the camera especially when it has a high velocity or the player is interacting with it.
12/10/2009 (7:43 am)
I haven't had alot of time on this in the last few days, but here is a little more video...Most of the improvement that you see here is a better correction implementation. When the correction arrives from the server i apply it immediately to the client physics object, but the rendering state converges on the new position/rotation over several ticks.
I also compressed the correction state pretty heavily. It was about 52 bytes per correction before and 19 bytes now in most cases. I could probably shave more off if i had two methods... one for high precision low velocity and another for low precision high velocity.
My next bit of work has to be on adusting the update priorities to heavily favor corrections nearest the camera especially when it has a high velocity or the player is interacting with it.
#15
Not usable yet for physical simulations but a massive step ahead for a basic level of physical interaction in multiplayer.
Would be great to know how DICE did this stuff in FrostBite 2 :)
12/10/2009 (8:26 am)
Looks very good.Not usable yet for physical simulations but a massive step ahead for a basic level of physical interaction in multiplayer.
Would be great to know how DICE did this stuff in FrostBite 2 :)
#16
Frostbite does it by not networking rigid body physics at all.
They only send explosion events to the client... the client disables parts of the pre-fractured buildings and throws debris.
None of the physics affects gameplay... yet its extremely convincing to players.
12/10/2009 (8:45 am)
@MarcFrostbite does it by not networking rigid body physics at all.
They only send explosion events to the client... the client disables parts of the pre-fractured buildings and throws debris.
None of the physics affects gameplay... yet its extremely convincing to players.
#17
Same goes for beeing killed by a explosion based flying in humvee.
Though my favorite is the "jump with humvee through the wall and drive over sniper behind it" kill :)
But you might naturally be right that there are more tricks to it.
Though the decrease of players indicates that much more is going on than in the previous battle fields
12/10/2009 (10:07 am)
I don't know, beeing killed by a breaking building at least to me is quite a bit game play affecting (BF BC2 Beta) ;)Same goes for beeing killed by a explosion based flying in humvee.
Though my favorite is the "jump with humvee through the wall and drive over sniper behind it" kill :)
But you might naturally be right that there are more tricks to it.
Though the decrease of players indicates that much more is going on than in the previous battle fields
Associate Kyle Carter
You may want to consider grouping physics state updates by island, so you do a whole island (or at least related objects) in one update.
Definitely should not be interpolating anything but the render position. It'll be wonky in any case but interpolating will definitely cause problems (unless you do some sort of collision filtering based on who's interpolating, but I don't think there's any easy fix there).
You might be able to get around some of the determinism by forcibly rounding state every tick. But unfortunately if you are using a global solver the global state affects solutions.
Glenn Fiedler has some neat work on networked physics he recently made public. You might be interested in taking a look at it.