ActivateGhosting and object streaming
by Dave Young · in Torque Game Engine · 07/09/2007 (11:21 am) · 61 replies
Yes, this is a ghosting question!
After reading through a great many posts on ghosting, and TDN articles, I thought I had my brain wrapped around ghosting and scoping.
So I began to dig into the engine in an attempt to research more to try and get some form of object streaming in.
*Assumption* Reading through the TDN and forum posts about ghosting, it seemed like onCameraScopeQuery was where it was decided which objects to scope/ghost, and thus theoretically there was no need to worry about object streaming, as they wouldnt get ghosted to the client unless they were determined to be in scope.
Yet when I was looking into void NetConnection::activateGhosting(), I can see that every object is put into scope if it is ghostable at all, seemingly ignoring other scoping rules. I was wondering about this as it seems to decrease mission loading speeds, especially if there are a lot of objects. I tested it by putting an object on the far edge of a terrain and seeing if it got unpacked as a ghost in the ghosting process, which it did.
I was hoping that it would get sent down as I moved towards it, but it got sent down in mission load instead.
Thoughts?
Goals:
Decrease mission loading time on an object heavy server (like a large TGEA map)
After reading through a great many posts on ghosting, and TDN articles, I thought I had my brain wrapped around ghosting and scoping.
So I began to dig into the engine in an attempt to research more to try and get some form of object streaming in.
*Assumption* Reading through the TDN and forum posts about ghosting, it seemed like onCameraScopeQuery was where it was decided which objects to scope/ghost, and thus theoretically there was no need to worry about object streaming, as they wouldnt get ghosted to the client unless they were determined to be in scope.
Yet when I was looking into void NetConnection::activateGhosting(), I can see that every object is put into scope if it is ghostable at all, seemingly ignoring other scoping rules. I was wondering about this as it seems to decrease mission loading speeds, especially if there are a lot of objects. I tested it by putting an object on the far edge of a terrain and seeing if it got unpacked as a ghost in the ghosting process, which it did.
I was hoping that it would get sent down as I moved towards it, but it got sent down in mission load instead.
Thoughts?
Goals:
Decrease mission loading time on an object heavy server (like a large TGEA map)
#22
The key challenges are basically:
Server to Server Ghosting (with follow-on ghosting to clients)
There isn't anything fundamental in Torque networking to block this, except for the assumption that any particular object is either a Server object, or a Ghost object. Solving this could be as trivial as creating a third type (call it "mirrored object"), and modifying the isGhost, isClientObject, and isServerObject checks to properly evaluate the third state and act appropriately.
Migration of non-control objects across a server boundary
This one isn't all that difficult--simply need to define the boundary areas, and come up with a hysteresis based solution (to smooth cross-over thrashing) solution that causes an object to "promote" from a mirrored object to a server object, and "demote" from a server object to a mirrored object, and reverse the data transmission flow between the two servers, as well as make sure clients that have that object in scope are properly synchronized with the change. It's important to note that two clients viewing the same object may be receiving their data from two different servers--on client A for example they may be receiving a ghost update from a mirrored object, and client B may be receiving a ghost update from a Server object.
Migration of control objects across a server boundary
There's always a rub--and this is it. Timing this handshake is going to be complex, since not only do we have high fidelity updates coming from the controlling server (see control object updates), but we also have moves being transferred from the client to the controlling server, and finally we have client side prediction being applied to the client simulation. Given the nature of how client side prediction and control object updates work, you will need a 3-way synchronized handshake for this transfer of server control to keep things smooth and accurate during the transition.
Doable, but not trivial.
07/10/2007 (9:44 am)
Not going to quote (post was too long!), but Dave, you've isolated and identified most of the major requirements for large scale seamless zoning across multiple servers in an MM0-style architecture, while missing just a few of the key challenges. Good work :)The key challenges are basically:
Server to Server Ghosting (with follow-on ghosting to clients)
There isn't anything fundamental in Torque networking to block this, except for the assumption that any particular object is either a Server object, or a Ghost object. Solving this could be as trivial as creating a third type (call it "mirrored object"), and modifying the isGhost, isClientObject, and isServerObject checks to properly evaluate the third state and act appropriately.
Migration of non-control objects across a server boundary
This one isn't all that difficult--simply need to define the boundary areas, and come up with a hysteresis based solution (to smooth cross-over thrashing) solution that causes an object to "promote" from a mirrored object to a server object, and "demote" from a server object to a mirrored object, and reverse the data transmission flow between the two servers, as well as make sure clients that have that object in scope are properly synchronized with the change. It's important to note that two clients viewing the same object may be receiving their data from two different servers--on client A for example they may be receiving a ghost update from a mirrored object, and client B may be receiving a ghost update from a Server object.
Migration of control objects across a server boundary
There's always a rub--and this is it. Timing this handshake is going to be complex, since not only do we have high fidelity updates coming from the controlling server (see control object updates), but we also have moves being transferred from the client to the controlling server, and finally we have client side prediction being applied to the client simulation. Given the nature of how client side prediction and control object updates work, you will need a 3-way synchronized handshake for this transfer of server control to keep things smooth and accurate during the transition.
Doable, but not trivial.
#23
In regards to the scoping queries, if that ends up being a bottleneck for me I would try a couple things:
We've got 2 loops running on the scope query, outer loop for all world objects and inner loop to determine if something's in scope already for this client. The inner loop might already be optimized (starts at the appropriate counter) but it would be neat to see how many times that runs.
1) Optimize the MaxGhostCount (currently 4096)
and/or...
2) If I wasn't expecting the object to be anywhere near the player (ie it was not in the same designated subarea as the object, perhaps stored on the object itself) I would skip an inScope test.
Along the same lines, could use smaller sized groups instead of Sim::getRootGroup() as the outer loop, and keep track of items in smaller groups. This might not help for a smaller map, but in a rather large map I think it would be of tremendous value. In case we are worried about "just outside the border" objects not being detected, we could even use a 9 grid and still have a much smaller set of objects to search through for scope.
As objects moved around the map (for dynamic objects) they would change groups.
That objectInScope gets called quite a bit, so any low level optimizations would probably be useful!
07/10/2007 (10:53 am)
This is not a technical answer by any means to the bottleneck issue, only one of those half-baked pseudo answers.In regards to the scoping queries, if that ends up being a bottleneck for me I would try a couple things:
We've got 2 loops running on the scope query, outer loop for all world objects and inner loop to determine if something's in scope already for this client. The inner loop might already be optimized (starts at the appropriate counter) but it would be neat to see how many times that runs.
1) Optimize the MaxGhostCount (currently 4096)
and/or...
2) If I wasn't expecting the object to be anywhere near the player (ie it was not in the same designated subarea as the object, perhaps stored on the object itself) I would skip an inScope test.
Along the same lines, could use smaller sized groups instead of Sim::getRootGroup() as the outer loop, and keep track of items in smaller groups. This might not help for a smaller map, but in a rather large map I think it would be of tremendous value. In case we are worried about "just outside the border" objects not being detected, we could even use a 9 grid and still have a much smaller set of objects to search through for scope.
As objects moved around the map (for dynamic objects) they would change groups.
That objectInScope gets called quite a bit, so any low level optimizations would probably be useful!
#24
I have observe that fact in TGE and TGEA the light map is very huge which can eat RAM easily.
07/10/2007 (4:22 pm)
I am very interesting in this thread. I have observe that fact in TGE and TGEA the light map is very huge which can eat RAM easily.
#25
07/10/2007 (5:15 pm)
Is there a way in constructor to reduce the lightmap size lower than 32 ?
#26
but I think there are two problems remaining:
1, the RAM is not released when the objects are not in scope;
2, the interior objects have no light maps when they are in scope.
I don't know if I am correct.
07/11/2007 (4:12 am)
I have tested to remove ScopeAlways for interior and tsshape, i think it works for dynamic loading.but I think there are two problems remaining:
1, the RAM is not released when the objects are not in scope;
2, the interior objects have no light maps when they are in scope.
I don't know if I am correct.
#27
07/11/2007 (5:10 am)
Scope has nothing to do with releasing or allocating memory for the asset of which an object uses. That's something else entirely and you won't get that with stock code. Look into datablocks and how they are loaded.
#28
07/11/2007 (7:15 am)
If you remove scopeAlways for interiors and tsShapes, they wont be present when the lightmap is generated. It's a complication you will need to work around, one of the proposed solutions is to scope them, generate the lightmap, and then remove the scopeAlways (implementation not included (TM)), or to pregenerate the lightmap and then use that same lightmap in future loads. 
#29
Anyways, could the problem with the lightmap size not be solved with a sort of file comparison -and only sending the changed bits of it to the client ?
Would reduce the lightmap pack transferal considerably, and the comparison process 'ought' to be way shorter than the sending...
Just a thought!
07/11/2007 (7:53 am)
Im not at all sure I should post here, this is so far above my level that I'm having a hard time coping it..Anyways, could the problem with the lightmap size not be solved with a sort of file comparison -and only sending the changed bits of it to the client ?
Would reduce the lightmap pack transferal considerably, and the comparison process 'ought' to be way shorter than the sending...
Just a thought!
#30
Keep in mind that we don't network lightmaps (the server never has lightmaps), we network the objects that are used when the lightmaps are generated, and then let the client generate the lightmap. Since lightmaps are generated before the client enters the game world, the client needs to (stock implementation) receive data on all the objects that are part of this process before it can generate the maps (which is a one time thing).
07/11/2007 (7:59 am)
Quote:
Anyways, could the problem with the lightmap size not be solved with a sort of file comparison -and only sending the changed bits of it to the client ?
Would reduce the lightmap pack transferal considerably, and the comparison process 'ought' to be way shorter than the sending..
Keep in mind that we don't network lightmaps (the server never has lightmaps), we network the objects that are used when the lightmaps are generated, and then let the client generate the lightmap. Since lightmaps are generated before the client enters the game world, the client needs to (stock implementation) receive data on all the objects that are part of this process before it can generate the maps (which is a one time thing).
#31
Thank you very much for your comments.
As my understanding, in the client side, when a scene object is in the scope, the object will be added into a list, and when the object is not in the scope, which will be deleted from the list. This process has nothing with releasing or allocating memory, is it right?
07/11/2007 (11:31 am)
Stefan Lundmark:Thank you very much for your comments.
As my understanding, in the client side, when a scene object is in the scope, the object will be added into a list, and when the object is not in the scope, which will be deleted from the list. This process has nothing with releasing or allocating memory, is it right?
#32
In general, Torque never releases memory back to the operating system. This is actually very common in many applications (especially games and game engines), because memory operations to and from the operating system is one of the most expensive operations you can perform, so Torque uses a very common mechanism of asking for memory in large blocks, and not releasing that memory until the application is terminated.
07/11/2007 (11:39 am)
Other than the scoping list being maintained server side, not client side, you're statement is correct.In general, Torque never releases memory back to the operating system. This is actually very common in many applications (especially games and game engines), because memory operations to and from the operating system is one of the most expensive operations you can perform, so Torque uses a very common mechanism of asking for memory in large blocks, and not releasing that memory until the application is terminated.
#33
Thank you! I understand the mechanism of Torque further.
But that is to say Torque engine can not hold huge art assets in one level as World Of Warcraft.
If I do wish to allocate memory when an object in the scope, and release memory when the object leave the scope, do you have good suggestions for me?
07/11/2007 (1:02 pm)
Stephen Zepp:Thank you! I understand the mechanism of Torque further.
But that is to say Torque engine can not hold huge art assets in one level as World Of Warcraft.
If I do wish to allocate memory when an object in the scope, and release memory when the object leave the scope, do you have good suggestions for me?
#34
07/11/2007 (2:11 pm)
The engine behind World of Warcraft was coded to do what it does, and Torque is not coded for a huge seamless world spread across several servers as much as Doom III and Half-Life II ain't either. Engines are not drop-in to any game-type without any effort on the person using it (ie. better know how to code or someone who does).
#35
Quazal IrrLicht CryEngine 1&2 and a lot more....
But real mmo engines are way above what most around here can afford, say will be allowed to even get whitepapers for. But, that should not restrict persuit of trying to cope, or emulate the mechanics that drive those big ones. And a lot of the efforts, especially from the folks that been around the mmokit, sure have given a lot of insight, inspiration, spinoff etc.
That being said though, I again have to agree with Stefan -thinking that any mmo like game will be done without extreme amount of tweaking 'any engine' towards the mechanics is pure self fooling!
-its hard work, but the backup (info sharing) of the GG comunity is really good!
07/11/2007 (2:49 pm)
I have to agree with stefan that TGE(A) was not at all designed for mmos like:Quazal IrrLicht CryEngine 1&2 and a lot more....
But real mmo engines are way above what most around here can afford, say will be allowed to even get whitepapers for. But, that should not restrict persuit of trying to cope, or emulate the mechanics that drive those big ones. And a lot of the efforts, especially from the folks that been around the mmokit, sure have given a lot of insight, inspiration, spinoff etc.
That being said though, I again have to agree with Stefan -thinking that any mmo like game will be done without extreme amount of tweaking 'any engine' towards the mechanics is pure self fooling!
-its hard work, but the backup (info sharing) of the GG comunity is really good!
#36
Because I have all the source codes of Torque engine, so I can do the best to change it to suit my games.
Here, I only wish to discuss a technology issue about how to release or allocate memory in scenemanager of Torque. I have no other meanings. I also believe this is not a awesome technology difficulty we can not overcome.
I indeed wish to get as more as possible good suggestions about this technology issue.
Thank you again!
07/11/2007 (3:55 pm)
I agree with Stefan Lundmark also. Torque engine is great, I enjoy it.Because I have all the source codes of Torque engine, so I can do the best to change it to suit my games.
Here, I only wish to discuss a technology issue about how to release or allocate memory in scenemanager of Torque. I have no other meanings. I also believe this is not a awesome technology difficulty we can not overcome.
I indeed wish to get as more as possible good suggestions about this technology issue.
Thank you again!
#37
When a ghost goes out of scope it is freed from the client ghost list. Stephen is correct that the memory does not get released back to the operating system. It will be re-allocated by Torque's memory manager, however. Torque allocates a big chunk of memory, and then it manages calls to 'new', 'delete', 'malloc', and 'free'. (This is why, if you try to link STL into Torque without making modifications, it will complain about the 'new' operator.) So while, yes, the memory is not released to the OS, it can be considered deallocated for the all intents and purposes.
My suggestion to you is to follow creation of an object on the server, and trace it's path through the ghosting system, through the connection, onto the client, instantiation on the client, datablock assignment, and initial update. I recently had to do just this to track down a problem with networked components. The problem was a pack/unpack mismatch with component objects. The reason was because, on the server, when the initial update packet is written, the object writing the packet has been added and registered with the Sim (onAdd has been called). On the client, when the ghost packet is read, and the initial update packet is unpacked, the object is not added to the Sim, (onAdd has not been called).
07/11/2007 (4:07 pm)
Quote:It already does this.
If I do wish to allocate memory when an object in the scope, and release memory when the object leave the scope, do you have good suggestions for me?
When a ghost goes out of scope it is freed from the client ghost list. Stephen is correct that the memory does not get released back to the operating system. It will be re-allocated by Torque's memory manager, however. Torque allocates a big chunk of memory, and then it manages calls to 'new', 'delete', 'malloc', and 'free'. (This is why, if you try to link STL into Torque without making modifications, it will complain about the 'new' operator.) So while, yes, the memory is not released to the OS, it can be considered deallocated for the all intents and purposes.
My suggestion to you is to follow creation of an object on the server, and trace it's path through the ghosting system, through the connection, onto the client, instantiation on the client, datablock assignment, and initial update. I recently had to do just this to track down a problem with networked components. The problem was a pack/unpack mismatch with component objects. The reason was because, on the server, when the initial update packet is written, the object writing the packet has been added and registered with the Sim (onAdd has been called). On the client, when the ghost packet is read, and the initial update packet is unpacked, the object is not added to the Sim, (onAdd has not been called).
#39
I think what he wants is to release the asset which the object is using. Like a Shape File and it's textures.
If you feel like experimenting, you could check if other objects use the same shape file, and if there are none of those in scope, release the shape file, set it's pointer to NULL, and as soon as it get's back into scope again, do something similiar to preload (). You will notice small (and not so small) delays when this happens, and that's basically where I left it as I didn't feel it was worth it and I didn't want to play around with a worker thread there.
07/12/2007 (2:49 am)
Quote:
Pat Wilson said:
It already does this.
I think what he wants is to release the asset which the object is using. Like a Shape File and it's textures.
Quote:
Hongtao Wang said:
If I do wish to allocate memory when an object in the scope, and release memory when the object leave the scope, do you have good suggestions for me?
If you feel like experimenting, you could check if other objects use the same shape file, and if there are none of those in scope, release the shape file, set it's pointer to NULL, and as soon as it get's back into scope again, do something similiar to preload (). You will notice small (and not so small) delays when this happens, and that's basically where I left it as I didn't feel it was worth it and I didn't want to play around with a worker thread there.
#40
Yes, my English is bad, SORRY!
I wonder if you truly understand the mechanism of Torque about it's memory management, although the memory can not be used by the OS, but which can be resued by torque!
07/12/2007 (6:47 am)
Stefan Lundmark:Yes, my English is bad, SORRY!
I wonder if you truly understand the mechanism of Torque about it's memory management, although the memory can not be used by the OS, but which can be resued by torque!
Torque 3D Owner Mark Dynna
Additionally, I'm starting to think that the real solution will be making ScopeAlways conditional on TSStatic objects. In our case we have some large and fairly significant Statics (city walls), where the effect of them suddenly "popping" into view is quite noticeable. In those few cases I would like to be able to just set a flag for that object to have it "pre-downloaded" as a ScopeAlways object.