Ghosting Limits

#1 Jesse, you can increase the amount of allowable ghosts in sim/netConnection.h by changing the GhostBitSize (around line 821 in my 3.5.1 build). I have successfully raised it up to 16 bits, any higher and you start to have networking problems. In my game, I use path markers as spawning points for my AI, and have around 25,000 of them. This became a problem during editing, as it would take about a minute for all the ghosts to show up in the editor. I wound up switching off setScopeAlways() for the markers, and now only the markers within view distance are ghosted. Maybe your cubes have setScopeAlways set?

#2 A solid overview of ghosting and scoping is available at http://www.garagegames.com/products/torque-3d/documentation in the Reference Guide section, under Modules > Networking > On Ghosting and Scoping. For camera view distance specifically, this is how it is supposed to work:

Quote:
First, the server determines what objects are in-scope for the client. This is done by calling onCameraScopeQuery() on the object which is considered the "scope" object. This is usually the player object, but it can be something else. (For instance, the current vehicle, or an object we're remote controlling.)

So an investigation into onCameraScopeQuery() should reveal the exact method used for determining what gets scoped based on view distance.

If you go over 16 bits I think you're in for a bit of work, as Joseph points out. You'll end up having to tweak core bits of the networking system to allow for more data, and you'll increase the required bandwidth for your game.

I guess you'll need to ensure that setGhostAlways() is set to false wherever possible - it should be false by default but it never hurts to check....

#3 Thanks guys, some solid advice. I believe you've set me in the right direction, both in theory and application. I won't be able to look into this until later tonight, at the soonest, but this gives me a lot to consider.

@Joseph: Nice, 2 great examples all in the scope of a short paragraph! I believe both of your suggestions will come into play here, thanks!

@Richard: 10 Bonus points awarded! I appreciate it Richard, as always your input is very insightful. I can now see what's supposed to be happening, and you've given me the leads I need to (hopefully) see it through. I think 10 bonus points is definitely fair, that's the difference between a 'B' and an 'A' depending on who's grading the paper :P Haha, thanks Richard!

#4 Alright, I've had a chance to take a look at some of this. I'm still not seeing any sort of difference, regardless of what I try, in the number of ghosted objects based on distance. If I am understanding correctly, the default behavior of the NetObject is to try and ghost everything:

void NetObject::onCameraScopeQuery(NetConnection *cr, CameraScopeQuery* /*camInfo*/)
{
   // default behavior -
   // ghost everything that is ghostable

   for (SimSetIterator obj(Sim::getRootGroup()); *obj; ++obj)
   {
		NetObject* nobj = dynamic_cast<NetObject*>(*obj);
		if (nobj)
		{
			AssertFatal(!nobj->mNetFlags.test(NetObject::Ghostable) || !nobj->mNetFlags.test(NetObject::IsGhost),
			   "NetObject::onCameraScopeQuery: object marked both ghostable and as ghost");

			// Some objects don't ever want to be ghosted
			if (!nobj->mNetFlags.test(NetObject::Ghostable))
				continue;
         if (!nobj->mNetFlags.test(NetObject::ScopeAlways))
         {
            // it's in scope...
            cr->objectInScope(nobj);
         }
      }
   }
}

So, this being the case, if I am to setScopeAlways() to false this should allow the object to not be ghosted beyond view distance?

Quote:I wound up switching off setScopeAlways() for the markers, and now only the markers within view distance are ghosted. Maybe your cubes have setScopeAlways set?

How exactly do you use setScopeAlways()? I tried to set it when creating the TSStatic like so:

new TSStatic() {
...
setScopeAlways = "false";
...
}

I have since found the 'N' hotkey for the netgraph, which is pretty useful :) Using this, regardless of trying the above I'm not seeing any difference in number of scoped objects. Pretty sure it's just me not using the flag correctly, if not I'll have to plug in Vince's ghost limiting resource just to 'hard-limit' it. Just strange I'm not able to use the default ghosting flags, as it seems Joseph was able to. Should the flag be set when creating the object? As a function after the object is created?

#5 How are you rendering your cubes? If you're using the T3D examples (renderMeshExample, renderObjectExample, renderShapeExample), the flag is set in the source code. In renderMeshExample.cpp, for example, (an example of an example?:) line 58 sets the flag:

mNetFlags.set( Ghostable | ScopeAlways );

In this case, you would just delete the ScopeAlways argument so the flag won't get set. In my case, I had to delete a call to setScopeAlways()for the marker object.

#6 I think this is what your looking for.

In NetConnection.h

//Winterleaf Entertainment
/*
Little note here, we aren't building a game, we are building a world editor, and since each edit is a ghost
we need the ability to have alot of ghosts.  Since ghosts never update, they will only send a update once
to a single player during the lifetime of the session.

So for the default setting 12-3 = 9, and 9 can be stored in a 4 bit uint(0-15)
So for the new setting, 18-3=15, and 15 can also be stored in a 4 bit uint(0-15) thus no changes need to 
be made to the GhostIndexbitSize You can calculate it by Ciel(Log(GhostIdBitSize)/Log(2))

*/
      //GhostIdBitSize = 12, // This is 4096 max ghosts													//Winterleaf Entertainment L.L.C. 2013 Copyright
      GhostIdBitSize = 18, //This expands it to 262,144  18-3 = 15										//Winterleaf Entertainment L.L.C. 2013 Copyright
      MaxGhostCount = 1 << GhostIdBitSize, //4096,
      GhostLookupTableSize = 1 << GhostIdBitSize, //4096
      GhostIndexBitSize = 4 // number of bits GhostIdBitSize-3 fits into
   };

#7 @Joseph:

Quote:How are you rendering your cubes?

I am doing all of this in TorqueScript. I create a ScriptObject, then a function to generate cubes that the ScriptObject calls:

$Example = new ScriptObject(){
   class = "Example";
};

function Example::generate() {
   ...
   // nested for loops creating cubes here
   ...
}

Then, to actually render the objects I can type this into the console:

$Example.generate();

The major disconnect that I have when I am dealing with source code is how the actual source functions translate to something usable in the console or in TorqueScript. Based on your 'Example of an example' (which is totally necessary in my case, thanks so much Joseph it really helps), I can clearly see in the source code all the steps being taken to render the mesh. My confusion comes when I try to understand how to use that at runtime. The source is all precompiled, so when developing I feel forced into console functions or TorqueScript use? How does one create a function in source and then call that function at runtime since the console doesn't accept C++?

Nonetheless, since in my case I'm creating TSStatics, could I just delete the ScopeAlways argument in tsStatic.cpp (around line 94) so that any TSStatic was created without this flag?

@Vince:
Thanks man. I believe I am understanding this part now at least. I mean how to just increase the hard limit there isn't too big a problem for me I think at this point. What I'm trying to grasp now is how to turn the scoping/ghosting on or off depending on view distance. If I've got 1000 cubes in front of me, my problem is that if I walk away so that they are not in view distance anymore - they are still being reported as ghosts, despite the fact that they are not visible or within 'range' anymore. I think if I implement your resource, I could manually just set this range but I'm not 100% certain.

#8 @Jesse - check this out: http://docs.garagegames.com/torque-3d/official/index.html?content/documentation/Scripting/Overview/Introduction.html, under Scripting--> Advanced--> Engine To Script

You provide a script "hook" into an engine-side function or object method, then call it from TorqueScript at runtime. You are gonna love that stuff....

And you could remove the ScopeAlways flag there, but it would be better to add a way to set that to TSStatic. Maybe I'll fiddle with doing that this week....

#9 And

// not this
new TSStatic() {  
...  
setScopeAlways = "false";  
...  
}

// this....
%obj = new TSStatic(){
   ...
};
%obj.setScopeAlways(false);

#10 Yup, Jesse, you can turn off ScopeAlways for your cubes by deleting the ScopeAlways argument in tsStatic.cpp, as you suggested. Be aware that ALL tsStatics will be affected, so it depends on how that will affect your game setup, if it uses tsStatics.

Everything you can do in Script starts out in the Engine. If the Engine can do something that you can't do in Script, you can expose the Engine method to the Console, and then it can be scripted. See how it's done in the link Richard gave. If the Engine can't do something but you know how to write it, you can brew your own functions and expose them to script. That's one of the beauties of Torque!

Richard, I don't believe setScopeAlways() uses any arguments. Use it to turn ghosting on ( %obj.setScopeAlways()) and to turn ghosting off, use %obj.clearScopeAlways().

#11 @Jesse,

Yes my resource would do what your looking to do at the network level. It basically culls the network ghosts to objects that are over a certain distance to the camera.

You wouldn't have to mess with this setscopealways stuff.

#12

Quote:
Richard, I don't believe setScopeAlways() uses any arguments. Use it to turn ghosting on ( %obj.setScopeAlways()) and to turn ghosting off, use %obj.clearScopeAlways().

I was just looking at this (looked it up to answer the question), and I realize I was actually looking at T2D (which works as I described).... In T3D it should work as you describe but it appears that the NetObject engine method for clearScopeAlways is not defined (at least in 3.5.1), so I guess it needs to be defined.

That said, I like the older syntax - or make it a property so we can do this:

%obj.scopeAlways = false;

#13 Here's a crazy question.. why are you ghosting 1000's of cubes? Ghosting is meant to keep an object in sync between the client and server. I doubt 1000's of cubes justify being ghosted. What would make more sense is to contain your 1000's of generated cubes into 1 object (or multiples to break it up) and you ghost 1 object and in pack/unpack transfer the necessary data to recreate the 1000's of cubes.

#14 Well, the crazy answer is that they're TSStatics and are ghostAlways by default - which is crazy when you look at the definition of what ghosts are for....

One of the many underlying WTF's we should look into correcting.

#15 @Andrew Mac:

Quote:Here's a crazy question.. why are you ghosting 1000's of cubes?

That's exactly the problem Andrew, I don't want to ghost them. But by default if I add them to my scene they are being ghosted. These cubes may need to change states (visible, not visible, etc), so at some point I believe they will need to be ghosted - just not if they are far away. You are correct, though, there is no reason to have so many cubes ghosted. Ideally, I would want only some cubes in a radius around each player being ghosted.

• As a side note, if I am using TSStatics to achieve this will the cubes be able to change dynamically? (visible with collision/not visible without collision)

Quote:
What would make more sense is to contain your 1000's of generated cubes into 1 object (or multiples to break it up) and you ghost 1 object and in pack/unpack transfer the necessary data to recreate the 1000's of cubes.

If you can show me how to do this I'd love to try it. I'm creating a script object using a global variable and then storing the array data on it. Then I'm going through the array and dropping cubes at each point. In a way, it is one script object calling the functions but I'm not sure how to 'pack/unpack' the stuff as you describe.

@Joseph

Quote:Richard, I don't believe setScopeAlways() uses any arguments. Use it to turn ghosting on ( %obj.setScopeAlways()) and to turn ghosting off, use %obj.clearScopeAlways().

I believe this may be what I was looking for originally, just a way to change the flag in TorqueScript without resorting to altering the source.

@Vince

Quote:Yes my resource would do what your looking to do at the network level. It basically culls the network ghosts to objects that are over a certain distance to the camera.

You wouldn't have to mess with this setscopealways stuff.

Thanks Vince, I do believe you are right and tbh I think this is the path I will take once I have my development environment back up to par.

I've been busy setting up my new dedicated server computer, and also dealing with a corrupted Visual Studio install (ugh) so I hadn't been able to try your resource fully yet. I hope to be able to try this out by tonight at the earliest.

#16 The torque gods will not like this, but why not just use the server as a relay system, and have everything generated via the client side. You would have some work to do in order to accomplish this engine side, but just another thought. That way, you can send a seed or chunk or something from the server and then it generates on the client.

Then again, this completely goes against Torque's networking.

#17

Quote:
Well, the crazy answer is that they're TSStatics and are ghostAlways by default - which is crazy when you look at the definition of what ghosts are for....

Alright, so I dug into it and I'm kind of neutral on the topic. The limit that is being hit is the maximum number of ghost IDs. It seems torque will ghost these objects once and doesn't consider them "active ghosts". If you check net stats you won't have an active ghost count matching the amount of TSStatics in the scene. The plus side is this leaves room for static objects to be updated by the server. The downside is pretty much non-existent since they aren't actively updated (put a breakpoint in packUpdate in TSStatic, it's not called after the first time.)

Regarding the issue at hand, there doesn't seem to be a downside to following the methods of increasing ghost count since it's really just a maximum ghost ID count. I would just do that to solve your immediate issue.

With that said though, you're going to run into performance issues pretty fast when you start creating that many TSStatics. There's too much overhead for each block. I assume you're going for some kind of minecraft like setup where the amount of blocks is going to get insane very quickly. The trick to this is actually generating the vertices and building the vertex buffer yourself. You'll have to do this from inside the engine. Take a look at RenderMeshExample, createGeometry() function specifically:
github.com/GarageGames/Torque3D/blob/69838bdc8c9bc055b9b1ae76f42b0f28d2a33909/En...

It's a single cube. You'd want that one world object to be building/rebuilding the vertex/prim buffers (basically call your own version of createGeometry) and hold a maximum number of blocks based on where you find a performance sweet spot. Then your world would be made up multiples of these objects. That way you don't have to rebuild the entire world when one block is removed or changed. It's a bit of a challenge for sure, but so is the idea of having a world built out of thousands of cubes :D

#18 Since NetObject::clearScopeAlways() isn't exposed, you can drop this in if you feel you need it:

// in sim/netObject.cpp around line 280, I added

DefineEngineMethod(NetObject, clearScopeAlways, void, (), ,
	"@brief Always scope this object on all connections.\n\n"

	"The object is no longer marked as ScopeAlways and is removed from the list "
	"of ghosts for all applicable connections.  This function has no effect if the object "
	"is not marked as Ghostable.\n\n")
{
	object->clearScopeAlways();
}

Of course, I really like Andrew's solution. TSStatics were intended to be "lightweight" but he's absolutely right - for that many of them there is just too much overhead.

#19 Does being ScopeAlways have anything to do with TSStatic's 'lightweight' networking properties?

Thanks for that, Richard! PR plox? :3

#20 I would, but I'll be damned if I can actually get my development branch to sync properly with GG. I need to find a way to create a fresh fork....

Git is not my friend. Can we switch to Perforce?

And I don't really know what effect it'll have if you remove scopeAlways from TSStatics. Maybe the "popping" will be a killer - haven't tried it myself....