Ghosting Limits

#21 @Andrew Mac: Your recommendations are great, and in all honesty I think this is the best way to achieve the result. By generating the vertices and building the vertex buffer. I'm sure this approach is a bit beyond my current coding capabilities, but it's a great way to handle this sort of thing for someone up to the task. (Hint, hint! Someone get to work on this, because diffuse/normal/specular-mapped MineCraft sounds yummy...:)

Actually, come to think of it, the performance on all those cubes isn't nearly that bad if you have some occlusion culling in place (cough cough, finish your culling resource Andrew!! I'd be using it right now if you hadn't declared it not done!). Pair that with some creative LOD's and tbh it wouldn't be that bad at all...Let's not even begin with culling algorithms, where you could limit the cubes to only the ones touching 'air' and so on. All of this sounds great, and I may explore it somewhat, but it's not on the agenda atm :)

The Goal
I'm not after a world of cubes on the MineCraft scale. I decided very early on that, while MC-style is cool and all, the job of accomplishing this from scratch in Torque would probably be beyond me. Not to mention the endless sea of Minecraft clone games already emerging...What I am after, though, is a world of cubes in an isometric view. Nothing more, nothing less. This being the case, and with the camera angle being where I set it, I figure a combination of increasing the ghost limit and culling those ghosts based on view distance should suffice. The big problem has been that any static added to the scene is being counted regardless of camera angle, distance from player, or anything so the limit gets enforced far prematurely for the 'simple' scene I'm trying to create.

@All: Thanks so much for the in depth look into all of this. Even if you have to drag me kicking and screaming, you guys may make a developer out of me yet. With Richard's examples and posted resources, I'm (finally!) starting to understand EngineMethods somewhat. This is a huge step for me and while I've been an avid fan of Torque for quite some time, one of the biggest limits on my use of it has been making the connection between source and script. This is a very healthy and productive discussion for me, and I am deeply appreciative! Thumbs up for community today!

P.S. Server PC's up, and main PC's being finished up after an OS install - Hopefully by tomorrow I'll have some results to share around my original quandary.

#22 Might study: github.com/GarageGames/Torque3D/blob/69838bdc8c9bc055b9b1ae76f42b0f28d2a33909/En... for an example of a simplified class that sends a random seed from the server to the client and replicates functional renderable and collidable meshes. (disclaimer: not exposed to the world editor as it's been replaced by classes with more 'bling', and the placement is a bit off due to the typemask check, but the point is to pick something minimalist that's close to what you're looking for. so.)

#23 @Vince: derp, I just realized your resource is dealing with Players, not TSStatic shapes...so that's a no-go.

@Joseph: When I add %obj.clearScopeAlways(); I get errors in the console on creation of the statics that clearScopeAlways() doesn't exist. Also a no-go.

I also went ahead and changed this:
tsStatic.cpp

TSStatic::TSStatic()
{
   mNetFlags.set(Ghostable | ScopeAlways);
   ...

to this:

TSStatic::TSStatic()
{
   mNetFlags.set(Ghostable);
   ...

Still no dice. It seems that no matter what approach I take here, in script or source, there is no affect on the # of ghosted objects. I'm pretty much limited to around 4000 shapes and that's it. Any more than this, and the console will immediately complain that there are too many ghosts. So far, it appears that Joseph found some way around this for markers but whatever it is he's done isn't coming into play here. I'm not sure if it's an older build of Torque he's using? Whatever the case may be, I challenge anyone to limit TSStatic ghosting based on distance from the player :P Create a group of cubes, push N to observe the netgraph number of active ghosts, and tell me if you can move away from those cubes and have that active number of ghosts go down...

#24

Quote:
When I add %obj.clearScopeAlways(); I get errors in the console on creation of the statics that clearScopeAlways() doesn't exist. Also a no-go.

I mentioned earlier that clearScopeAlways() isn't exposed to script (which is weird), and provided a piece of code to drop into sim/netObject.cpp to remedy this.

#25 Thanks for the quick reply Richard, I must have overlooked it with all of the hardware/software updates I've been doing. I'll try it, and this is a perfect example to help me start exposing stuff to script :)

#26 Once you get a feel for how easy it is you'll want to use it all the time - but don't do it! Don't let the power go to your head!

#27 Haha Richard, you are definitely right. This sheds a whole new light on Torque for me, it really does. I am finally able to implement source changes and things I've always not fully understood how to...

But as for the issue at hand, I've been met with a half-success:

• The good part: Now the # of active ghosts only goes up once I am within view distance of the cubes. So if I spawn in 1000 cubes but I can only see 20, the # of ghosts will be around 20. If I approach the group of cubes, the # of ghosts goes up accordingly. Check.
• The bad part:The # of active ghosts never goes back down. Once I approach the cubes and they are added to the # of active ghosts, if I then run far away from the cubes the # of active ghosts never drops back down.

Getting closer, as I'm seeing a measurable difference in ghosts before they become ghosts. But it seems once they are 'locked in' as a ghost they are never culled off the 'ghost-list' again.

#28 After messing with this for a couple days, I've just become frustrated with Torque. How are developers making games that hold more than this amount of TSStatics (or any object for that matter)?

Here's why it's so frustrating: Out the box, Torque has the potential to just make a MineCraft-like game. There's one small problem of objects not dropping from the list of objects being ghosted, but that's it. If I am understanding all of this correctly, by default Torque is not supposed to ghost objects beyond view distance...yet it does so this is broken...and the only thing stopping anyone from creating a MineCraft-like game.

While the pack/unpack suggestions are really well thought out, they are ultimately unnecessary if the engine would just act like it's 'supposed' to and drop the ghosts that are out of view distance range.

Consider this:

• I can create as many objects as I want just fine if they are out of view distance. Hundreds of 1000s? Yep, no problem.
• If these object come within view distance, they become ghosts.
• This number of ghosts always accumulates more ghosts as more are encountered...until the limit is hit.
• If the player then travels away from those objects, they are not being updated to not be ghosts.

So the only problem here is that the ghosts aren't being dropped once they are outside of view distance. A behavior that Torque is 'supposed' to handle by default? So what's broken here?

#29

Quote:How are developers making games that hold more than this amount of TSStatics (or any object for that matter)?

Well, I imagine they're not. Of the very small amount of games that get made with Torque, I expect most of them aren't trying to create a terrain out of TSStatics. That said, TSStatic should definitely behave according to the principle of least surprise and obey the usual ghosting rules, and if developers feel like turning on some sort of optimisation like ScopeAlways then that should be an option.

Don't worry, we're all frustrated with Torque constantly. It's called using Torque ;).

And, to comment on your specific issue - I can almost guarantee you that Minecraft does not just create thousands of cubes for its terrain. Andrewmac's approach is reasonable, and you'd also want to look into actual voxel techniques. But of course, we should support the TSStatic route for prototyping, or maybe some other valid use case that needs that many objects. Did increasing the number of bits allocated to ghost IDs not help at all? That's definitely something we need to look into if that's the case...

#30 I think the ghost list isn't properly removing ghosts - and that could be an issue in other games. If the ghosts aren't removed, then are the ghost IDs being updated or changed when they're "re-ghosted" to the client - or do they keep the same ID (and therefore make it easier to translate a server object into a known object ID)?

#31 @Daniel:

Quote: Don't worry, we're all frustrated with Torque constantly. It's called using Torque ;).

Honestly, thanks, I was beginning to think it was just me :)

Quote: I can almost guarantee you that Minecraft does not just create thousands of cubes for its terrain. Andrewmac's approach is reasonable, and you'd also want to look into actual voxel techniques.

You are right on all fronts here, and I have done tons of research around the generation, culling, addition, and removal of cubes over a year or so...But the problem is if the ghosting and scoping isn't happening as it should on a networking level, all the fantastic algorithms for generation are of no use. Hitting a hard limit on ghosted objects that are never removed is a pretty big problem.

Quote: Did increasing the number of bits allocated to ghost IDs not help at all?

Well it does help, but it's like putting a Band-Aid on a flesh wound. For example: Say you can only make 4000 objects without an increase in bits. Then say you go ahead and up the bits and now you can make 20k objects. Regardless, if a player traversed enough in a single gameplay session he'd still hit the limit at some point. If the ghosts aren't properly being removed, it's just a fundamental flaw that will come up at some point.

@Richard:

Quote: I think the ghost list isn't properly removing ghosts - and that could be an issue in other games.

You are exactly right Richard. I can't see how it hasn't come up before now. In Torque, we can generate some pretty large terrains as is and populating that entire terrain with objects could become a nightmare if the ghosts aren't being removed.

Possible Solution:
Well I was impressed by Vince Gee's solution around ShapeBase Player ghosting. If the same solution were implemented but for TSStatics instead of Players it should work. With enough time, I'm sure I can search line by line through the resource and find what's actually taking place to cull out the players...but then I'd have to figure out the call for TSStatics in C++. I'm not fluent with C++, but I am slowly trying to make the conversion. It's a matter of time before it all 'clicks' I guess, and there's a lot of potential places to try and find what I'm looking for in the source code. A needle in a haystack, so to speak.

#32 Ok, this comment from netObject.h may be interesting:

/// Queries the object about information used to determine scope.
///
/// Something that is 'in scope' is somehow interesting to the client.
///
/// If we are a NetConnection's scope object, it calls this method to determine
/// how things should be scoped; basically, we tell it our field of view with camInfo,
/// and have the opportunity to manually mark items as "in scope" as we see fit.
///
/// By default, we just mark all ghostable objects as in scope.
///
/// @param   cr         Net connection requesting scope information.
/// @param   camInfo    Information about what this object can see.
virtual void onCameraScopeQuery(NetConnection *cr, CameraScopeQuery *camInfo);

Note the last line - by default, all objects are ghosted (see netObject.cpp). This is overridden lower down the inheritance tree - SceneObject calls scopeScene, and ShapeBase sets some values for this query like visible distance and angle (which is actually unused, apparently).

I'm a little fuzzy about how ghosts are killed after they're scoped. Have a look at this comment, though:

// [rene, 07-Mar-11] Killing ghosts depending on the camera scope queries
//    seems like a bad thing to me and something that definitely has the potential
//    of causing scoping to eat into bandwidth rather than preserve it.  As soon
//    as an object comes back into scope, it will have to completely retransmit its
//    full server-side state from scratch.

if(!(mGhostArray[i]->flags & GhostInfo::InScope))
   detachObject(mGhostArray[i]);

Which I thought would have some bearing on your problem - but it looks like the code Rene was commenting on is still there, i.e. the call to detachObject. Hopefully someone with more experience than me can step in...

One way to test this might be to replace all your TSStatics with StaticShapes and see if the same problem occurs.

#33 @Daniel: Thanks for all the help. At the very least, you've provided a handy reference to most of the source files in question here. At this point, it would appear that the call to detachObject is what is actually removing the ghost from the array of actives. Now, if I could just call to that for every active ghost that drops out of view distance (which I thought Torque was supposed to do by default) I think this would work. I'm still not really sure why the ghosts aren't being culled outside of view distance by default..

I did get a chance to replace the TSStatics with StaticShapes for testing and the problem did still occur. Additionally, it introduced a large performance hit (probably due to the StaticShape not being as lightweight as the TSStatic).

@All:Thanks a ton for all of the input in this thread. I'm already learning tons more about the source (and C++ in general) just by going over all of the suggestions. Ultimately I have to agree with Andrew Mac about building the vertex buffers, and Azaezel about pack/unpack to actually make this work 'efficiently'. Now, if only I had any inkling of a clue as to how to actually do any of that :) I mean, I do understand the theory of what's taking place but putting it all into play is another matter entirely.

#34 Yeah, it seems like that detachObject call should be removing ghosts that aren't scoped. We'll have to look into this. For the record, I believe Rene's comment was a little short-sighted: scoping is partly a network optimisation, but also an important way of limiting cheating in a multiplayer game. For example, in an RTS you could remove objects from scope that are beneath the fog of war, and there would be no way for a player to cheat and view enemy units using a client-side hack. It's also important in cases like yours that ghosting only happens to relevant items, if there are a lot of them, rather than letting things hang around that are of no interest to the client (distant objects).

#35 That's a good insight into scoping, thanks. Keep me posted Danny, for now the scoping problem is all that stands in my way of generating some pretty cool stuff in T3D! I'm just recently beginning my journey into C++, so I've got a long row to hoe before I'll be able to fix something as complex as this scoping in source. I'm messing with includes, declarations, variables, etc...Not even remotely in the same ballpark as networked scoped ghost arrays :)

Anyways, godspeed on a solid solution my friend. I'm buckling down to become more fluent with C++ this winter; perhaps I'll be of more use in these situations in the future :)

#36 Awesome. Frankly, just using the engine is really helpful, because it finds use cases that we haven't thought of and exposes subtle bugs like this one. But getting fluent with the C++ opens up so many options for working with the engine :).

#37 Alright, well I feel I may finally be on the right track regarding ghosting and scoping. Check this out. I believe I discovered why there was no defined Engine Method for clearScopeAlways in the first place. If you look closely at the comment from the original code...

netObject.cpp

void NetObject::clearScopeAlways()
{
   if(!mNetFlags.test(IsGhost))
   {
      mNetFlags.clear(ScopeAlways);
      Sim::getGhostAlwaysSet()->removeObject(this);

      // Un ghost this object from all the connections
      while(mFirstObjectRef)
         mFirstObjectRef->connection->detachObject(mFirstObjectRef);
   }
}

...you notice that it's un-ghosting the object from all connections. So then what happens if the same object still needs to be ghosted to other connections, who may have the object within view range at a later time?

I looked a little closer, and here's what I found out. There already does exist Engine Methods to scope and clear the scope of objects on a per client basis! This made my day, and here's how it works if I'm correct: You use scopeToClient() when you want to scope the object to a specific connection and clearScopeToClient() when you want to drop it from scope per connection. There is even a little bit of clarification in the source comments:

netObject.cpp

DefineEngineMethod( NetObject, scopeToClient, void, ( NetConnection* client),,
   "@brief Cause the NetObject to be forced as scoped on the specified NetConnection.nn"

   "@param client The connection this object will always be scoped tonn"

   "@tsexamplen"
      "// Called to create new cameras in TorqueScriptn"
      "// %this - The active GameConnectionn"
      "// %spawnPoint - The spawn point location where we creat the cameran"
      "function GameConnection::spawnCamera(%this, %spawnPoint)n"
      "{n"
      "	// If this connection's camera existsn"
      "	if(isObject(%this.camera))n"
      "	{n"
      "		// Add it to the mission group to be cleaned up latern"
      "		MissionCleanup.add( %this.camera );nn"
      "		// Force it to scope to the client siden"
      "		%this.camera.scopeToClient(%this);n"
      "	}n"
      "}n"
   "@endtsexamplenn"
   
   "@see clearScopeToClient()n")
{
	if(!client)
	{
		Con::errorf(ConsoleLogEntry::General, "NetObject::scopeToClient: Couldn't find connection %s", client);
		return;
	}
	client->objectLocalScopeAlways(object);
}

...

DefineEngineMethod( NetObject, clearScopeToClient, void, ( NetConnection* client),,
   "@brief Undo the effects of a scopeToClient() call.nn"

   "@param client The connection to remove this object's scoping from nn"
   
   "@see scopeToClient()n")
{
   if(!client)
   {
      Con::errorf(ConsoleLogEntry::General, "NetObject::clearScopeToClient: Couldn't find connection %s", client);
      return;
   }
   client->objectLocalClearAlways(object);
}

I'm still not entirely clear on how I'll create a detection around the player (radius) to call each method if the object is within view distance...but I'm slowly getting there! Cheers!

*edit: In hindsight, clearScopeAlways() is still useful so you can clear the scope to all connections so that items no one is within range of are not scoped for no reason...So if the other methods are called properly per connection you are essentially optimizing the bandwith usage. Just gotta figure out that view distance method to scope/unscope :D

#38 There's more...I also just discovered why Vince's resource doesn't work in the case of TSStatics. TSStatics are enherited from SceneObject, and so are GameBase objects (from which ShapeBase objects are derived, which is what Vince's resource adjusts). So for the ghosting resource to be complete, it would make better sense for the changes to take place on a SceneObject level instead of a ShapeBase one. I found some relevant code that I think can lead to a solution:

sceneManager.cpp - Ln547

void SceneManager::scopeScene( CameraScopeQuery* query, NetConnection* netConnection )
{
   PROFILE_SCOPE( SceneGraph_scopeScene );

   // Note that this method does not use the zoning information in the scene
   // to scope objects.  The reason is that with the way that scoping is implemented
   // in the networking layer--i.e. by killing off ghosts of objects that are out
   // of scope--, it doesn't make sense to let, for example, all objects in the outdoor
   // zone go out of scope, just because there is no exterior portal that is visible from
   // the current camera viewpoint (in any direction).
   //
   // So, we perform a simple box query on the area covered by the camera query
   // and then scope in everything that is in range.
   
   // Set up scoping info.

   ScopingInfo info;

   info.scopePoint       = query->pos;
   info.scopeDist        = query->visibleDistance;
   info.scopeDistSquared = info.scopeDist * info.scopeDist;
   info.connection       = netConnection;

   // Scope all objects in the query area.

   Box3F area( query->visibleDistance );
   area.setCenter( query->pos );

   getContainer()->findObjects( area, 0xFFFFFFFF, _scopeCallback, &info );
}

According to this, sceneObjects are supposed to be scoped based on view distance...which makes me wonder why I'm not seeing this happen.

sceneObject.cpp - Ln665

void SceneObject::onCameraScopeQuery( NetConnection* connection, CameraScopeQuery* query )
{
   // Object itself is in scope.

   if( this->isScopeable() )
      connection->objectInScope( this );

   // If we're mounted to something, that object is in scope too.

   if( isMounted() )
      connection->objectInScope( mMount.object );

   // If we're added to a scene graph, let the graph do the scene scoping.
   // Otherwise just put everything in the server container in scope.

   if( getSceneManager() )
      getSceneManager()->scopeScene( query, connection );
   else
      gServerContainer.findObjects( 0xFFFFFFFF, scopeCallback, connection );
}

I'm still investigating, but again I'm not fluent with C++. I think someone like Vince could look at this and make it work :) Here's hoping, at least...Cheers!

#39 I'm pretty sure Vince's resource should operate on all ghosts, because it alters the distance used by scopeScene. Also, TSStatics are not a kind of ShapeBase, they're a kind of SceneObject. The hierarchy looks something like this:



(I stuck a couple of vaguely related classes in there for context.) So TSStatic and ShapeBase are both SceneObjects, but TSStatic is not a ShapeBase. But, like I said, it shouldn't matter in the case of Vince's resource.

Ooh, thanks for reminding me of scopeToClient and clearScopeToClient. I wonder if that will be able to overcome this sticky ghosting issue. Though I wouldn't be surprised if calling clearScopeToClient didn't fix the underlying issue - it might tell the engine not to always scope that object to that client, but the engine might decide that it should anyway for the same reason it's deciding to now.

Same with clearScopeAlways, I think - it might clear the ghost from all connections, but the object might be re-ghosted anyway if the engine's scoping rules decide it should be.

#40 Well, about the TSStatic enheritance...that's basically what I just said. Maybe I worded it wrong, but your enheritance diagram is exactly what I was trying to say...:)

/shrug

Quote:
TSStatics are enherited from SceneObject, and so are GameBase objects

Anyways, yes, about clearScopeAlways()- I have noticed that this is what's happening: I call %obj.clearScopeAlways() on creation of the objects. So long as those objects are out of viewDistance, I'm seeing no ghosts from them. As soon as I approach them so that they are within viewDistance they begin to accumulate, never to be unghosted.

About Vince's resource, I dug through it line by line and here's the part that shook me for a loop: The only place I'm seeing the viewDistance being used is in shapeBase.cpp, besides just some variables being declared elsewhere. In other words, I don't believe the viewDistance is being altered on a sceneObject level. I may be looking at it wrongly, although looking closely at the resource I'm not seeing where it would affect SceneObjects specifically. If it's only affecting ShapeBase objects, it could exclude TSStatics (because they are derived from SceneObject).

Quote:
Though I wouldn't be surprised if calling clearScopeToClient didn't fix the underlying issue - it might tell the engine not to always scope that object to that client, but the engine might decide that it should anyway for the same reason it's deciding to now.

That sounds about right Danny. I think you're seeing what I'm seeing here. Basically, I might be able to clear it...but what really needs to happen is these TSStatics should automatically be scoped/unscoped based on view distance as I believe they are supposed to. This is a pretty big bug tbh, a large level could have simple objects ghosting to players across the map for no reason. By default they do; even if the player spawns across the map he's being ghosted a simple pickup item he hasn't even come into contact with yet.

edit: I'd also like to clarify that I have no intention of discrediting any of Vince's (much appreciated) resources. Quite the contrary, I only feel that it is possible to improve upon them. As indicated above, it is obvious that both ShapeBase and TSStatics derive from SceneObject. I was merely wanting to point out that perhaps a 'wide pass' to scope all of them in one fell swoop could be an improvement. Much respect for Vince and his team, and all that they continue to aspire to :D

edit2: Another thing to consider is SceneObject may not be the best way to approach this since things like ScatterSky, Sun, etc. are affected. Perhaps some sort of scoping specific to the TSStatic is in order.