Ghosting Limits

#41 To be fair, most pickups are Items, which are ShapeBase and not TSStatic. But you're still correct. Also, that line in ShapeBase modifies the CameraScopeQuery, correct? Which is then passed upwards to the SceneObject method, which uses the CameraScopeQuery to scope the scene. So the value set by ShapeBase ends up being used.

#42 Ah yes, I noticed the reference to the item wasn't really a good example. I figured you'd get onto me :)

Thanks for the clarity around the passing of the CameraScopeQuery, that makes a lot more sense I think. I mean, of course, I'm new with the source and surely the vets understand these things much clearer!

TSStatics
TSStatics being ghosted across the scene is sort of a gray area really, and here's why I think so now: It is easy enough to just make them not ghostable at all if that is the intent. Additionally, to be honest if I need any additional functionality out of that shape I will probably need it to use a datablock...as in (ta da!) it may end up being derived from ShapeBase after all.

StaticShapes
I reported to you incorrectly about these before. I had actually tested ShapeBase and it introduced a performance hit. Get this, I finally set the datablock up correctly for StaticShapes and actually now have an improvement in performance! So moving forward, in my case the StaticShape is working better (probably due to the same datablock being called for the shapes, but just a guess). Still not seeing any difference in the ActiveGhosts though, which leads me to...

NetGraph
I have been monitoring this 2 ways: Using the NetGraph(hotkey 'n') and the command ServerConnection.getGhostsActive();. Both are yielding the same results, the ghosts go up but don't come down. Is there perhaps an error in my testing method? Or environment?

#43 I'm going to try to reproduce this in a barebones environment as soon as I can. I always forget about the netgraph, I should add that to t3d-bones.

EDIT: okay, I can reproduce your issue with a StaticShape. Here's the branch. Next test: dynamic objects like an AIPlayer.

EDIT: okay, so AIPlayer is removed from scope as soon as it goes out of visible distance.

#44 Aha. So yes, if an object doesn't call setMaskBits on the server, it'll just stay in scope indefinitely. To force an object to unghost, you've gotta make a change. For example, in the branch I linked above, if I run up to the cube so it starts ghosting, then run away out of visible range, then in the server console call `Cube.position = "1 20 1";`, the ghost count will immediately decrease as the ghost gets destroyed.

That's very interesting.

#45 Okay, at least we've confirmed I'm not going crazy (depending on who you ask :P). lol. Yea that's interesting for sure. Sorry, I dug through your branch link but didn't see any familiar files...haha, man you really do mean 'bones' right? Wow, that's amazing, I'd like to learn more about bones and potentially adding source to that as a base...but anyways before I get off course...

I did make some progress around this, check this out:

TSStatic
Okay I figured out how to dynamically ghost/unghost TSStatics. Doesn't matter if you are looking right at it or it's far away.

// this will remove it from the list of active ghosts and hide it 
%obj.setHidden(true); 

// this will add it to the list of active ghosts and show it 
%obj.setHidden(false);

So I've managed to actually solve the original quandary around TSStatics. This is with no source changes. Woot.

StaticShape
Now that we're on the subject, though, yea the behavior of StaticShapes seems funky. I can't use the same call to %obj.setHidden() on the StaticShape (you can use it to show it, but not to remove). I did confirm all of your observations Danny, managed to have the StaticShape unghost out of viewDistance after making a change to the object. Interesting indeed.

Edit:
Figured I may as well post the source that led me to the solution around TSStatics. Maybe it can help someone a bit more experienced figure out the StaticShape problem. The comments are particularly interesting here:

sceneObject.cpp - Ln509

void SceneObject::setHidden( bool hidden )
{
   if( hidden != isHidden() )
   {
      // Add/remove the object from the scene.  Removing it
      // will also cause the NetObject to go out of scope since
      // the container query will not find it anymore.  However,
      // ScopeAlways objects need to be treated separately as we
      // do next.

      if( !hidden )
         addToScene();
      else
         removeFromScene();

      // ScopeAlways objects stay in scope no matter what, i.e. even
      // if they aren't in the scene query anymore.  So, to force ghosts
      // to go away, we need to clear ScopeAlways while we are hidden.

      if( hidden && mIsScopeAlways )
         clearScopeAlways();
      else if( !hidden && mIsScopeAlways )
         setScopeAlways();

      Parent::setHidden( hidden );
   }
}

I'm glad, at least, that my original hypothesis about SceneObject being the root of the problem with TSStatics was accurate :)

#46 Be aware that setHidden will affect all clients. Will do more testing tomorrow...

#47 That dawned on me finally after doing a little more testing. I appreciate all your help Danny, here's hoping for an easy fix :)

#48 Note that increasing the number of ghosts allowable will allocate more memory in these fixed-size arrays. Need to check what the size of those structs is to figure out how much impact this would have.

Really, the whole array of ghost info objects is a little hairy. Not sure why it doesn't use a linked list, or even a Vector.

#49 Vector gets my vote.

The array was probably a classic case of "there will never be a need for more than this many." Static allocation, all that.

For TSStatics, maybe a modification of the ghost system is in order? These objects might come or go, but don't move, right? Can you add or destroy them in the game? Isn't there some better way to simply make them "part of the terrain" in essence? Dunno, just thinking on paper....

#50

Quote:For TSStatics, maybe a modification of the ghost system is in order?

That's the million dollar question Richard, and pretty much the center of the problem. Just seems to me if AI can get culled automatically out of view distance so can TSStatics. I mean, wow, the AI are far more important to ghost yet they can come and go from the ghost list? Surely there is a way to do it...without an overly long thread.

Quote:Isn't there some better way to simply make them "part of the terrain" in essence?

Indeed there is. The terrain itself is composed of geometry that never gets ghosted. Also the foliage, etc. I can have an ton of meshes (generated with fxShapeReplicator iirc) and never have a single one ghosted. Really I just think these objects turn off the 'ghostable' flag. The problem is, in a block-based game with multiple clients involved you'll have an array of block objects. Those objects may be visible or not, and may change material. Pretty basic, yet I do believe they still need to be ghosted to each client when they see it.

But only when they see it...that's the problem. I know it's possible, with resources like Vince's culling players based on viewDistance. It's just a matter of time before someone figures it out I guess. The thing that's frustrating about the whole bit is by default Torque is supposed to drop ghosts that are out of view distance. Yet what it is actually doing is ghosting static objects across the map for no reason :/

#51 I think I see the root of this - NetObject::onCameraScopeQuery(NetConnection *cr, CameraScopeQuery* /*camInfo*/). The camera query is dropped. How does the NetObject know if it's in the query data or not? How would it know it's no longer in scope?

I think perhaps the version in ShapeBase should be the version that's actually in NetObject - "scope the universe by default" is the mantra scattered throughout the code, but is that really good? I suppose we'll have to decide if the more complete search is worth it, but I think it probably is. Maybe we can find a way to optimize the query, too.

#52

Quote:
The thing that's frustrating about the whole bit is by default Torque is supposed to drop ghosts that are out of view distance. Yet what it is actually doing is ghosting static objects across the map for no reason :/

It doesn't "drop" them, it stops sending updates for them ( or at least its supposed to ). Think about it from a multiplayer standpoint. If you have all your players on the left side of the map you're still going to have a list of ghosted items that contains everything for when a player walks over there. You just filter it for each player. The issue you're running into is reaching a maximum number of ghost ID's assigned. Even if you aren't sending updates about items on the right side to the client on the left, they'll still have the same assigned ghost IDs because it's cumbersome to keep dropping and reestablishing ghosts with different ghost IDs (you'll run into sync issues pretty fast, not to mention performance)

Are these cubes ever going to move or utilize network functionality at all? If not it shouldn't be too hard to build a non-networked lightweight cube class.

Also, you say you're doing isometric. With a fixed camera angle and distance you should be able to easily calculate how much of an area is visible. You don't have any of the weird depth issues that come with using standard 3D camera angles. You can just ruthlessly not draw anything outside of your fixed view area due to your isometric angle. Take advantage of this. Something like occlusion culling is a waste for your case if you're truly doing a fixed isometric camera angle.

#53 What about objects that are not in scope for anyone connected to the server? Do they really need a ghost? Shouldn't those ghosts that are not GhostAlways be flushed?

From Jesse's description the cubes can be added and removed from the scene and change materials, so they would need some sort of network support I should think. I do see a possible downside here though - if the material has changed then the player may see it "pop" when the ghost is updated as it comes back into view.

I do like the idea of taking advantage of a fixed camera angle by just chopping everything that's outside of the frustrum.

#54 Richard, onCameraScopeQuery is called on the control object only, and is responsible for starting the query that decides what other objects are in scope. Please see this post - it's working exactly as it should. NetObject's onCameraScopeQuery never gets called because your control object is always a SceneObject, which doesn't call the parent method. It's not the problem here.

The 'problem' is that only ghosts who have an update to send to the client are updated at all. The only way for a ghost to be removed from the client is for it to be updated while out of scope on the server, and then a 'kill' packet will be sent that destroys the client's ghost. That's why AIPlayer goes out of scope when it leaves visible distance - because it's calling setMaskBits fairly regularly, which causes the server to try to update all clients, but in doing so realises that it's out of scope for some clients, and sends the kill packet.

So, a simple workaround for now would be to schedule rolling calls on every single terrain cube that cause it to send an update to the client. Just set %obj.position = %obj.position or something, should do the trick. Call this in a schedule repeatedly on every terrain cube, and you should see them drop off the net graph. Play with the schedule frequency, and try to break up the loop over multiple schedule calls (i.e. every call triggers 15 objects or whatever) so reduce the CPU and potential network load.

You could also get smart and just update objects when a player moves out of their range. Maybe divide the world into trigger bins, and when a trigger is empty, update all the objects inside it. Or something. Needs more thought.

That should solve your issue for now. But maybe we should look into long-term ways to solve this, or have a compile-time opt-in solution. At the moment, this approach obviously makes sense from the perspective of how things worked in Tribes. The map contained several static objects, not enough to cause any max ghost count issues, and as you walked across the map and encountered each static object, you'd get it across the network once, and then you'd have a ghosted copy for the rest of the game, and not need to spend any more network traffic on it.

It wasn't built to accommodate thousands of these objects. That must have, to some extent, inspired the replicator classes. Those replciated shapes aren't TSStatics or StaticShapes - they're their own special client-side-only shape class. Only a single object is ghosted - the replicator - which then creates all the geometry on the client side.

The ideal solution to this cube terrain problem is to do something like that, of course, but maybe there are other valid reasons you'd want to have thousands of static objects and not have them all always ghosted.

Andrew - ghosts are indeed actually dropped and completely destroyed when they update out-of-scope. Next time the object is encountered it's not guaranteed to have the same ghost ID. But, as I said above, if object's don't update when out of scope, then their ghosts are never destroyed.

#55 Ah, the control object... lol - should have caught that. I'm going to have to start actually stepping through this; just browsing is not enough....

#56 Nice, a lot to consider here. I really appreciate all the input, you guys are great! Alright, from the top:

@Andrew Mac:

Quote:Are these cubes ever going to move or utilize network functionality at all?

I believe so, yes. The issue is the cubes will be able to be added/removed by each client. If clientA removes a cube, clientB should be able to see that cube was removed.

Quote:
With a fixed camera angle and distance you should be able to easily calculate how much of an area is visible. You don't have any of the weird depth issues that come with using standard 3D camera angles. You can just ruthlessly not draw anything outside of your fixed view area due to your isometric angle. Take advantage of this.

This is interesting to me. While it may not be a complete solution, it surely may end up being part of it. Back to ghosting, though, this is where the problem comes up. Even if only the cubes being shown at this angle are rendered, the problem is that once they are rendered they are added to the ghost list. They never leave that list, so if the player viewing from this angle travels across the map he's just accumulating ghosts until he hits the cap. The entirety of the problem lies with the ghosts not being removed client-side. If that were to happen, I'm pretty confident I can solve all the rendering problems(isometric or otherwise, as this ghosting problem is honestly all that stands in the way of a full 3D option as well).

Quote:Something like occlusion culling is a waste for your case if you're truly doing a fixed isometric camera angle.

Not necessarily. The cubes aren't just a flat layer of cubes on a plane. They go down, down, down. So if the top layer were to occlude all of the layers below...that would be fantastic! Also with only the top layer showing at all times, we're reducing the potential ghosts and cubes in the first place. Instead of rendering thousands of cubes below.

@Richard: Pretty much nailed it. I approve this post :) Er'body look out! Richard's on the hunt!

@Daniel:It's obvious you have been researching. That's much appreciated, I honestly feel that there will be some resolution to this eventually. Until then, I'll keep testing and trying different things.

I did try something similar to what you've suggested, but I hit a snag Danny. What I did was I made a function that performs a containerRadiusSearch around the player. I am able to perform operations on the objects caught inside this container (and that are in LOS, of course :P). The problem is, though, I'm trying to do exactly the opposite. I'd like to have objects that have been within this radius container to call a (workaround) function once they leave that container. Something that should be happening automatically by viewDistance though, to be fair.

Quote:You could also get smart and just update objects when a player moves out of their range.

How is an object's 'range' determined? Because by default I'm not seeing the field...as a matter of fact by default an object's range seems to be the entire map (lol). I think creating so many triggers might end up introducing a lot of overhead. I would love to just create a function that said: if %client moves out of %range of %obj, stop scoping %obj. But that's not quite as cut and dry as it sounds. Trust me, I've devised some super complex maths and algorithms in script to generate the cubes etc. in the first place. Not to mention the containerRadius loop, XML saving of said cube array, etc. I'm honestly not just reaching into thin air for an answer, I'm giving it a solid attempt...but being held back by a silly scoping method being defined in Torque's networking code.

Thanks for all your input guys, you continue to give me more things to consider. Definitely a good thing, as I'm coming up short so far. Cheers, here's to hoping that after all this is said and done we can all benefit from an improved ghosting functionality. I do feel there is room for improvement when we have Players dropping from scope and statics not.

edit: I wanted to drop this here, just to clarify. Honestly it's not the number of objects that's the issue here. I can make a million cubes if I wanted. By use of some clever culling, lod's, and script this really is a non-issue. The only issue is the ghosts not leaving scope client-side.

#57

Quote:How is an object's 'range' determined? Because by default I'm not seeing the field...as a matter of fact by default an object's range seems to be the entire map (lol).

Sorry, I'm not referring to an actual built-in thing here. You have to determine the object's range! It should of course be based on the visible distance (half that is the range from the camera at which ghosts appear), so start from there. Break the world up into bins. Those could be Trigger objects or whatever. When a player leaves a bin (which Trigger will give you a callback for), then it calls an update function on every cube inside it to trigger ghost deletion on the clients they're no longer relevant to.

Of course you'd want to do something a little more complicated, and you'd have to overlap bins, I think, to ensure this actually worked. It's not a trivial piece of work :/.

Also, unrelatedly, consider whether you need to create a solid mass of cubes for underground, or whether you can just create the top layer of cubes, and dynamically add cubes as you start to dig. Won't solve your networking issues, but it's an optimisation. Of course occlusion culling would stop them being rendered, but it won't take them out of the physics simulation, etc.

#58 It was a poor attempt at sarcasm on my part :P Of course I would need to do the legwork on finding the range...Still seems like a ton of trouble for something that could be optimized in the source's networking code. If the statics could drop ghosts outside of view distance like players do...none of that would need to happen. It's obviously possible, I'll just have to keep digging or hire a programmer (sigh).

Sorry Danny, totally not trying to come across as rude man. I can see you're trying to help, and you've given me more hope that this can be fixed than anyone else has managed to. But now the thread is turning again (as most do) into a discussion on how I can optimize it myself when there is an obvious flaw in the source. I mean, seriously, now we're talking about cluttering up the entire grid with triggers to force updates to trigger a ghost deletion that should just occur by default. In script no less, when a paid programmer would probably fix this the same day I told him about it within a few lines of source code.(Ahsahn drop me an e-mail lol)

About the cube generation, again I've got all this part solved. All I need are the ghosts to drop outside of view distance. This may sound arrogant but it's the truth.

Quote:I wanted to drop this here, just to clarify. Honestly it's not the number of objects that's the issue here. I can make a million cubes if I wanted. By use of some clever culling, lod's, and script this really is a non-issue. The only issue is the ghosts not leaving scope client-side.

Please don't take offense to any of this man, I respect you for all you do to continue to improve Torque. I'm more inclined than most to do the work, read the docs, solve the problem. This is out of my hands though; there's no reason I should have to do all that in script. When developing with an engine, the engine should be working for me not against.

Sigh, I've been an avid fan of Torque for quite some time. I've stuck it out through thick and thin, and pushed myself to do better time and time again. In the face of other, flashier, time-saving engine options I've opted to just stick it out and get better at what I'm doing with Torque. That's definitely the case. I've improved. I can do fantastic things with the engine now, things I'd only dreamed of just a couple years ago. I've even gone as far as to help artists at other studios improve their products specifically for Torque use. Torque's networking has always been one of the strongest points for the engine, and now that even this is failing me I'm losing faith. I thought that maybe, just maybe, my fellow community would be able to come through.

Anyways, it's late and I've been pushing myself way too hard on this. Thanks for enduring.

#59 Test this method out:

function SimSet::refresh(%this, %period, %objectsPerIteration) {
   %this._refresh = %this.schedule(%period, refresh, %period, %objectsPerIteration);

   if(%this._refreshIndex $= "") {
      %this._refreshIndex = 0;
   }

   %i = %this._refreshIndex;
   %max = %this.getCount()-1;
   for(%j = 0; %j < %objectsPerIteration; %j++) {
      if(%i > %max) {
         %i = 0;
      }
      %obj = %this.getObject(%i);
      %obj.position = %obj.position;
      %i++;
   }

   %this._refreshIndex = %i;
}

You can see it in action here. Call it on the group or set your terrain is in. %period defines in MS how often the function will be called, and %objectsPerIteration determines how many objects will be refreshed per function call. So to calculate how long it will take to refresh the entire group, it's (%group.getCount() / %objectsPerIteration * %period / 1000) seconds.

That should get you started! A more sophisticated solution would be lovely, but the engine really isn't designed to do what you're setting out to. You can tweak the values of %period and %objectsPerIteration so it doesn't cost you too much performance. Ideally this could be handled at the engine level, and be more discriminating about sending packets - obviously you only need to make sure a ghost is updated once after it's left scope, and only for the client whose scope it left. But there's a whole lot of crazy stuff that goes on with Torque's networking to make it reliable and performant, which I'm not game enough to tamper with. This is why, for example, we're stuck with the mask bits in the first place, and can't, for example, add sub-bits per object. The engine stores these mask bits per-client, not per-object, and will go to great lengths to roll them back if packets are dropped for one client connection. So I wouldn't be surprised if any modification along the lines of dropping ghosts would have to touch that code, or at least work with it in mutual understanding - understanding which I think few people in the community have these days.

I know I'm coming off as a bit of a killjoy, but if something's worth doing, it's going to take effort to do it. Torque's got a long way to go before it pleases everyone, but by having you do this, we're running up against its limits and it'll only make the engine stronger. It's prompting people to look into issues like this, assumptions that were made when the engine was first designed, and question them, and hopefully improve on the status quo.

Also, frankly, don't fall prey to the sunk costs fallacy. If learning to use Torque is teaching you to program, that's a skill that will transfer to any other engine you use. And doing something like programming a bin/update system will really test you and ensure you know your stuff. Any game will require solving hard problems like this, if you're not retreading ground that's been trod before.

And finally... keep at it! You've come a long way, and you're making great leaps. Don't let issues like this kill your enthusiasm and drive! There's always a workaround, and it's persevering and going the hard yards to implement things that separate the good devs from the great.

EDIT: you probably understand that I don't think this is something that will be fixed with 'a few lines of code', at least not by someone who doesn't deeply understand the ghosting model. I could definitely be wrong.

#60

Quote:
I know I'm coming off as a bit of a killjoy, but if something's worth doing, it's going to take effort to do it.

Sounds like my favorite:

Quote:
Non-trivial results usually require non-trivial effort.