GetUpdatePriority problem - jumping/jerking AI ships

#1 It's intended, and you are correct in that it has to do with update priority.
I'll post a link or two when I get back from work.

Edit: I should learn how to read the whole post before opening my mouth.

Quote:
Just tried adding a AIconnection as the control object, but the same is observable. Ships jerking.
Additional bonus info is, that it seems to only happen with more than 1 ship!

If this is apparent with AIConnections too then you can disregard my above comment. Sorry about that.

#2 Hehe Stefan. I was sneaky and did the test after I wrote the initial post. Realized that I hadnt tested that particular case, and that someone would bring it up :-D

I might have done it wrong though!! So here is what I did:

echo(aiConnect( NameGenerator::getRandomTeamName("UK") ));

Never mind the NameGenerator part. The id I got back I fed into my test spawn code

spawnTestBattleship(2464, "0 0 0", "UK");

function spawnTestBattleship(%client, %position, %team)
{
	%aiship = new Ship() {
		dataBlock = American_Battleship;
		AIControlled = true;
		team = %team;
      receiveSunLight = "1";
      receiveLMLighting = "1";
      useCustomAmbientLighting = "0";
      customAmbientLighting = "0 0 0 1";
      scale = "3 3 3";
      client = %client;
	};
	MissionCleanup.add(%aiship);
	%aiship.setTransform(%position);
	%aiship.setTeam(%team);
	
	%client.player = %aiship;
   %client.setControlObject(%aiship);
}

I think thats 100% correct, but who knows if I missed a little bit!

The ships are a derivative of shapebase - not a vehicle based class. So it should take the default update priority.

It seems very wrong that it jerks while looking at the object - but not when not looking at it.

And as I wrote somewhere above, I've tried both values around 0 and values around 1 (even 5 and 10, but that froze the engine) with the same result.

#3

Quote:It seems very wrong that it jerks while looking at the object - but not when looking at it.

sure does. ditto that it jerks when close to it, and not when far.

#4 I just know that there are issues with AIPlayers (warping, jitter) in certain sitations that do not show when using a connection of some sort. Ie: AIConnection/Player and regular Players work just fine.

What happens if you increase the speed of which the ships travel? Does the warping go away? I guess not, but it might be worth a try.

#5 I have been seeing the exact same thing for quite some time now.

Perhaps collectively we can come up with a solution.

In my case, I noticed that when I am only looking at 1 vehicle controlled by an AIPlayer, the movement is smooth and beautiful. However, once another AIPlayer controlled vehicle appears on the screen I definitely see the herky-jerky movement. Changing the speed of the vehicle (faster or slower) does not mitigate the problem.

Maybe someone at GarageGames has an idea?

#6 This helped a little bit. It's still jerky just not as badly. Please try this and let me know how well it works for you.

In gamebase.cc find the function GameBase::getUpdatePriority.

Look for the section:


// Weight by interest.
F32 wInterest;
if (getType() & PlayerObjectType)
wInterest = 0.75f;
else if (getType() & ProjectileObjectType)
{
// Projectiles are more interesting if they
// are heading for us.
wInterest = 0.30f;
F32 dot = -mDot(pos,getVelocity());
if (dot > 0.0f)
wInterest += 0.20 * dot;
}
else
{
if (getType() & ItemObjectType)
wInterest = 0.25f;
else
// Everything else is less interesting.
wInterest = 0.0f;
}


Change the line

if (getType() & PlayerObjectType)

to

if (getType() & (PlayerObjectType || VehicleObjectType))

then recompile.

#7 Thanks Robert

Doesnt help me, as I already got the ships configured as

mTypeMask |= VehicleObjectType | PlayerObjectType;

But I also saw that part of the code. It didnt seem to contain anything about a connection need. I'll try tonight to dig into a more scientific approach on the priority numbers. Until now I just tested the boundaries (0.01 and 0.99 + the default + some unrealistic numbers). Will dump out the number for the ships when barely outside the screen, and while on screen. Might give some clue.

But it is super nice to hear that someone else is having this issue too. There must be a bug hidden somewhere.

#8 Thomas,

Absolutely. Let's kick this around and see what we can come up with. I ran some comparisons between having my changes and not having my changes and there is a perceptible difference. I think you aren't seeing anything because your mTypeMask will cause if (getType() & PlayerObjectType) to return a true value which accomplishes the same thing. Thus the wInterest variable is being set to 0.75f which is currently the highest value in the function.

Wonder what would happen if we set wInterest to > 0.75f?

I'll try that and let you know.

Robert

#9 OK - did a fast test (couldnt wait till tonight).

When the ship (2 on screen, both moving) is moving smoothly on the edge of the screen, its updatepriority is hovering around 0.62 (changes are on the 4th or 5th digit).

When the ships (both of them) are jumping around, the updatepriority value is either 0.72 or jumping up to 1.2-1.4.

So while moving smoothly it has a fixed value of rounded 0.62. I wonder if its the doubled priority (for whatever reason its doubling) that is causing the jumping motion. As if its "stuck" at the high value, and then being updated properly at low value.

Will do a few tests more where I try to input fixed values from above to see if I can provoke the same update behaviours)

------

Little more. I now made a little script interface for testing, and it confirms the weirdness of this.

When there is 1 ship in the scene, then it doesnt matter at all what I set as a priority. 0 -> 10 - the ship moves fine without a hitch.

Throw in another ship that has a priority of 0.5, then the weirdness starts. The 0.5 ships _always_ stopped dead in its track when viewed upon.

When I set the priority on the initial ship to 1.2->1.5 (didnt try higher) it started to update perfectly. But the 0.5 ship was stopped.

There is nowhere in the code that I could find, that it said anything about 2 objects of same class being treated specially. So why should 2 ships make a difference to 1?

Another little curiosity - try to open the world editor while your shapes are visually stopped moving. You can see the position being moved smoothly at all times. Now this might be the interpolation kicking in. Not sure - just a curious observation.

#10 K, let's talk about some of the things that are in your packets being sent from server to client:

Part A: ACK/NACK, Sequencing info : Several bytes used to contain information about the packet, and the server/client coordination itself

Part B: Control object update -- what is written to the bitstream with the control object's writePacketData()

Part C: Net Events -- all information required to pass any NetEvents to the client.

NOTE: Stock Torque chatting is all script driven net event based. It works for small fps games (and even rather large ones), but when it comes to 100's of users this needs a different system. I'll explain why in a moment.

Part D: Ghost Updates, in order of netPriority. If you run out of room before a particular object that is ghosted and in scope gets it's update sent, then that update will be skipped, and hopefully stuffed into the next packet to be delivered.

Ok, now keep in mind that in stock, you only have 200 bytes to deliver all that information, every network tick (200 ms).

Now, on to describe what is going on for you:

Short Answer

Your update packets are sending way too much data, causing even just one more object to cause overflow and skipping. Since every time a ghosted object's update is skipped, it's netPriority is increased for the next packet, this explains why you see the fluctuating priorities--it's the system working as intended.

In addition, you're interpolation/extrapolation (physics basically) is whacked. This is proven by your "curiosity" (which is actually a critically important troubleshooting step!). Due to the nature of the world editor, when you have the world editor active, you are actually rendering server objects. Yes, that's what I said--you are no longer rendering client side positions, but are rendering server side positions instead--and note they are moving smoothly as you said, which points to your client side objects not properly interpolating and/or extrapolating.

Long Answer

A) ok, but why are your packets overflowing? It's only 1 object!

--actually, it's a minimum of 3--your two ships, plus your camera--and I can almost guarantee that you are using your very own advancedCamera--a great example resource for camera work, but one of it's few downsides is that it is very much not network optimized. How do I know this? Well, way back when, I wrote the initial implementation of the orbit mode--and I didn't understand networking very well at all! In the last version of the advancedCamera object I looked at (quite a while ago I admit), you and I were violating the best practices for ghost object updates by not following the "bits not bytes" rule...in other words, work on slimming down the information packed into the update stream for advancedCamera::packUpdate(). ANY 32 bit sends, or raw "write(" calls are very unoptimized.

B) Even if I compress the stream well, will this hold up for laggy clients?

--No, it won't, at all. There is actually nothing at all with skipped packets from time to time--it's how Torque Networking takes care of having to deliver bursts of data and still keep the client/server in synch, all the while minimizing bandwith use. However, the ability to skip packets has a critical assumption: your interpolation and extrapolation code must be deterministic, and accurate. In addition, it must allow for proper backstepping to handle re-synchronization when needed.

--check your physics code carefully, both in processTick() as well as interpolateTick(). Note that there are several cases for objects:

--"I"m this client's control object, applying a move directly to the client sim". This is our client side prediction phase. We need to make sure we are managing the delta struct properly, since this allows us to backstep later.

--"I'm this client's control object, applying/interpolating an update sent from the server". Client side interpolation, since I have a current position in my sim, and the server is telling me what my "real" position was a while ago (short time ago, in ms hopefully). I need to backstep, apply the update, and then re-apply my moves again...and all this happens by iteratively calling processTick()/interpolateTick() as appropriate.

--"I'm a client side ghosted object of some server object"--the "second ship" in this case. Make sure again that you are properly interpolating and also can extrapolate, since by definition for this class of networked object, the client is always interpolating, unless packet streams are blocked from the server, when it moves into extrapolation.

Note: Little known fact of Torque Networking: for "normal" ghosted objects (not control objects), the first update sent by the server is saved, but nothing is really done physics wise. Clients wait for a second packet so that we will have two points and a time to use for interpolation. There is then the expectation that we should always have a "future position" that we haven't reached yet with an associated time to reach it delivered as an update from the server.

--"I'm a server object that is a control object for a specific client, and I have a move struct"--physics important, delta struct important, etc.

C) Why did you say that net event based chat systems could be a problem with massive traffic?

--Note that NetEvents are part C of any particular packet sent to a client, while ghost updates are part D. You can fill up a packet completely with NetEvents, and no ghost updates will be sent in that packet. This happens especially with lots of strings that are unique and therefore can't be tagged (chatting). You only would see this really if you had both a lot of net events of whatever type, combined with both a control object update (Part B), and ghost updates (Part D)...which is what happens when you add a second boat.

#11 Couple more notes about your specific test cases:

NetPriority means nothing for a control object when being ghosted to the client that controls it. This case, the update info goes into Part B for trend information.

NetPriority is dynamic, and shouldn't be manually modified from outside the scoping/priority execution stack for anything but troubleshooting/learning. It changes per object depending on the overflow status of the update packets.

What's different about your two objects is not their class, but their control state. One is a control object, and one is a normal ghosted object. In addition, one (the control object) has a camera attached to it.

#12 @Stephen:

Quote:
ANY 32 bit sends, or raw "write(" calls are very unoptimized.

May I ask why NetEvents send their class ID bit as part of a normal raw write/read call, if it's so unoptimized? Wouldn't writeInt/readInt be better?

#13 Because the possibility exists that you may need 32 bits to represent the class ID is my assumption--I've not tracked that specific element of optimization personally.

"Optimized" simply means that you've analyzed the data flow, and the write/read mode you are using is appropriate. For example, if you have a state variable declared in your class such as:

U32 mState;

and this variable can have one of 4 values:
Perfect
Damaged
Disabled
Destroyed

If you use a

stream->write(mState);

Then the networking code will look, see that it's a U32, and use 32 bits to send the data.

However, since we know through game play design that it is in fact a variable that only needs 2 bits (only 4 states), we should actually optimize with something like:

stream->writeInt(mState, 2);

which says "use 2 bits to send this data, you won't ever need more".

In the case of the advancedCam orbit mode for example, I remember specifically having trouble understanding the whole optimization thing, and I would up using raw "write" in at least a few places.

#14 I assumed as much, just wanted to hear you say it :) Thanks and sorry for hijacking the thread. The information provided in here is tons useful though so I'm considering making it a TDN article.

#15 If you have the time that would be great--it fits in nicely to the existing one in the TGE coder area (TorqueNetworking).

#16 Hi Stephen!

Thanks a bunch for the write up! It does point to some interesting things, and definitely worth investigating. One if the code is optimized properly for the least bandwidth possible. If what you say about updates being skipped is true, then that should be easily looked upon.

I'm actually not using the adv camera, as Phil already had the camera stuff coded for Air Ace when I joined. So its "stock system". Unfortunally, as I had hoped to rewrite the camera system due to me also being more knowledable now than previously about the network part. Just needed an excuse, but Air Ace isnt it.

There are a few things that I'm still very unclear about.

The thing that gets me, is why would the 2 ships (neither of them have a control object BTW! Both are fully AI driven - which would both put them into the part D of the packet) jump while looked at - but move nicely when not being looked upon. Sorry if you already answered this in your reply, but my mind is clouded a bit with this problem.

A fast way for me to test the lack of bandwidth/packet size problem would be to increase the packet size the hard way (in code). Just to check.

I'm pretty sure the interpolation works as expected though. I've looked it up/down (I didnt write it - code was made by a contractor based on a prototype), and I cant see anything wrong.

#17 Well, in your specific circumstance, I can just about guarantee that your client side interpolation and/or networking is broken. If a ship is not moving when viewed normally (client side), yet when you enter the world editor and it immediately jumps to a different position and starts moving smoothly, that is 100% guarantee that either:

--network updates are simply not being sent
--network updates are not being applied properly
--interpolation is not operating correctly
--no trend information is sent, so extrapolation cannot occur

Now, you re-mentioned something that may be important: your ships are AI controlled. Making a guess here, you've adapted the basic Kork ai stuff (setMoveDestination, etc) on the server, and it is moving to a destination either directly,or node by node.

Is the ::processTick() on your server objects properly detecting movement and marking dirty bits to pack up an update at all?

The other thing that's possible if you are correct in that on the client, it is literally just not rendering in the right position--and that points to an incorrect renderTransform calculation somewhere. However, unless it's literally being zeroed out or something, the visual effect of that particular bug would be a "hopping" as the client side object moves from a physics tick position and then hops to the next physics tick position, with no render positions in between. Your description doesn't really point to this, but it is possible.

One more debug question: when you enter the world editor and the ship starts moving, does it warp from where you see it on screen to some (relatively) drastic new position, does it simply start moving very smoothly immediately from it's current position, or something else?)

EDIT: DOH! I really need to make sure I do more than scan earlier posts. Just viewed your movie.

Your problem here is that you aren't properly calculating a renderTransform, and/or messing with it at the wrong times. The hitching you are seeing when looking directly at the entire model is certainly the ship moving from phyiscs tick position to physics tick position with no rendering in between at the renderTransforms that should be calculated.

Why it goes away when the centroid of the ship isn't in the view frustrum I'm not sure, but look at your renderTransform usage, and your interpolateTick()//advanceTime() stuff on the client.

It does not actually appear to be a networking issue at all from what I can tell.

#18 Ahhh - I see what you are saying.

I might have to be more specific regarding the world editor. When turning it on, I see the bounding box move! The object itself is still "visually stuck" as before! Quite an important information if I read your postings right!

The AI is a home brew system. I dont like way the aiPlayer is made, so the ships rudders and throttle are controlled from a script based AI. A simple scheduled piece of code that reads the steering behaviour information from the engine and then applies rudder/throttle to match the behaviour result vector.

But you are heads on that this is a network size problem. I just ran through the code, and I found that each turret mounted onto the ship had an unneeded
stream->writeAffineTransform(mObjToWorld);
Commenting this out saves (in the demo case) 6 of these, as each ship has 3 turrets. The ship code sets the position of the turrets manually - as to save network bandwidth and not getting uncoordinated updates on the turrets/ships causing visual lag.

Running it now without those 6 matrices in the packets, the ships sail smoothly.

Thanks for your help BTW!

#19 Okie dokie! Saw your edited post.

You more or less convinced me that this was a network problem though! If it was a problem in the renderTransform, then I would think the error would also show when only 1 ships is there - and regardless of camera position.

But running through the code, I set renderTransform in
processTick
interpolateTick

and the transform in
processTick
readPacketData
unpackUpdate

#20 Interesting...well, with the packet savings you found that implies two things:

--Your interp/extrap code is probably still buggy. Because you can't simply avoid the skipped packet scenario, you have to be able to handle it (or when you get 3 players playing you get the skipping, etc etc).

--I'll back off of my "certain" about the render position, but I think it's a factor for sure. That little skipping/hopping you saw is too consistent to be ignored, although it may not be render position exactly.

I'm also wondering something else now--maybe your warp threshold is too low, which in conjunction with non-determinstic/non-working interpolation would cause your ships to immediately warp to any networked update that was "too far away". Combine that with no accurate interpolation, and it could also explain what we're seeing.

A very rough rule of thumb in my personal testing--you should be able to skip 5 or 6, or in some game types even 20 packets for a particular update and still be able to accurately extrapolate a 1st order position (with no changes in trend information of course). Worst case, you ship would be off position a small amount, and the synchronization process would correct, at worst causing some form of spline-ish path to the new position.