After increasing packet rate, critical error occurred
by Mquaker · in Torque Game Engine Advanced · 07/24/2007 (11:23 pm) · 6 replies
I have increased packet rate to reduce loading time and to make our response speed faster.
(The speed I found faster is much faster than what you can see on Starter.FPS)
At first, it seems to have no problem at all.
But after few test games I could notice that there is a critical error after increasing packet rate!
The screen gets frozen randomly and to get out of the game, I needed to shut down the game with the window command or sometime had to restart the whole computer. It doesn't happen frequently and there isn't any rule on this but it happens randomly.
To explain what I did exactly,
I have changed the below:
checkPacketSend(false) -> checkPacketSend(true)
After changing that my loading and response speed got noticeably faster.
But somehow client sides gets frozen screen and cannot normally exit game after that happened.
I believe this is happening after increasing packet rate. My dedicate server side is running infinite loop with the full usage of CPU.
My question is "by doing what I have done, can this kind of error happen?"
If so, is there any way to solve this?
And also if you have other thoughts please let me know!
(The speed I found faster is much faster than what you can see on Starter.FPS)
At first, it seems to have no problem at all.
But after few test games I could notice that there is a critical error after increasing packet rate!
The screen gets frozen randomly and to get out of the game, I needed to shut down the game with the window command or sometime had to restart the whole computer. It doesn't happen frequently and there isn't any rule on this but it happens randomly.
To explain what I did exactly,
I have changed the below:
checkPacketSend(false) -> checkPacketSend(true)
After changing that my loading and response speed got noticeably faster.
But somehow client sides gets frozen screen and cannot normally exit game after that happened.
I believe this is happening after increasing packet rate. My dedicate server side is running infinite loop with the full usage of CPU.
My question is "by doing what I have done, can this kind of error happen?"
If so, is there any way to solve this?
And also if you have other thoughts please let me know!
About the author
visit : http://www.quakedev.net
#2
What's going on is that the Torque server expects moves at a fixed and constant rate from the client. Disabling that code is now sending moves at a variable rate (as fast as possible), and flooding the server's move input window. The move credit system (an anti-cheat system) is eventually shutting you down, and probably should do it cleaner, but in general since you're removed the "play at the same fixed rate" mechanism, one or the other is eventually over-flooding the receiver.
There are quite a few ways (with good understanding of the entire network process and systems) to optimize load times (no longer sending every datablock every time and using a checksum system to make sure the client has what they need is just one), but this isn't the way to do it.
07/25/2007 (9:47 am)
For the record, you aren't "increasing packet rate" with that code change, you are disabling the timing mechanism that keeps client/server synchronization stable, allows moves to be generated once per physics tick so they are a 1 to 1 correspondence with server side physics ticks, and most importantly (which is what is probably causing your problem) you are removing the mechanism that keeps Torque Clients "playing well with the server" when it comes to upstream bandwidth and move credit tracking.What's going on is that the Torque server expects moves at a fixed and constant rate from the client. Disabling that code is now sending moves at a variable rate (as fast as possible), and flooding the server's move input window. The move credit system (an anti-cheat system) is eventually shutting you down, and probably should do it cleaner, but in general since you're removed the "play at the same fixed rate" mechanism, one or the other is eventually over-flooding the receiver.
There are quite a few ways (with good understanding of the entire network process and systems) to optimize load times (no longer sending every datablock every time and using a checksum system to make sure the client has what they need is just one), but this isn't the way to do it.
#3
Including every code related with has the same rate and we made the below code to true:
Only for the server side which will allow server to send packet to client very quickly.
You have mentioned that the Torque server expects the fixed and constant rate from client, but my client is just doing that for sure. Also you have mentioned that the move credit system is shutting client down because it thinks that client is using a cheat system, but since I changed it only for the server side the client won't send too much of an information to overflow. How should I understand this? Is this the same case both sides? Sorry that I am confused but could you please make it clearer?
When we have more players in the mission (up to 32), its frequency gets higher, so I'm guessing it may have something to do with it too.
And one thing I definitely want to hear answer to is: where is the code that shuts down the client? You mentioned it kicks the client which the server thinks the client is cheating with a hacking program. And where is the code that senses the cheating program?
08/01/2007 (12:12 am)
My packet rate is up to:void NetConnection::checkMaxRate()
{
// Limit packet rate to server.
if(gPacketRateToServer > 32)
gPacketRateToServer = 32;
if(gPacketRateToServer < 8)
gPacketRateToServer = 8;
// Limit packet rate to client.
if(gPacketRateToClient > 32)
gPacketRateToClient = 32;
if(gPacketRateToClient < 1)
gPacketRateToClient = 1;
// Limit packet size.
if(gPacketSize > 450)
gPacketSize = 450;
if(gPacketSize < 100)
gPacketSize = 100;
gPacketUpdateDelayToServer = 1024 / gPacketRateToServer;
U32 toClientUpdateDelay = 1024 / gPacketRateToClient;
if(mMaxRate.updateDelay != toClientUpdateDelay || mMaxRate.packetSize != gPacketSize)
{
mMaxRate.updateDelay = toClientUpdateDelay;
mMaxRate.packetSize = gPacketSize;
mMaxRate.changed = true;
}
}Including every code related with has the same rate and we made the below code to true:
checkPacketSend(true)
Only for the server side which will allow server to send packet to client very quickly.
You have mentioned that the Torque server expects the fixed and constant rate from client, but my client is just doing that for sure. Also you have mentioned that the move credit system is shutting client down because it thinks that client is using a cheat system, but since I changed it only for the server side the client won't send too much of an information to overflow. How should I understand this? Is this the same case both sides? Sorry that I am confused but could you please make it clearer?
When we have more players in the mission (up to 32), its frequency gets higher, so I'm guessing it may have something to do with it too.
And one thing I definitely want to hear answer to is: where is the code that shuts down the client? You mentioned it kicks the client which the server thinks the client is cheating with a hacking program. And where is the code that senses the cheating program?
#4
I could not find the code that kicks client out,
And also the code that skans(?) the hacking program of the clients.
08/06/2007 (7:08 pm)
Nobody knows about this? Anything will help, I need some of your advice.I could not find the code that kicks client out,
And also the code that skans(?) the hacking program of the clients.
#6
--Study the flow and impact of the ghosting system, specifically how the scoping system, network update --priorities, and create instrumentation to track what data is being sent, where it is going, and how.
--run the client under a visual debugger, and/or with profiling enabled to determine where the freezes are occurring.
--define your reasons for wanting to change the packet rate more completely, and evaluate alternate solutions.
For example, you mention wanting to decrease load times (a very valid concern), but a good alternate solution here would be to cache datablocks client side, instead of downloading them all every mission load, and only validate checksums.
Your statement
Is one that makes no sense to me...what exactly is "faster"? Simply jamming packets faster doesn't accomplish much when it comes to networking--the system as designed de-couples packet rate from simulation replication (through extensive use of client side interpolation and extrapolation), and provides instantaneous user response for control objects via client side prediction. Just throwing more packets at the client isn't going to do much in the way of accurately increasing any "speed" of any of these inter-related systems without careful thought as to how they interact.
I guess the only way I can describe this in a short statement is to compare Torque to a car. If you want the car to go "faster", you could theoretically simply press the gas pedal to the floor (increasing the amount of gas provided to the engine), but if you never shift out of second gear, you will be limited in your total increase. Sometimes to go "faster", you don't want more gas, you simply want to shift gears to a more appropriate gear ratio for your current engine speed and load demands.
Move input processing, specifically the move credit system. Also client side, if the server does not appear to be clearing moves from the move event queue, the client will get backlogged, and eventually give up. Start looking at code at/near
in gameProcess.cc.
On the server side, we track move credits in gameConnectionMoves.cc, and if our mMoveCredit value becomes higher than MaxMoveCount, it is capped at MaxMoveCount. This has the effect of no longer accepting moves from the client, and if the rate of move generation is higher over a period of time than the rate moves are consumed on the server, eventually the client will consider the server as not being connected (see above), and will eventually freeze itself.
08/07/2007 (7:55 am)
--Change the packet size, not the packet rate.--Study the flow and impact of the ghosting system, specifically how the scoping system, network update --priorities, and create instrumentation to track what data is being sent, where it is going, and how.
--run the client under a visual debugger, and/or with profiling enabled to determine where the freezes are occurring.
--define your reasons for wanting to change the packet rate more completely, and evaluate alternate solutions.
For example, you mention wanting to decrease load times (a very valid concern), but a good alternate solution here would be to cache datablocks client side, instead of downloading them all every mission load, and only validate checksums.
Your statement
Quote:
make our response speed faster.
(The speed I found faster is much faster than what you can see on Starter.FPS)
Is one that makes no sense to me...what exactly is "faster"? Simply jamming packets faster doesn't accomplish much when it comes to networking--the system as designed de-couples packet rate from simulation replication (through extensive use of client side interpolation and extrapolation), and provides instantaneous user response for control objects via client side prediction. Just throwing more packets at the client isn't going to do much in the way of accurately increasing any "speed" of any of these inter-related systems without careful thought as to how they interact.
I guess the only way I can describe this in a short statement is to compare Torque to a car. If you want the car to go "faster", you could theoretically simply press the gas pedal to the floor (increasing the amount of gas provided to the engine), but if you never shift out of second gear, you will be limited in your total increase. Sometimes to go "faster", you don't want more gas, you simply want to shift gears to a more appropriate gear ratio for your current engine speed and load demands.
Quote:
where is the code that shuts down the client?
Move input processing, specifically the move credit system. Also client side, if the server does not appear to be clearing moves from the move event queue, the client will get backlogged, and eventually give up. Start looking at code at/near
bool ProcessList::advanceClientTime(SimTime timeDelta)
in gameProcess.cc.
On the server side, we track move credits in gameConnectionMoves.cc, and if our mMoveCredit value becomes higher than MaxMoveCount, it is capped at MaxMoveCount. This has the effect of no longer accepting moves from the client, and if the rate of move generation is higher over a period of time than the rate moves are consumed on the server, eventually the client will consider the server as not being connected (see above), and will eventually freeze itself.
Torque 3D Owner James Spellman
I have one project where we were happily sitting at 1500 in TGE 1.3 but when we ported to 1.5.2, it crashes. I've not bothered to track it down yet as we were able to get it moving again at 1000.
I've not seen the checkPacketSend code, so I can't help you there.