Network problems
by Sam Redfern · in Torque Game Engine · 03/09/2008 (3:44 pm) · 6 replies
My game (Darkwind) is continuing to give problems to some players - they freeze up when entering an event, sometimes. They can enter other events ok, and other people can enter the event that is causing problems for them. For some people it happens often, for other maybe only once every few weeks. And most people never see the problem at all. So it's certainly something related to their own hardware or network setup.
When a freeze-up happens, it's always at exactly the same spot.
Someone managed to log the packets they were receiving and it seemed like the same data over and over - so something is mangling it, and it keeps requesting a re-send from the server, I think.
Some people have fixed it by making router changes (e.g. bypassing security for my game) - I would assume some kind of packet inspection was interpreting a particular sequence of data, by coincidence, to be matching a virus signature.
But what I find most strange is that this doesn't seem to happen in other games. It would surely be discussed here if it did happen.
The only change I have made to the low-level networking code is to increase gPacketSize from 1500 to 15000 (this was a quick work-around to stop overflows as some of my vehicle game objects are quite large when all their cargo data is packed in).
Could gPacketSize=15000 be causing problems for some client network cards, under certain circumstances but not others??
Or does anyone have any other ideas?
When a freeze-up happens, it's always at exactly the same spot.
Someone managed to log the packets they were receiving and it seemed like the same data over and over - so something is mangling it, and it keeps requesting a re-send from the server, I think.
Some people have fixed it by making router changes (e.g. bypassing security for my game) - I would assume some kind of packet inspection was interpreting a particular sequence of data, by coincidence, to be matching a virus signature.
But what I find most strange is that this doesn't seem to happen in other games. It would surely be discussed here if it did happen.
The only change I have made to the low-level networking code is to increase gPacketSize from 1500 to 15000 (this was a quick work-around to stop overflows as some of my vehicle game objects are quite large when all their cargo data is packed in).
Could gPacketSize=15000 be causing problems for some client network cards, under certain circumstances but not others??
Or does anyone have any other ideas?
#2
I tried lowering the packetsize back to 1500 for someone that was having the problem, and it made no difference.
I don't see how it can be something like this, since most players (all running identical code, of course) can load into an event without problem, even when this event is causing a problem for one specific player. I even tried logging in as that player, just to make sure I was receiving identical data to them, and I got in fine.
It's not this - we don't send move structures at all, I have made all cameras client-side only (and the players never directly control anything else).
Since some people have managed to fix it by bypassing their router security, etc., I'm going to try identifying when the issue is happening, and then adding random dummy data into the stream, to avoid sequences with identical bytes in from repeatededly sending and repeatedely getting mangled.
03/09/2008 (5:59 pm)
Thanks for the reply Stephen,I tried lowering the packetsize back to 1500 for someone that was having the problem, and it made no difference.
Quote:make 100%sure in all cases that your packUpdate/unpackUpdate are matched perfectly. With conditional logic in the mix, it can be pretty tricky to guarantee everything is always matched. If in some logical branch, things are unmatched, you are effectively corrupting memory, and anything can happen.
I don't see how it can be something like this, since most players (all running identical code, of course) can load into an event without problem, even when this event is causing a problem for one specific player. I even tried logging in as that player, just to make sure I was receiving identical data to them, and I got in fine.
Quote:It's also possible that the client is somehow generating too many move structures, and getting ahead of the server's move credits for that client, although this would normally have a different indication.
It's not this - we don't send move structures at all, I have made all cameras client-side only (and the players never directly control anything else).
Since some people have managed to fix it by bypassing their router security, etc., I'm going to try identifying when the issue is happening, and then adding random dummy data into the stream, to avoid sequences with identical bytes in from repeatededly sending and repeatedely getting mangled.
#3
03/10/2008 (5:19 pm)
Just an update on this... one of the causes (although possibly not the only one) appears to be a bloated packUpdate on a vehicle with a lot of cargo (about 40 weapons in a lorry). I removed his weapons, respawned the game and he loaded in fine. So possibly his network card or router did something nasty to the data that arrived at unpackUpdate. 
#4
03/11/2008 (6:02 am)
Edit: Oops... Dont wanna hijack the thread 
#5
It definatly could.
This is the maximum size of your packets, so it won't have any effect until packets are generated that actually are larger than 1500. Which is probably what you are experiencing with this lorry.
I'll guess the transmission problems are either caused by faulty network card drivers (do your clients have updated NIC drivers?), or most probably that the client is on a network which doesn't allow fragmented IP packets (this could cause problems even with 1500, some networks require as low as 400).
In the last case you have to make sure that you never generate packets larger than the network MTU.
if your server allows pings, the client can test for this issue using the ping command, something like this:
ping -l X IP_ADDRESS
X is the packetsize you are testing
if it works with a low size, and you with a high size either don't get a reply from the server, or get a message indicating that fragmentation isn't allowed, it shows that fragmentation is an issue.
then you can experiment with the size, to see what the maximum is for this client.
Pherhaps the best solution to your problem is to send the inventory of the lorry as separate packets, instead of stuffing it all in one big packet?
03/11/2008 (8:52 am)
"Could gPacketSize=15000 be causing problems for some client network cards, under certain circumstances but not others??"It definatly could.
This is the maximum size of your packets, so it won't have any effect until packets are generated that actually are larger than 1500. Which is probably what you are experiencing with this lorry.
I'll guess the transmission problems are either caused by faulty network card drivers (do your clients have updated NIC drivers?), or most probably that the client is on a network which doesn't allow fragmented IP packets (this could cause problems even with 1500, some networks require as low as 400).
In the last case you have to make sure that you never generate packets larger than the network MTU.
if your server allows pings, the client can test for this issue using the ping command, something like this:
ping -l X IP_ADDRESS
X is the packetsize you are testing
if it works with a low size, and you with a high size either don't get a reply from the server, or get a message indicating that fragmentation isn't allowed, it shows that fragmentation is an issue.
then you can experiment with the size, to see what the maximum is for this client.
Pherhaps the best solution to your problem is to send the inventory of the lorry as separate packets, instead of stuffing it all in one big packet?
#6
Yes, I'm going to send the inventory of vehicles as separate packets.
03/11/2008 (9:22 am)
Jan, thanks for the info - good idea about pinging with sized packets too.Yes, I'm going to send the inventory of vehicles as separate packets.
Torque 3D Owner Stephen Zepp
While I don't think it's your root issue, I'd look at lowering that down to a couple of K or so bytes and see if it still fixes your original problem, but also increases other performance areas.
Back to your problem--this is a tricky one. Code wise, a couple of areas I'd look at:
--make 100%sure in all cases that your packUpdate/unpackUpdate are matched perfectly. With conditional logic in the mix, it can be pretty tricky to guarantee everything is always matched. If in some logical branch, things are unmatched, you are effectively corrupting memory, and anything can happen.
--it is possible that the packet size you are using is indirectly causing this issue. A 100% "freeze", as opposed to a crash, is indicative of an issue staying in synchronization with the server--possibly lagged out, possibly simply not keeping up with the data.
--It's also possible that the client is somehow generating too many move structures, and getting ahead of the server's move credits for that client, although this would normally have a different indication.
My general suggestion would be to learn the ins and outs of the networking debugging capabilties (there is a define you can set to give more information), and keep a careful eye on the bandwidth used, and the processing of both client and server during these freezeouts. A lot of data is really going to be the only way you can nail this down for sure.