Technical: Network packet header questions..
by Bryan Turner · in Torque Game Engine · 03/19/2002 (2:08 pm) · 6 replies
Hello,
I've been poking around the code a bit recently and was studying the network packet headers. I'm curious why some decisions were made regarding structure sizes:
- 9 bits for Sequence # and Ack #
Why 9 bits? 2^9 = 512.. so we have sequence numbers 0..511 but a window size of only 32? Do we REALLY expect to recieve a packet 480 steps out of order? Or was there some other design decision that played into this?
- 3 bits for # ack bytes
This one confuses me the most.. after calculating how many bytes of Acks are following (from 1 to 4 bytes), there is some code that checks for and allows > 4 bytes of Acks to be passed through.
The kicker is.. on the other side, anything > 4 bytes is tossed out as if it was an out of order packet! This seems like a self-destructive loop to me, since this case will only occur when network traffic is not going well, and this case will continuously pump out longer and longer packets (until it wraps at 8 bytes.. which is another error situation waiting to happen).
-----
It appears that the protocol could assign 6 bits (thus, it tollerates packets up to 31 sequences out of order) per sequence number (or even 6 bits for seq and 5 for ack). And two bits for ack length.
The rest of the errors should be handled at a higher level. ie: knowing that the other side's ack buffer is full, we should stop sending data and only send ack packets until they start acking us back. Thus mitigating the downward spiral in congestion situations.
This also saves a full byte per header (3+4+1 bits).
-----
Has anyone else looked into the networking code at this level?
Thanks!
--Bryan
I've been poking around the code a bit recently and was studying the network packet headers. I'm curious why some decisions were made regarding structure sizes:
- 9 bits for Sequence # and Ack #
Why 9 bits? 2^9 = 512.. so we have sequence numbers 0..511 but a window size of only 32? Do we REALLY expect to recieve a packet 480 steps out of order? Or was there some other design decision that played into this?
- 3 bits for # ack bytes
This one confuses me the most.. after calculating how many bytes of Acks are following (from 1 to 4 bytes), there is some code that checks for and allows > 4 bytes of Acks to be passed through.
The kicker is.. on the other side, anything > 4 bytes is tossed out as if it was an out of order packet! This seems like a self-destructive loop to me, since this case will only occur when network traffic is not going well, and this case will continuously pump out longer and longer packets (until it wraps at 8 bytes.. which is another error situation waiting to happen).
-----
It appears that the protocol could assign 6 bits (thus, it tollerates packets up to 31 sequences out of order) per sequence number (or even 6 bits for seq and 5 for ack). And two bits for ack length.
The rest of the errors should be handled at a higher level. ie: knowing that the other side's ack buffer is full, we should stop sending data and only send ack packets until they start acking us back. Thus mitigating the downward spiral in congestion situations.
This also saves a full byte per header (3+4+1 bits).
-----
Has anyone else looked into the networking code at this level?
Thanks!
--Bryan
#2
I was looking for a more difinitive design decision though like "it must be 9 bits because 8 bits will cause such-and-such error to occur". 480 packets * 32 ms ~= 15 seconds.. that's really far out of order.
On another note, the 3 bit ack length was resolved. Aparently it does not send 0..3 +1 as I had imagined, but rather 0..4 as the length (which requires 3 bits to store). So values > 4 are illegal, but it still needs the third bit in order to encode the full range 0..4.
Anyone else have info on the window size?
Thanks!
--Bryan
03/20/2002 (7:09 am)
I haven't checked the server send rate, but the client sends packets every 32 ms. So 32 packets * 32 ms = 1024 ms.. about a second. I guess the extra window size is needed if packets come over a second out of order.I was looking for a more difinitive design decision though like "it must be 9 bits because 8 bits will cause such-and-such error to occur". 480 packets * 32 ms ~= 15 seconds.. that's really far out of order.
On another note, the 3 bit ack length was resolved. Aparently it does not send 0..3 +1 as I had imagined, but rather 0..4 as the length (which requires 3 bits to store). So values > 4 are illegal, but it still needs the third bit in order to encode the full range 0..4.
Anyone else have info on the window size?
Thanks!
--Bryan
#3
Personally, I doubt that one would ever see a packet 480 packets out of order, and if one did, it would be a rare packet that would still be at all relevant 15 seconds later.
Maybe the thing to do is add some logging so you can see some hard numbers about out of order packets and then play a few games on some crappy connections.
Josh
03/20/2002 (11:48 am)
I don't really think you can ever say that a design decision was made to make something like that a certain length. It is far more likely that testing showed that 8 bits wasn't enough, so they went to 9.Personally, I doubt that one would ever see a packet 480 packets out of order, and if one did, it would be a rare packet that would still be at all relevant 15 seconds later.
Maybe the thing to do is add some logging so you can see some hard numbers about out of order packets and then play a few games on some crappy connections.
Josh
#4
By setting the seq number to 9 bits, it means we can differentiate packets within a window of 15 seconds. This means that when packets fall outside the current window, they can be discarded.
Otherwise, some stale, 14 second old packet might be mistaken for a current packet.
03/20/2002 (11:59 am)
Bryan, the sequence number range has to be much bigger than the window size.By setting the seq number to 9 bits, it means we can differentiate packets within a window of 15 seconds. This means that when packets fall outside the current window, they can be discarded.
Otherwise, some stale, 14 second old packet might be mistaken for a current packet.
#5
The reason for 3 ack bits is that it is possible to have 0 ack bytes - 0...4 = 5 numbers, 3 bits.
03/25/2002 (11:37 am)
Well... the reason for 9 bit sequences was I wanted to be sure that no out of order packet would ever be confused as fitting into the current window. We never did any hard testing on this - I just decided 9 bits would be more than enough.The reason for 3 ack bits is that it is possible to have 0 ack bytes - 0...4 = 5 numbers, 3 bits.
#6
Thanks :) That's exactly what I thought (9 bits = magic number), but I wanted to get it from somone who knew difinitively.
I've been mucking around in the bowels of the networking code for over a week and I've not seen a single out of order packet - even when the traffic gets bad.
I'll keep an eye on it, but my guess is 6 or 7 bits may actually be more than enough. Although, 3 bits per packet is pretty cheap for piece of mind.
--Bryan
03/25/2002 (6:30 pm)
Mark,Thanks :) That's exactly what I thought (9 bits = magic number), but I wanted to get it from somone who knew difinitively.
I've been mucking around in the bowels of the networking code for over a week and I've not seen a single out of order packet - even when the traffic gets bad.
I'll keep an eye on it, but my guess is 6 or 7 bits may actually be more than enough. Although, 3 bits per packet is pretty cheap for piece of mind.
--Bryan
Torque Owner Josh Goldshlag
31 possible out of order packets is entirely too small. Packets can often be delayed more than a second, and while some would be no longer relevant, I think enough would be to make it worth spending the 3 bits per packet. Remember that this stuff has been tested on a much larger scale than anyone here is capable of, with T2. Many people complained about many different things in T2, but packetloss and high pings were not often mentioned.
Josh