Network lag compensation

#1 Well I like all gamers hate when lag plays a roll, but as internet connections get faster and better almost constantly I don't notice it anywhere near what I did back in the day. On most quality/popular fps games now a days the network coding is very solid and most gamers who play on any given server have a 30 or less ping to it, this makes lag not play much of a roll at all. And most hardcore games who like to play fps online have dsl cable or the pretty sweet fiber optics connection now a days so everyone has very fast connections.

Lag compensation is a tough thing to program in without making it way to easy for cheaters. Without knowing much programming at all you history idea is interesting to me. But other people who actually know how to program would have a better idea of how that would work out

I certainly don't think forcing everyone to a 150 ping would please gamers, since most people pay a good $40 a month for their super fast connection they don't want that limited just because others don't want to pay for a good one.

#2 Actually he refered to read-ahead 150ms, decreasing the actual ping, to compensate for the duration.

Just to add clarity. Nice idea!
- Ronixus

#3 @ J Sears, just to clarify, ping is not really influenced by the speed of your connection. It's more related to distance or more specifically, how many network 'hops' occur between you and the server.

Australia to USA is a minimum 200ms ping time due to the amount of 'hops' involved. 30ms and less is the norm for playing on a server connected to your own ISP.

@Christoper, this thread is an offshoot from this blog which may also be of interest to you.

#4 Duncan, your "history" idea is great, and I think if can "live".

As for the "not using projectiles" - this is a thing I use for my own project. Actualy, I'm using it, but just to "show" the way where the bullet is going - it adds nice visual effects.

And yeah, this is interesting discussion, and I'm looking forward of seeing more ideas/etc.

edit: Just realized... look at the thread ID :) grats! GG community passed the 60k threads, hehe.

#5

Quote:
(a) Don't use projectiles where possible, ( No curved ballistic trajectory) Instead use line of site or laser type weapons. This will allow the firing solution to be calculated on the client using a cast-ray (client side) and a "hit" sent to the server for distribution to other clients.
At least the client will get to hit what he sees on his screen, irrespective of lag. But it does open the door for cheating clients to send fake hits ( I hate cheaters)

Haha, hold it! Why would it open the door for cheating clients to send fake hits? You do not *send* hits. You ask to shoot, the server executes and the client should predict in the same manner. Hopefully both are in a similiar state and more or less in sync, and you will get a perfect simulation.

This is not in any way heavier network-wise than what you described above. Curved ballistic trajectories have nothing to do with lag, since it is replicated on the client after the server has sent the ghost ID.

By predicting image fireing states as well as the projectile, you can get away with sending no packets at all. Perfect for rapid fireing weapons.

#6

Quote:Hopefully both are in a similiar state and more or less in sync

um.. but they're not. hence this thread.


Duncan -
client hacks aside, another problem with client-side authority for hits is that not all clients will agree on the hit. what if you and i are both lagging, and at exactly time T i raise my shield of invulnerability on my client, and on your client you pull the trigger of your instantaneous energy fragger ? both our clients are acting in good faith, but only one can win.

Interesting thread.
I like the idea of establishing a minimum ping of 150ms or whatever.
It might suck in practise, but it's definitely interesting.

#7

Quote:
um.. but they're not. hence this thread.

We have players with constantly 200-300 ping and the game behaves fine there. Better than stock Torque or Tribes anyway. You just have to add the trip time to the prediction simulation. Care to show me how it would not work, given your obvious tests in this area?

Mind you, Quake and BF uses the very same technique.

#8 Ahh lag, topic I love to hate. I am an avid player of TT (ThinkTanks) for over 2 years and we as a community have been battling this topic for awhile. We have many players that play from overseas and with 56k connections. Things that we have done to battle lag:
1. open ports for servers and clients to help with downloads and play
2. Turned down the visabilty. ( has a lot to do with a players scope)

If you have played TT you know that we typically play with smaller maps but do have a lot of bullets flying. For us the things that cause the most lag:

1. Bad connections. connections were data is being lost due to a bad router. Servers hate this big time
2. Weak computers. Had one person play on an old mac then jump to a newer mac. Huge difference in performance for them and the server.
3. -dsleep function. For TT the -dsleep function was never enabled for windows servers. If you have anytype of lag, you can really feel it using a windows server. The dsleep function is like a smoothing function.

My thought is that it just not so much how far the distance from the server the person is but, how is the connection, how is the persons computer (obviously not taking 56k into account), how is the server set up, how much eye candy does the game have. Obviously if you have a high poly count game and weak computers and 56k players, the game will lag. if you keep the eye candy down, require at least dsl and newer computer (let others decide that) Torque networking should be able to handle people from AUS.

That being said I am very interested in seeing how a server like MoM's can handle a FPS game......

#9 Lowering far distance is a good way to limit scope and how many ghosts you have at once. That is a pretty good advice if your game design covers it.

#10 You didn't specify what sort of network architecture you were envisioning.

The problem with most FPS games is that you generally have a smallish number of players, and the server happens to run on one of their machines (this is the way console network matches work as well).

In that case, it's much harder to control cheating... even if you can avoid the client cheats, the person controlling the server has a whole host of undetectable cheats available to him... most of these involve using another PC in between his client/server machine, and his connection to you. That router can introduce lag specific to any connection. This is a common form of cheating, particularly in the console world.

Having some third entity (the game producer, etc.) control the servers helps immensely.

Presuming that you meant for that to be true:

a) You already mentioned the biggest problem here (clients sending false hits)... maybe there are clever ways to mitigate this possibility (one technique is to allow clients to 'vote', in the server, and for the server to change a trust metric for clients that seem to be losing the vote too often -- this one has always seemed particularly clever to me, because it's a dynamic heurisitic that can allow you to identify people who are running reflectors and other hacks).

b) This might be susceptible to client cheats involving proxies that create artificial client lag and adjust it dynamically.

c) This is generally a good idea, regardless of other issues, I think, and 150ms seems like a reasonable amount of compensation to add for most connections because it doesn't seriously degrade play. I think some games go one step further and also adjust the compensation basis by averaging measured client lag, but any time you do this, you open yourself up to people futzing with router hacking and introducing artificial lag.

One way to mitigate the possibility of clients introducing artifical lag is to encode actual client origination timestamps into the stream (assuming you're already tracking client clocks, or doing some sort of clock sync'ing), and make it hard to figure them out (i.e., encrypt the stream)... this requires someone to actually write a proxy which can rewrite packets, which really is an order of magnitude (or two) more difficult than simply sticking a router in there. I'm not too familiar with how newer games encode their packet streams, but (at least originally) EverQuest used encryption with 32-bit keys to encode the network stream... you couldn't discover anything about the stream or mess with it without the 32-bit key from the client (which you can trivially get by reading memory on the client, but again, that's 10x-100x more difficult for your average Joe than configuring a route with artificial or dynamically-adjustable lag).

A modern PC has enough extra cycles to run the FPS at decent frame rates, run the simulation, and handle encoding/decoding an encrypted stream that has a cipher with a key of decent length. In an FPS sim, the server is probably not handling so many connections that this isn't possible (in an MMOG, encrypted streams can become a problem because of the sheer number of connections, but, you can buy boxes that will offload all that work).

Interesting thread!

BTW, there's been a ton of academic research around this topic in the last 5 years or so.

#11 @Tim
For option (b) I would envision having the server enter a timestamp index into all packets sent out. Clients would then add that same time stamp index into packets being sent back to the server. The server then uses the return timestamp index to checks its history for the location of objects at the moment the client fired his weapon. The time stamp index can be a random number used as a lookup in a table containing the actual time on the server. That way its impossible to perform the cheat you mention because a hacker wont know what server time the random number represents. Only the server has that info.

Orion Elenzil mentioned an interesting situation as well but I think the time stamp method will allow the server to figure out (scrathes its head) if client B had his shields up in the time frame when client A fired his weapon, or not.

@Stefan Lundmark

Quote:
Curved ballistic trajectories have nothing to do with lag, since it is replicated on the client after the server has sent the ghost ID

My point on curved ballistics is that the server updates the projectile location with each tick and then checks if it hits anyhing. Its very much a time dependent thing because projectiles don't instantly arrive at their destinations, which means other players may or may not be at the impact point by the time it impacts. This is very much a lag dependent situation because a player thinks he ducked behind a wall before the rocket projectile got there but the server has data which says the projectile arrived while the player was still there.
So the player gets pissed of, spams some lag abuse on the chat lines and leave the game.

By not using ballistics you can mod the code and have the client do an instantanious castray to see if it hit anything based on the data it has thus eliminating any lag related discrepancies. Yes I know that normally you ask to shoot and the server does the processing, but the normal way of doing things does not fully address the lag issues.

Yes, client prediction does help but its a prediction and therefore not reality. A player who zig-zags will usually appear to pop from one place to another on a laggy client because client prediction assumed he was continuing in one direction but when the server update arrived, he was actually somewhere else. So the client now fires at the 'new' location, it takes 200ms (or more) for his shoot request to get to the server and its still a miss because 200ms was enough time for his target to have moved beyond the place he aim at. A ballistic projectile just adds more time and hence lag discrepancy to the situation. Yes you can predict a ballistic trajectory once the weapon fires but you can't predict where players and vehicles will be when it impacts.

@NUTS!

Quote:Torque networking should be able to handle people from AUS

Wrong, nobodies network code is going to handle the packet delay introduced by the 13 to 18 router nodes enroute from Aus to USA. A network lag is the delay introduced by the network. "Lowering far distance" and other such game related things will affect your frame-rate and may appear to make the game more responsive for frame-rate reasons but in reality it has zero impact on network delay. Those 13 routers are still there regardless of how well you made your game.

#12 @Duncan
True, but if the server is set up for it using the -dsleep function it can handle it to some degree. In TT, we have had a few people from AUS play the game as well. The ones that had the newer computers and decent broadband still pinged in the 230 but didn't lag to bad, or the server. If I understand the -dsleep function correctly it automatically adjust the server to compensate for the lag, like a delay. As for the framerate impacting networking, directly no, indirectly possibly. If the client cpu is being "slowed" by eye candy it can slow down the response time. So no real imapct on ping, just on the client cpu and its response time. Get TT jump on sometime, be happy to review what we have found.

#13

Quote:
Yes, client prediction does help but its a prediction and therefore not reality.

Okay. This is game programming and design. Obviously you want something better than prediction while still catering for quite a high latency connection. What is your solution? (:

Quote:
A ballistic projectile just adds more time and hence lag discrepancy to the situation.

Well it depends what you compare it to. Hitscan weapons are easier but not all game designs can cope with it.

#14 @Stefan

Quote:
What is your solution? (:

Option (b) and (c) as described in the top post.

Option (c) is easier to implement but is limited to about 150ms before it become game intrusive to low lag clients. This option allows the use of 100% accurate ballistic weapons. (from all clients perspectives)

Option (b) should also be easy to implement in its simplest form but for improved accuracy, may require some clever algorithms to solve situations as described by Orion earlier. It can also solve ballistic issues but it means that a trajectory will have to be calculated for each client from the moment that weapon was fired in that client's time frame. You would then need to use a voting basis (thanks Tim) to determine who actually died. i.e. If a missle kills Jack and John according to two clients but only Jack dies according to 5 other clients time frames then John gets to survive that vote.

In theory option (b) can cater for a broad range of ping times and leave everybody 95% happy with the result.

#15 I am slightly confused. You mention latency and secure in the same sentence and then you go on to say that prediction is not enough. Then what on earth is a client hit-scan solution if not unresponsive and non-practical for a lagging player? His gamestate will be very far from the others and he might strike a hit eventhough there clearly is none.

It does not matter if you use b) there because the client could adjust his ping time as he sees fit in the right moment to get around it. This is essentially the issue HL1 battled with before they switched to a server/client predict system.

1. Hitscan on the client. Play animations and particle emitter.
2. Hitscan on the server, damage the object.

Of course: it always comes down to what you need for your particular game and how much your design can cope with any eventual cheats. In our project, we just could not accept such behaviour and we wanted to get rid of it as much as possible.

#16 Point #1
----------
First let me state that I don't believe in supporting 56k and sacrificing high-quality content or performance. If I may, allow me to throw some stats at you.

http://www.internetworldstats.com/articles/art030.htm

I'm not going to get much love for this, but I'm not designing games for 10 years ago. It's a broadband world and that's not changing. At least now you know where the rest of my commentary is coming from. ;)

Point #2
----------
PlanetSide, an MMOFPS by SOE, works on Client-Side Hit Detection (CSHD). The server performs packet validation as far as time order and frequency (SpeederXP et al), but it doesn't prevent manipulation of data within packets themselves. The net result (pun intended) after 4 years of mediocre network prediction resolution in addition to client-side hit determination, is that the game has become an absolute field day for memory hacks and dual core "abuse". Which is a great segway into ...

Quote:
(a) Don't use projectiles where possible, ( No curved ballistic trajectory) Instead use line of site or laser type weapons. This will allow the firing solution to be calculated on the client using a cast-ray (client side) and a "hit" sent to the server for distribution to other clients.

Anything you leave client-side starts the clock on the time remaining before your game has someone abuse it. Allowing the client to peform anything as opposed to the server reduces FPS gaming to "he who has the fastest rig wins". Server-side ghosting in the traditional Torque model equalizes the playing field between the guy who hasn't upgraded in 3yrs and the guy who got his new Alienware box yesterday. Furthermore, while performance may be spectacular from the point of the person pulling the trigger, from the perspective of the person getting fired at, it's not fun. *BAM* you're dead ... because Mr. Alienware registered 3 hits on you on his rig in the 400ms it took for the server to tell you that you died.

Point #3
----------

Quote:
(b) To avoid cheating clients, let the server store a transform history ( say up to 500ms in 32ms intervals) of mobile targets such as players and vehicles etc.
When the client fires his weapon, the server checks the clients ping time, doubles it, and then checks its history for calculating the clients firing solution. That way the client still gets to hit its target but now he can't cheat. You will have to limit connections to those clients whose ping time fall within your history allocation.

Forcing extra calculations on the server only increases the minimum requirements of the server running the game. As stated eariler in this thread (and that my experience supports), it's a rarity to have someone host a "permanent" game server on the internet that isn't being rented. Accepting that as the case, the majority of servers are either rented servers or ad-hoc servers setup on the fly on a client machine. This increases the overhead on servers for hosting companies OR you reduce the performance overall because one of the clients is forced to perform all those extra calculations.

Ever time a ray cast calculation? If you don't think that 64 players in full combat in an FPS can't fire over 1,000 projectiles a second, I think you might be in for a surprise. That's 1,000 ray casts that must not only be computed, but then interpolated against some table of previous transform data on the server.

It might sound good in theory, but I don't think that it's going to deliver the gains that you're going for in a very practical sense.

And that leaves...

Point #4
----------

Quote:
(c) Introduce an overall lag compensation of about 150ms for all players. This means a formula of 150 - ping gets added to all clients whose ping is below 150. This has the added benefit of making near perfect client-server sync for all players below a 150ms ping making vehicle collisions and racing games a real pleasure. Of course the 150ms can make the game feel just a little unresponsive but the advantages may out weigh the annoyance. The value of 150ms can obviously be fine tuned to suit the game but at least a player with a 200ms lag will now only lag by 50ms.

This seems a "communist" approach to solving the problem. Some people have sucky connections, so we shall make all players have equally sucky connections. Combining Points #1 & #2 from both perspectives... If I'm Mr. Alienware and I just spend $3,500 on my new rig I'm not going to play a game that makes it perform "a little sluggish"... not when there's 1,000's of titles out there to choose from. Furthermore, I believe it's far more likely that an individual will be able to upgrade from a $10/mo 56k connection to a $30/mo broadband connection than it is that that same person will spend $1,000 to upgrade their PC. For some, that is not an option, but that just can't be helped.

Don't misunderstand, I'm all for exploring new ways to reduce network traffic to the game server.

In our experience the NUMBER ONE way to improve server performance instantly is to relegate your map server to the sole task of processing the game simulation. Offloading chat, scorekeeping, and all other "non-game essential" network traffic to another server farm, and have that processed on the client.

If you try to be all things to all customers the only thing you succeed at is being nothing to everyone. Eventually, someone is going to be left behind... your role (as a game designer) is to determine who that is going to be.

#17

Quote:
Furthermore, I believe it's far more likely that an individual will be able to upgrade from a $10/mo 56k connection to a $30/mo broadband connection

A lot of people dont seems to get this subtle point.

Broadband will NOT reduce network lag or latency. It will reduce load times and help handle the quantity of network traffic you can handle but your "sucky connection" theory has nothing to do with network latency.\

Each router the network packet gets passed through has to handle or process that packet before it sends it further down the line. Each router has internal electronics which works on digital clock cycles and thus takes time to process each packet and that time is absolutely unaffected by the speed of your dial-up or broadband.

Don't believe everything the salesman tells you.

Broadband lets you download more packets at the same time but the packets all travel at the same speed, regulated by those routers on the internet.

#18 Yes option (b) will require more CPU time which is a negative. It is still an option for that extra server you speak of. Ideally I would also run all AI 'thinking' on a separate server as well.

Option (c) can seem like an invalidation of the expensive rig you own but not really because it has nothing to do with graphic quality or number of items on screen etc. It will just regulate your network connection to put you on an even par with most other players and thus give you an overall better game experience.

#19 Actually option (b) in its simplest form wont take up many extra cycles because the server would be doing be doing a 'hit-scan' or hit check etc anyway. It's just getting its tranform data from a lookup table. (In its simplest form that is)

The problem with multi-player games is that you need a common time-frame to evaluate the game conditions with. Routers spoil that ideal by changing the relative time frame for each player.

The trend to using the server time as the common time-frame is not the best solution, in fact its just dead wrong because only the player hosting the game will be in real-time.

Option (c) is the fairest solution, followed by some form of option (b)

#20 In the "real world" I'm the IT Director of a company w/ 3 offices with more workstations and servers than my entire family has fingers and toes. Believe me, when I have to firewall deny YouTube and MySpace access so the Sales department in Ventura can access the reporting server, I do understand the difference between bandwidth and latency. :)

Consider my post in it's totality. What I am suggesting is before making drastic changes to engine networking code, utilize all the packets possible for game data. If you take advantage of a broadband connection by creating several connections to multiple servers for different bits of data, you can improve game server performance be reducing the amount of data it must process. You don't change how fast the data gets there, you split the load of processing that data between multiple sources. Those packets could then be filled with hard-core game data. Further addressing your point, if you have "master" servers that are placed strategically along internet backbones you can almost guarantee < 100ms performance for that ancillary data. How much more responsive would your game servers be if they never had to send join/leave, chat, score, or other ancillary data updates?

I still contend that your estimate of 150ms being "acceptable" are greatly overstated. I, for one, will avoid servers for 99% of games that have a ping of over 100ms. Why bother with it when there are a plethora of servers with < 100ms pings?

There's a paradigm that is bandied about from time to time which states that one is not supposed to design for today, but for tomorrow. Developing a project in 2007, with the Digital Age in full ascendancy, to have such expectations of user's ability to "live with" lower performance standards I feel is a fundamental mistake.

Either design a game that is broadband or a game that is for 56k, or a different flavor for each... but to limit the growing population of broadband gamers (with ever increasing content expectations) seems a bit antithetical to making a product that will stand the test of time.

Again, I'm not arguing a rethink of certain network performance issues, I just think that your intended targets based on your technical implementations may not be realistic as to what the modern... and future... gamer is willing to tolerate.