Torque Networking Speed

#1 Your problem lies in your huge number of projectiles and other objects on the screen at once. It can be both your processing power and torque's networking (since even in single player, there is still a server and a client).

For instance, in the single second you fire your 50 rounds, your computer is actually processing 100 rounds since it is created on the server, and then ghosted to the client.

Even further, if you were to have another player join the game, that means yet another 50 rounds has to be ghosted to another client for every second you hold down the trigger. Quite a bit of work, especially if that other player intends to shoot back at you!

Some things that might help you out is to greatly reduce your particle effects and explosions since your primary objective seems to be quantity. You can also shorten the projectile life as much as possible without effecting gameplay as much. As your worst case scenario, just reduce your firing speeds and projectile amounts.

#2 You are also flooding your update packets with tons of initial updates for your new projectiles, which is going to cause you all sorts of issues.

In this case, I would create a new c++ class, call it "ProjectileCluster" if you like, and instead of networking 100 projectiles, network a single ProjectileCluster, and give it custom rendering and collision properties that are determinstic in your client and server.

that alleviates your huge network packet over-stuffing, as well as allowing you to gain optimizations in render performance.

Finally, running it in debug build mode will give you mulltiple orders of magnitude slower performance in some cases and some systems.

#3 @Jacob - The problem in this case is not rendering the objects. 100 rounds (with full particles) are something really negligible for current video cards. And in that case I would get a rendering stall on saturation which is not happening. The problem is that the objects are not being created at all.

@Stephen - Thats a nice idea but I need to thinker about this a lot. It would be a special kind of projectile where each projectile is controlled independent, and I would need to allow particles and such. Just a question... do projectiles update their position to clients each tick or they are just used for collision tests and ?

Running in release mode didnt help a lot. Effectively looks like the bottleneck is in the networking system.

Luck!
Guimo

#4 HMMMM... not as easy to do... this new object should stay located at the fire position and each individual bullet position has to be calculated each time the weapon is rendered and each time a collision is tested... not really funny... but has the potential for other weapons like the flamethrower.

Luck!
Guimo

#5 I can 100% guarantee that your issue is update packet overload. It would take me 2-3 hours to type up all of the reasons why, but basically what is going on:

server:

--you create a ton of projectiles on server
----for each projectile, it sends an initial update copy to the client (well attempts to)
----once the update packet is full (200 bytes is NOT a lot), it marks every one of the projectiles that wasn't packed in yet as 'skipped'
----the single update packet is sent to the client
----200 milliseconds later, it starts over again, filling up a packet very quickly

client:

--an update packet is received, probably containing at most 5-10 projectile initial updates
--these are created client side, and updates applied

server:

--next network cycle, it will send (smaller) normal (non-initial) updates for the projectiles with the highest net update priority, and try to squeeze in a few more initial updates as it has more room
--sends these off

Long story short: you cannot network hundreds/thousands of objects with any expectation of immediate creation client side. They will eventually all be networked to the client, but it may take several seconds.

The solution I described above is your primary best practice for this type of need.

#6 Hi Stephen,

I completely understand the problem and the cause, you don't have to detail it. My problem now lies on how to implement the alternate solution. The main problem being that at this moment I'm not that skilled with Torque to solve all those things. Anyway I guess I can leave that for the end of my production.

If you have any code that I may use as a starting point I'd be grateful. I'l start by understanding the projectile code.

Luck!
Guimo

#7 50 projectiles per second is hardly a "ton". We used to spawn 12x as much without issues.

#8 @Guimo: look at how the fxShape and fxFoilageReplicator classes work--they are very much similar in a structural manner for what you want.

#9 Thanks Stephen! I'll do it.

Stefan... 12x50 means 600 projectiles each second?

As I said in my case the first seconds (say 2 or 3) the projectiles come out fine, the problem is that if you maintain the rate of 50 projectiles/sec then you get many problems.

Maybe you are testing in an awesome network setting... I'll do more testing and will bother you later.

Thank you very much for your patience!
Luck!
Guimo

#10 What is the rate of fire and projectile speed for your weapon?

Projectiles with a speed of over 150m/s tend to have problems rendering on the client. The projectile travels faster than the network packets will update. As you increase the speed of projectiles, you will see them render further and further away from the muzzle on the client. The projectiles still exist and create collisions, but just don't make it to the client in time to render where you would expect them to.

I have a weapon in our game that fires 100 rounds a second at 300m/s. There's no tracer fire, so projectiles rendering a little slowly isn't an issue, but I can load up 10,000 rounds of ammunition and mash the fire button until they are depleted and every single projectile renders. I've tested this both on a local server, a client connected to a remote server via wireless, and a client connected to a remote server via CAT5.

How much alteration have you done to the underlying weapon/networking/engine code? From the description of your problem and the lack of progress from the responses posted, I wonder if you haven't inadverantly broken something.

I suppose that it's also possible that your router or a CAT5 cable is on the fritz and is tossing packets without generating a resend.

Have you tried using your weapon in a "fresh" starter.fps? Does the problem still persist?

#11 Completely off topic, but Bryce did you happen to get an email I sent you last week? If not, could you send me a new good email for you?

#12 I think this has been covered fairly thoroughly but I noticed no one mentioned variables and script performance related issues so I'll touch on them real quickly.

I can't explain this as well as a coder could but the way it was explained to me is that the more variables and ghost objects you create (in your case the projectiles are ghosts) the more you chew through the processor cycles.. I think the ghosting is covered ok but something some people dont think about is the difference in performance between script funtions and building functions into C++...

Script is great. And Torque script is one of the best scripting languages Ive used (and Ive used a few) but when you start laying down some network intensive interaction between client and server... it would do you well to have a coder take a look at your scripts and roll as much into C++ as possible.

When Sanctus Legacy was using Torque they claimed to have over 200 players connected for a load test. Steve @ dreamRPG claimed to have a similar number. I havent done any load testing yet ('soon'(tm)) but I have done performance analasys of network traffic patterns and datastream activity and I cut out.. Im going to say over 75% of my overhead by carefully bundling data correctly in C++. This is not modifying TNL at all. I'm not going to touch that for a long time. It's just a matter of taking processes that your game will call frequently and coding anythign you can into C++ instead of script..

I guess the best example of this is the starter.fps.. Look at the ammount of hard coded info.. Specifically health and energy.. (And I think the projectile code is hard wired isnt it? hell I cant even remember anymore haha). The reason that was done is because in starter.FPS those blocks of code were going to see a large volume of dynamic activity and the games performance could suffer.

Anyways I need to jet out. I hope this explanation helped a smidge.

Good luck

#13 I wouldn't blame it on the scripts. I doubt that's the overhead here and it would certainly help to create the projectiles in C++, but you won't see that much of a difference. Not sure where TNL got into the picture though.

#14 @Stefan: Hmm sorry maybe I misread, been up waaaay too late for the last week heh. The way I saw the original query it was reagarding the ability of Torque's network layer to handle Guimo's function. He notes that the way he was doing it was grinding the system performance. I just wanted to point out, as I think others did here, that the issues he was experiencing were definitely not related to the network layer in so much as they were relating to the sheer number of ghosts he was tossing into the datastream. A logic issue if you will rather than a code issue.

As far as blaming scripts: I am not faulting the Torque scripting compiler at all.. I said very clearly that Torque script is one of the best scripting languages I have ever worked with. However, for all of it's efficiency over other scripting languages, when you try and parse variables through the script compiler, every variable is looked up through the variable tree and if you are doing looping or even just detailed parsing you can really add latency to your server and a large ammount of CPU cycle overhead.

Reversely, if you move your high volume script work to C++ and just have Torque call the function and let C++ do all the backend processing.. the difference IS substnatially. I know because I've done it and performance tested it. Again it's absolutely application specific so someone running a Space Invaders type of application over a LAN probably doesnt need to give this much thought. However running a larger scale game over the internet that involves the parsing of literally 100s of variables per object, the difference in performance is night and day.

Just a quick example.. Running 3-4 people over an Internet game and parsing their player object vars in script I could spike the CPU useage anywhere from 20-50%. After migrating most of the parsing load to C++ the CPU load spikes between 5 and 10%.

Sorry if I was unclear in my post =) It's been a long couple weeks with very little sleep heh.

Mark

#15

Quote:
Just a quick example.. Running 3-4 people over an Internet game and parsing their player object vars in script I could spike the CPU useage anywhere from 20-50%. After migrating most of the parsing load to C++ the CPU load spikes between 5 and 10%.

Again, I have no clue what you are talking about. We have been using scripts for all kinds of things on our 1.4ghz dedicated and have yet to get it to perform badly, and that's with tons more advanced stuff that a measily variable check. Not saying you are wrong or that I'm correct, just that I have no clue what on earth you were doing to get such bad results.

Edit: Np about the sleep.. didnt think your post was difficult to understand.

#16 @Stefan : I guess what I am saying Stefan is that if a person wants to optimize their script, passing variable parsing and and processing to C++ makes a serious impact on the server load. I am not saying there is anything at all wrong with writing parsing and variable handling in script, I am sure a majority of the projects done with Torque do just that and it's very robust.

When I was talking about variable checks above, I was not referring to %myvar= "this string"; I was referring to the fact that the script compiler builds a pointer list at runtime for all of the script variables.. Everytime a variable is 'touched', wether that be to perform a complex parsing function or a simple strcmp, the engine must parse the pointer list for every single variable..

Let me show a completely exagerated statement below just to see if I can clarify:

%mydata = %clientobject_[%clientid]@" "@%clientobject_[%clientrace]@"
  "@%clientobject_[%clienthead]@" "@%clientobject_[%clientbody]@"
  "@%clientobject_[%clientsex]@" "@%clientobject_[%clientscale]@"
  "@%clientobject_[%clientmaxhp]@" "@%clientobject_[%clientcurrhp]@"
  "@%clientobject_[%clientmaxsta]@" "@%clientobject_[%currsta]@"
  "@%clientobject_[%maxmana]@" "@%clientobject_[%currmana]@"
  "@%clientobject_[%maxstr]@" "@%clientobject_[%currstr]@"
  "@%clientobject_[%maxdex]@" "@%clientobject_[%currdex]@"
  "@%clientobject_[%maxcon]@" "@%clientobject_[%currcon]@" ... etc. etc. etc.

The above function parses the variable pointer tree once for my data, and then twice for every single variable object in the list. (EX: once to get "%clientid" and once again to get "%clientobject_[whatever clientid is])

It is fairly obvious I think that changing all of this to a C++ array and simply calling:

%mydata = player.mydata;

has just eliminated literally 100s of pointer lookups per player connection by the engine. If you have 10 connections actively checking their data for example you couldve easily saved 1000 pointer lookups.

That is the only point I was trying to make =) I'll bow out of this one for now as I'm really not well informed enough about the inner workings of the pointer tree to be tutoring on it. If you are making a pacman game or simple RPG is it an issue? Probably not.. If you are attempting to optimize your code for 200 player connections running player objects with hundreds of variables (IE stats, inventory, buffs, spellbooks, skillsets, questlogs, etc etc.) yes, using a C++ array over the torque script is going to have a massive impact on your server processor cycle. =)

I'm willing to agree to disagree though. Just posting my experience with optimizing the playerdata handling for my game significantly.

All the best.
Mark

#17

Quote:
The way I saw the original query it was reagarding the ability of Torque's network layer to handle Guimo's function.

While it may not be performance related, the Torque network layer is optimized in stock to not handle hundreds of projectiles created per second that intend to be networked.

You will fill up the 200 bytes available to you in a non-tweaked client update packet, and projectiles will be skipped until the next network cycle. They may continue to be skipped depending on just how many you have until the number of times they were skipped has increased their net priority enough to ensure they go into the packet--but this could be seconds of delay before they show up on the client.

#18 Hi all, thank you for your posts!
@Bryce:
---------------------------------------------------------------
My projectiles travel at about 60 to 90 m/s (depending on the weapon) and they dont inherit velocity so its less than the 10m/s you are considering.

Hmmmmm... changes on the code on weapon/networking... I have them noted...

Added the Advanced Camera Resource plus a modification in order to adjust the camera speed.
Modified the engine so all the objects and flying vehicles stay always at 20m above the ground (just force the Z coordinate to 20, nothing else).
Added modifications to the Flying Vehicle in order to give them independent Speed, Turn, Damage, Armor levels. So I can set a vehicle to have more speed than other of the same kind even when using the same datablock.
Added the Guided Missile resource
Added the Flamethrower resource
Added the Laser Resource
Added a modification to the GUI and rendering (presentation only, no networking there)

A question... say I have a speed powerup in my game so if you get it you are faster than other players using the same ship (and the same datablock). Which one will work better?

a. Create an SpeedBonus attribute for the FlyingVehicle which is used in the movement computations, then create a network flag and send the data on the packUpdate/unpackUpdate methods. So when I collide the object I just set the bonus and the value will be transmitted to the clients.

b. Create an SpeedBonus attribute for the FlyingVehicle which is used in the movement computations. When I pick up the powerup I set the value in the server but also send a server comand to the client with the vehicle ghostID and the gained bonus as a parameter so the client can set the value.

Note that I have the first option running and it works just fine... at least apparently.

I'll try to test the weapon in the started FPS. Thanks for the idea.

Besides that, my PC has:
Celeron 2.4Ghz
Intel 945 Mainboard
DLink Wireless 54G Network Card
DLink router which supports that speed (dont recall the model).
nVidia 6600 Video card w/256Mb RAM.

Im not ruling out the problem is with the hardware. But my question is... If I create my game (not as a host, just single player). Why I'm still affected by the lag?

--------------------------------------------------------------------
@Flybynight
I really liked your comments on C++ vs TorqueScript. As you say, a script language will never match C++, but I hardly think thats the problem. In my game I have:
a. A single fxReplicator object forming an asteroid ring of about 2000 asteroids limiting the game area.
b. A starship.
c. 10 Asteroids created as simple unmanned Flyingvehicles (they have built physics, particle and explosion support) which fly from one side to the other of the ring. When they reach the other side they are destroyed. These asteroids can be shot and break in smaller pieces which fly to the asteroid ring and get destroyed.
d. 10 powerup pickups created as Item objects. They have gravity 0 and no movement. They are created each 12 seconds and stay on the game for 120 seconds before being destroyed.
e. 10 weapon pickups created as Item objects. They have gravity 0 and no movement. They are created each 12 seconds and stay on the game for 120 seconds before being destroyed.

So I have about 31 objects in the game at any time (the asteroid ring is considered terrain). When I fire my weapon my objects increase in about 20 projectiles each second. For what you say this is far lower than the expected performance. But 31 object which are computed each second against 2000 asteroids with a cube collision box... will that cripple the Celeron?

---------------------------------------------------------------------------------------

@Stephen
Im now working in my ClusterProjectile object (I'll post it as a resource when finished). The object stays in the starting position and the projectiles form an arc around the direction vector. The arc angle may be specified. Not working yet but I'm getting closer.

But now that I think about it... I have noticed that when I start firing the weapons, objects in the simulation stutter and "jump" from one position to the next one. But you say projectiles are discarded... isnt that a contradiction?

In other question... is there any way to test the network? And sorry for my newb questions... the idea of this game is for learning purposes. If anyone would like to try the game I can put it in a server.

Luck!
Guimo

#19 Guimo: You really need to reread some of the above ;)

The question was answered pretty clearly by some very experienced people. It's got nothign to do with the speed of your projectiles or how many Megs per second your wireless card can support.. You are sending your shots out as ghosted objects in Torque and Torque is designed to optimize the entry of ghosted objects into the network stream. Doesnt matter if you are playing single player or not you are still creating ghosts (as I understand it).

IMO from what I can tell you are going about your method wrong for how to do a FPS with high rates of fire.

Anyways just FYI lots of FPSs in torque work very well. I'm using Torque for an MM Oengine and it is doing amazing work for me. Currently Im using the stock network layer. From what I can read you need to read up on how Torque works with ghosted objects and optimize your logic to work better (IE play nicer) with the ghosting process.

Best of luck.

#20 This has been a very interesting thread.

Are there any debug/profiling tools built into the engine that would show how many objects are being skipped due to the updatepacket filling up or any tools that would clearly show that this is the problem?

Thanks
Todd