ActivateGhosting and object streaming

#1 The mission load sequence basically does the following:

--streams all datablocks in the mission.
--streams objects marked as ScopeAlways.
--initializes client side needs, such as replicators, and lighting.

At that point, the server side GameConnection::onClientEnterGame() callback handler is triggered, creating the client's secondary camera (for free camera mode), the avatar, and sets the avatar as the control object.

Next network cycle is the first time where a control object for that client is in the mission, and is the first time onCameraScopeQuery can actually be called on objects to determine if they are in scope.

With that being said, the only way I can see objects being delivered down as part of the mission load sequence is if they are datablocks, or marked as scopeAlways.

In other words, I think your inspection, and therefore your statement "I can see that every object is put into scope if it is ghostable at all, seemingly ignoring other scoping rules." shouldn't be correct. It should only be if they are marked as scopeAlways.

#2 Stephen, is there a reason why all TSStatic's and InteriorInstance's are "ScopeAlways" objects besides light/shadow map generation?

#3 Nope...that would be the exact reason in fact. Since they have no networking traffic other than initial creation, and are needed for light/shadowmaps during phase 3 of mission download, the easiest technique is to make them all scope always.

And good catch by the way, that would explain exactly what he's seeing.

#4 Would the fact that they are ScopeAlways have an affect on rendering performance? Say for example, I face the camera direction towards where a large group of TSStatic's reside, but they're outside of my Visible Distance. Could that cause lag in the game?

#5 Not under normal circumstances, no. Scoping and rendering have pretty much zero coupling--don't let the name "onCameraScopeQuery" mislead.

That being said, Torque doesn't do much occlusion other than view frustrum, visible distance range, and some circumstances within interiors (as well as legacy terrain does occlude if I remember correctly), so if you turn to look at a massive series of shapes all within view frustrum and visible distance, and they have no LoD, you'll get a very large render hit.

The only real coupling between rendering and networking is that you can only possibly render what's scoped to the client, since rendering occurs on the client.

Edit: that last sentence is slightly incorrect--you can only possibly render what's in the client's simulation, and game objects that exist on the server must be networked to the client's simulation. I decided to leave the sentence in place just in case it was confusing to edit it directly.

GUI's, and client side only objects (foliage replicated objects for example) are never networked, because they only ever exist on the client.

#6 Well,
network scoping does affect rendering performance in a way
because objects which go out of network scope are actually deleted on the client,
meaning they don't even get considered for rendering.

Mod Note: Ouch, moderator mis-use! Sorry guys, I hit moderate thinking it was a quote button, and messed up Orion's post--my bad.

#7 Ok, so Visible Distance should cull the objects then. Essentially, should the performance be the same if I point the viewing frustum toward an area with dozens of objects outside visible range as when I point the frustum in an area where there are no objects at any distance?

#8 At the theory level absolutely--of course, the act of culling itself does have some performance hit, so there is a measurable performance hit even with the objects not being rendered--but it's not a render hit, it's a cpu/processing hit.

I'll be up front though, this is an area I'm not recently familiar with at the implementation level, so it's possible that the collection of objects "to possibly be rendered" precludes automatically any objects outside of view distance...I'm not sure of the actual algorithm used to collect the pre-render list.

#9

Quote:
Well,
network scoping does affect rendering performance in a way
because objects which go out of network scope are actually deleted on the client,
meaning they don't even get considered for rendering.

That is a true statement, I agree :)

But take it for what it's worth--reducing your scoping ranges is not necessarily a good rendering optimization design, since scoping is primarily designed to make sure objects are on the client early enough so that there isn't obvious user delay, yet late enough so that we don't overload our bandwidth.

Orion--glad to see you in the convo, since you've done a lot of work with both network and rendering optimizations :)

#10 Ironically (or interestingly) for those (like myself) who are using TGEA and Atlas, since Static Shadowing doesn't work, I could remove the "ScopeAlways" flag from TSStatics and InteriorInstance's and get a loading time performance boost. At least, until Atlas is affected by light maps.

#11 that is very interesting to hear.
we're looking at doing something similar in TGE for missions composed of isolated Interiors which the player is always inside of. ie, turning off ScopeAlways for interiors.

#12 Theoretically, if the light map generation procedure were altered to run server-only, and then always sent to the client, couldn't ScopeAlways be permanently turn off?

#13 That's what I'm hoping for! I'm pretty sure if we looked at the circumstances which cause mission relighting we could calc the lightmap one time and give it out in the client distribution, set the scopealways flags to off for statics and interiors, thus making good use of the built in object ghosting for zones setups that really need it.

#14 It could, but that's not a good idea. Even if it was, you'd be sending lots of data (the actual lightmap) when and if the mission gets moderatly large.

There are ways to solve this, but it depends on your game and how it's designed. Where I work we use a seperate Torque installation to edit our maps, and cache the lighting which is then shipped to the client. I would look into something similiar if you seriously want to get rid of interiors which are out of scope.

You could flag all interiors as ScopeAlways upon load, and when the load is done you could set them back to regular scope. Just throwing around ideas.

#15 That's exactly what I mean Stefan, to include a lightmap with the client distribution. I do like the idea of modifying the scope flags after lighting too, but a faster mission load might be preferable. Thanks for the tip.

#16 Didn't see your post before I posted mine :) Cool.

Another cool thing that I've seen some games use (Battle Field, Neocron) is creating any eventual caches (like Lightmaps or special Shaders) upon the first time they are launched and then never again as long as the system configuration hasn't changed.

#17 Don't forget one thing guys (and no, this doesn't apply to large Atlas style maps, but the principle is still one to not ignore)--one huge advantage of scopeAlways objects is that the time required to network the initial update is done when the user is most willing to accept it--during the start up phase.

Either way, the information will have to be networked--and if you push the large sets of data to a "on demand" style of ghosting, then you are setting yourself up for a different set of problems. None are insurmountable, but you don't get something for nothing, either.

#18 I completely understand that, it was very well written in your TDN article too :) This is a bit of experimentation really. I am a big fan of the networking model and how it works and the reasons why it does the things it does (and when).

Bit by bit, I'm trying to cut down the speed of map loading as both a proof of concept, and as a little research project I guess you could say. I do realize the most secure way of doing datablocks and objects is to let the authoritative source (server) send em on down to the client and let em load when the margin of acceptability is higher: when the game is loading. [Let me qualify first by saying that it is well stated that an Indie will have a tough time finding that much content to put in to populate a single large map, that isolated zones have been used quite effectively in other games and in fact people don't seem to mind them at all.] On a single large map like a nice Atlas, or a large square size TGE map the number of interiors used and sent down to the client, as well as statics (non-replicator trees, rocks, etc) can grow to be quite high. This not only can nudge the loading time past the point of acceptability, but it can bog down the client with all the objects necessary to make the lightmap and scopealways all those objects.

To complicate the issue further, I am wondering how additional server processes can be used to run a single map. Here it is assumed that a single server process, even with all its static content offloaded, is going to have too large a number of dynamic objects to run well. It's a slightly empirical statement, ie I've seen what unoptimized script-based AI can do to a server, and even just running all those anims for more than a few hundred creatures, players, objects might be too much. So, *theoretically* how can other servers be used to share the load?

I have already prototyped datablock caching, so they are written out and saved the first time they are received. I've done this as both a single flat file (preserving order) and individually, one at a time. The idea is to local exec them before the main menu, and not clear or retransmit them between map loads unless it's a new datablock which came into play sometime after the initial blast or distribution. This should take quite a bit of time off mission load/zoning.

The next step is to take all static objects and do the same thing, cache them and synch them and do a kind of local exec/creation, working out some method of proximity loading (perhaps based on normal scoping rules, or some other block enter/exit scheme). I would like the servers to only be ghosting dynamic objects, and have several servers responsible for a single large map.

The quickest way to share a mission *I think* is to add a field to SimObject or NetObject to mark each object in the mission as belonging to one server or another. Each server therefore has a single copy of the terrain as well, and the client also knows the entire terrain and static objects. When the mission loads, the server will only load those objects it is responsible for out of the mission file/database. If I use a database it becomes more interesting, and maybe a little easier. When the player moves across boundaries he is connected to the next server, his scope list is modified, and he becomes a new object on the new server, and it begins ghosting.

As a player moves across invisible borders, I'm hoping to transfer them to the next server and make it seamless. It will involve some manner of ghosting objects to the next server I think, using some additional ID details to preserve identity.

The idea is that if the client knows about the static objects, it can be told to load them/unload them using a combination of the scoping list and some other methods. Alternatively, using normal scoping rules but taking sopealways off the statics, even the static objects could be trickled down, though it means more bandwidth use.

The point of controlobject switch, any "on the border" content, what happens to the playgui while this is going on, how to ghost to another server, etc, are still unknown complications with unknown solutions I still need to research.

Anyway, that's the why of it, that old friend seamless zoning/multiple servers per mission. Why? I made a very large and (IMO) nice looking terrain in TGE ala Asheron's Call style, and began to wonder what it would take to actually use it with several other servers.

#19 Great thread.

Quote:I've seen what unoptimized script-based AI can do to a server, and even just running all those anims for more than a few hundred creatures, players, objects might be too much.

fwiw, in the lounge, our current server-side bottleneck seems to be the determination of network scope itself. i think animations don't actually run server-side; just their high-level state is kept track of.

#20

Quote:
fwiw, in the lounge, our current server-side bottleneck seems to be the determination of network scope itself. i think animations don't actually run server-side; just their high-level state is kept track of.

I can see that breakdown in performance making sense--given the standard use case (Tribes style mission sizes), it's not an area that has been optimized, and it is after all having to do a containerRadiusSearch style poll every network tick.

You are correct on the animations server side--it simply keeps track of animation state on the server--in other words, it steps through the sequence, but doesn't actually move model polys around any.