How to use ghosting?

#1 Well for anybody who doesn't understand ghosting I will try to explain it as far as I know how.

First of all ghosting is a way to transmit objects between client and server while maintaining bandwidth (I suppose, I have yet to see this in actual practice with a lot of traffic, the demo is actually rather low traffic even with a lot of players and I have no idea how much it would take to choke it).

To do this you must create a new simobject (somehow related to netobject, which is what I used) and give it persistant fields (the persistant fields are sent to the client). The object must be declared as IMPLEMENT_CO_NETOBJECT_V1(FooObject); You must also include pack and unpack methods for this object. You must also set the ghostable net flag.

In script you would then create your object with %bar = new FooObject(); and set the persistant fields.

If the client is not ghosting yet you can use %client.activateghosting(); to get it started. Anytime you have a new object to send use %bar.scopetoclient(%client).

If the object has the ghostalways flag, it will be ghosted to ALL clients, if it doesnt then you must specify each client one at a time.

In order for the updates to work you need to create a function to change the fields and set the action update mask. The object will not be updated to the client if the update mask is not reset each time you change a field.

A good simple example of how ghosting works is missionarea.cc and missionarea.h. There is some documentation on this in NetObject.cc as well.

I am working on creating a generic ghostobject class that sends tagged fields when they are added or changed (automatically and dynamicaly from script) to make it a LOT easier to ghost, and so i wont have to go into c++ code everytime i want to add a new field to something ( I dont want my clients to have to download the exe every single time i add a new item class). If anybody is interested I can post it as a resource when I am done.

*edit* fixed a few typos and things

#2 Couple of things:

1) Take a look at the dOxygen documentation--it gives a very nice overview of the ghosting system.

2) You absolutely don't want to be manually scoping various objects to specific clients without a -very- good reason. The ghosting system uses onCameraScopeQuery() to handle when objects should be ghosted, and "un-ghosted" based on code criteria.

3) I honestly can not see a valid reason to create a new class with the intent of ghosting persistent fields. It's possible that your specific implementation strategy might need this, but if that is the case I would suggest you might want to re-address your project implementation strategy! Based on what you said you want to do, you are talking about basically making pack and unpack dynamically modifiable during runtime...and that's not somewhere you want to go for quite a variety of reasons, performance being a paramount one! In any case, you add a new persistent field, you have to go into C++ anyway...and dynamic fields shouldn't be ghosted in the first place.

#3 Zepp:
1: yeah i mentioned that......

2: WHY. There are a lot of things I won't be wanting to scopeALWAYS, specifically inventory items, interior items (objects laying around in player structures), and my game is rather inventory-heavy.

3: WHY. "I honestly can not see a valid reason to create a new class with the intent of ghosting persistent fields." I agree which is why I am ghosting tagged fields and not persistant member fields. Why should dynamic fields not be ghosted? How is performance an issue? The performance issue I am looking at is the fact that I need to add about 400 different item classes and if I do that in c++ my exe file is going to get too big too fast, not to mention the players will have to redownload it every single time I have to change one single field on an item, or add a new item (which will be inevitable). This may be acceptable on the server end, but it is not acceptable on the client-side. There is no need to have all these classes on the client end if I can just use a generic pack/unpack function that covers all these items.

Ghosted items only send packets when they are updated right? If I need objects that are constantly updated (say players or vehicles) I can just use the classes that already exist for that purpose.

But I need a way to be able to send object data between the servers, and to the client without having to rely on "command to server" and "command to client" which are much slower than pack/unpack (I tried this route already), plus I have to have the receiving server/clients create their own object based on this data, when ghosting automagically creates the item for me. It is working a LOT faster than commandtoclient/server is.

I guess I should mention that I have my game using server clusters, but I dont really want to explain that in detail for security reasons. Suffice to say, when you have players with 200+ inventory slots, please for the love of god DONT USE COMMAND TO CLIENT to send them! :)

#4 2. Because manually scoping stuff using scopeToClient is very similar to ScopeAlways. Just let the automatic scoping code take care of it, instead of manually shoving stuff into the pipe! If you have to fix a problem, do it at the source rather than fighting symptoms.

3. A generic ghost object actually sounds like it would be a great resource for people to use. It's not by any means as efficient as a hand optimized C++ packUpdate implementation will be but for some cases it could be handy. Make sure you use the network string table so that update sizes for fields are kept to a minimum.

Honestly, what you really want is a strong autoupdate infrastructure so that if you add an item type or something you can just stream a new class info table or somesuch down the wire - if you're a serious MMO, you don't want to give up any bit you don't have to in bandwidth. But it's good to have some "super flexible" tools for people to play with...

#5 1) Sorry, I did see that at the end of your post but forgot to go back and change mine!

2) Those objects are already scoped specifically to people that are within the scoping rules. Torque has a complete set of modules to do exactly what you want to do, if it's used properly.

3) What data exactly are you talking about that you want ghosted like this? I think we may be talking about completely different things, or that you may not be taking into consideration many of the reasons why Torque networking is set up the way it is. Basically, the idea is to absolutely minimize the number of updates ever sent, period...and without constructing the pack/unpack manually using the bitmask system set up already, you are going to be eating into the performance capabilities of the updates in a big way is my fear.

Given your example, it makes me feel even stronger that you won't be happy with the performance of a generic "ghost object" that auto-magically ghosts any newly created dynamic field. Personally, I would consider re-factoring my design, analyze what is the minimum amount of data that is needed to be networked for item classes as a whole, and represent that data via datablocks (which are transmitted at the start of the game in stock TGE, or can be mirrored on the client during your release with just a little work). I'm not saying that you can't do what you are trying to do, or even that it won't work decently...I'm just saying that the way you describe the problem indicates that you are going to eventually run into a wall about the amount of data you are updated each and every network tick, and that refactoring your data representations may be the more logical and efficient way around the task.

Example: Eventually, you will have defined all of the properties of the various types of objects in your game as values within specific fields. At that point, you will be able to fully implement them into your source code, and they will be part of the executable the client gets, and therefore you won't have the problem you describe.

Also, take a look at it from this point of view: let's say that you do create a new object that has a completely new type of dynamic field, and your "ghostObject" class auto-magically ghosts it to the client...problem is, what is the client going to do with this information? It's a new field, so by definition the client scripts/executable don't know what to do with the new data field without being updated themselves.

#6 Well I am under the impression that if i don't scopetoclient then the ghosted object will never make it to the client. Am I wrong? Is there some other way to send it to the client from scratch or do I just refer to it from client side or what?

I am also under the impression that the only time ghosting will send an update is when it encounters an action mask update in packupdate (which I am using) right?

Since I am using cluster servers that can spawn multiple instances (such as an item server) I am not worried about stress on the server, what I am concerned about is stress on bandwidth and the players. I do not want my players having to download a lot of information they don't really need (like a 20 meg exe file everytime something new is added to c++ code). IMO its better to have huge server software than huge client software any day, after all the server software isnt taking up 20 gigs on each players harddrive and hogging up bandwidth.

Zepp:
"Example: Eventually, you will have defined all of the properties of the various types of objects in your game as values within specific fields. At that point, you will be able to fully implement them into your source code, and they will be part of the executable the client gets, and therefore you won't have the problem you describe."

Yeah I know what you mean, but again the game will be up before it is "done", and I will be consistantly adding new stuff for the life of the game. Like I said I need to add about 400 item classes (to start) and this will just make the client source code way too bulky and I will pay for it in the end with tons of lost bandwidth from having to upload it to them so much. You are right though, I will probably end up putting a lot of them in code anyway, but having a generic ghost object will be handy for a lot things, at least as a tool to use in the meantime for testing and such.

#7 No, for objects that actually have a position in the 3D world, and are set as Ghostable in their netFlags, they will be scoped automatically as appropriate (and also unscoped when no longer appropriate). If you use scopeToClient(),the object will -always- be scoped: even if it's in someone else's inventory! scopeToClient is a script console function that actually shouldn't be used all that much, although there are occasionally reasons to do so.

actionMask is just one of the 32 different types of logically grouped data updates.

Let me ask you this: it sounds as if all of this is simply for your inventory capability (at least so far), and it sounds as if your characters can carry a very large amount of objects in their inventory. Additionally, it sounds like you are delivering a lot of data to the client each and every time they pick up a new object, regardless of if they have it in the inventory already or not.

Have you considered abstracting the inventory itself to simply being a container object with it's own rendering and user interaction capabilities? Server side the inventory would have access to each of the objects, but client side those "objects" in the inventory don't have to be (and most likely shouldn't be) actual Simulation objects. Example: One guy is carrying 25 iris flowers: are you going to scope 25 different server side simulation objects to them, or one "iris flower" object, and an isCarryingCount of 25?

I do in a way understand why you don't want a huge set of downloads for your users, but I also think you are missing a couple of points:

1) Every MOG in the world delivers content in this way. Completely new objects (with previously undefined properties, etc) are almost never created and delivered to the clients on the fly, because this eats up huge bandwidth compared to one time deliveries.

Finally, why do you keep thinking that source code will be changing? At most, you'd be creating a new datablock when you create a new object, and as I mentioned above, if you are actually defining and creating completely new tagged fields (as opposed to providing new data in previously existing fields), then you will run into a lot more difficulties than just networking...how is the client going to know what to do with a data field that it has no prior knowledge of?

#8 "If you use scopeToClient(),the object will -always- be scoped"
Thanks thats all I needed to know, I guess. I can't ever seem to get a clear answer from you guys. Anyway I dug around in the source for three days (about 36 hours) and found that I don't need to ghost the object at all, I can just use pack/unpack to send it once, and that is all I need. I still don't completely understand ghosting though, which only reinforces my original posting that we need documentation for the scriptors out there who aren't c++ geniuses or don't have the time to be digging through 500k lines of code for 3 days.

"Additionally, it sounds like you are delivering a lot of data to the client each and every time they pick up a new object, regardless of if they have it in the inventory already or not."

No, I am only delivering a lot of info when they first login (they download their entire inventory), but then again its not a LOT of info, just locations of the dts file(s) and some descriptions for the 3D inventory UI.

"Example: One guy is carrying 25 iris flowers: are you going to scope 25 different server side simulation objects to them, or one "iris flower" object, and an isCarryingCount of 25?"
He is going to to get as many that are not exactly the same. If they are the same they will stack, if not then they won't.

"1) Every MOG in the world delivers content in this way. Completely new objects (with previously undefined properties, etc) are almost never created and delivered to the clients on the fly, because this eats up huge bandwidth compared to one time deliveries."

If you mean previously undefined properties as in completely new ones that serve no purpose whatsoever because there is no supporting code then there would obviously be no point. Therefore I am assuming (also based on your Iris example) you mean properties that are dynamic and variable in which case you are wrong, a lot of MMORPGs are doing this now, and it is only going to get more complex. In SWG for example, just about every item in the game is unique and created on the fly as the player crafts it or loots it. ATITD also does this to a limited extent (with customized plant seeds). DAOC also did this but only with items that were actually crafted by players. WOW, EQ, and UO pretty much just copy from a prototype item and ad a "crafted by " string.

"how is the client going to know what to do with a data field that it has no prior knowledge of?"
Um...with new scripting sent to them on the fly and executed in mid-game... not another exe download.......

You got this all wrong, You are assuming I am sending all my info to the client. To do so would be incredibly stupid, they only need to know what they only need to know. Nothing important is done clientside. I am mainly concerned with server to server sending (again I have modified TGE to work on a server CLUSTER, not just the server to client TGE demo, I think you keep forgetting that).

Anyway I decided I would go ahead and heed Bens advice and just spend the huge amount of time putting all this crap into the source code, I will just have to take the servers down a lot when I have new crap to add. I really prefer to work with a higher level language and try to keep the source as a support for scripting, as scripting is a lot more dynamic. Maybe someday in the future when we have unlimited bandwidth and processing power it might be feasible :)

#9 By the way, I tested scopetoclient, and it doesnt scope to EVERYBODY, it only scopes to the client you specify (as long as the scopealways flag isnt set).

#10 Well, not to be rude, but when you ask a very detailed question but don't give a frame of reference (or a generic question about a very detailed sub-system), you are never going to get a "clear answer from you guys". When we don't know exactly what you are trying to do, we cannot answer in a detailed manner.

As we've said dozens and dozens of times, we understand the need for additional documentation on many systems, and a better way to present it. That's why Ben, Adam, Justin, Pat, myself, and several others (including Associates) have spent hundreds and hundreds of hours on TDN...in fact, I spent more than 40 hours writing up a TDN article on this exact topic (ghosting).

I never said scopeToClient() scopes to everyone. I said that once you have called it, it is always scoped to that client regardless of what else happens in the simulation (until the object is destroyed on the simulation side). As it turns out, that's pretty much what you want in this case, but you didn't make that clear for quite a while, and only gave it to us in bits and pieces.

BTW, you hit the nail on the head in your last statement, in regards to why we don't have an "on the fly downloadle scripting system", and why we send data in datablocks at the start of the mission. With unlimited bandwidth and processing power those things might make sense.

No, I haven't forgotten anything about you working on a server CLUSTER...because it has absolutely nothing to do with the topic at hand!

I guess what I am getting at is that if you want clear and concise answers to your design issues, then you might want to ask clear and concise questions, giving a brief explanation of what you are trying to do.

PS: I could be wrong here, but I think you are incorrect in the assumption that the games you mentioned (or any game) allow the server (users, events, whatever) to dynamically create completely new properties of items "on the fly". I can almost guarantee that most (sane at least) designs pre-define all the properties of items, and then simply populate those properties with values "on the fly".

#11

ConsoleMethod(NetObject,scopeToClient,void,3,3,"(NetConnection %client)"
              "Cause the NetObject to be forced as scoped on the specified NetConnection.")

If you check the docs via dump it will print a very similar thing.

So while I can understand that you're frustrated by thiis process, the docs on that function are right there, have been there for a long while, and, I think, are pretty clear on what's going on with that function. We generally try not to insult people's intelligence by assuming they haven't tried using dump() or looking at the implementation of the function in question - I'd think the firest thing that you'd look at if you were curious about what a function does would be the function itself!

The Tribes networking paper published several years ago by MarkF and TimG gives an excellent overview of how ghosting works. You can read the section on ghosting at www.garagegames.com/articles/networking1/ghoststreammanager.html - the paper is referenced again and again on the forums. You can also read the TNL (Torque Networking Library) documentation, which is also quite informative - opentnl.sourceforge.net/doxydocs/archoverview.html is a good place to start reading.

One key issue here is that ghost IDs come from a limited pool and that ghosted objects are deleted when they go out of scope. So using the ghost system for things like inventories is bad, because it blows through IDs that are better "spent" on the truly dynamic ideas. You want to download that once - which means NOT using the existing ghost interface at all! - and then update as necessary, not use a system designed for highly dynamic scenarios.

Actually, all the examples you're giving seem to make a compelling case for having one single (or small subset) of Item classes that have slots for a great deal of functionality, and can represent lots of different objects. For instance, the default Item class in Torque can represent general powerups, grenades, and I've seen it used for a lot of other scenarios. I think with a little forethought, you could easily expose most information to clients. Given that the client doesn't have to do full prediction for arbitrary inventory items, you can send even less information, like a model name and some sort of compressed string describing it.

But anyway, I think you understand that a generic object is going to eat bandwidth, and that ghosting any sort of static or rarely changing non-realtime content is a bad idea. So we're set. Good luck with your project!

#12 Ben gave a really good overview there...and hit the core of what I was trying to say without ever actually saying it: inventory objects such as you describe really shouldn't be ghosted...they most accurately should be handled with a NetEvent I would think, since that way they can be sent once (use guaranteed probably for delivery), and then all you have to do later if the object is used/destroyed is a NetEvent to delete them from the client's inventory.

#13 Thanks Ben, I actually ended up implementing a generic item class that reference modular objects that define its behaviors.

I went ahead and used netevents for sending any netobject through just once which i might post as a resource for scriptors, it seemed a lot more appropriate to what I am doing than ghosting, it just took a while to figure out what was going on with the networking.

As for the comments in the code, they did help a bit, but it still takes a while to go though and figure out what is happening. There are a few things such as the notification features that have no explanation at all. Anyway I learned a lot so its all good.

Thanks for the help guys.

#14 I'm a little unclear on what you did but I look forward to seeing the resource!

TNL does have full docs on all aspects of its networking and represents the canonical networking solution. But it may be a while before it makes it back into mainline Torque (it'd be a bit of a rewrite, and we have a lot of other, more important projects than fixing something that isn't broken).

#15 Is there an "in English", "NoOb","Not a programer" discription of what this is?

I have read the thread but i am unclear as to what it is still. Im not a coder, so im asking for a discription with the common man in mind.

Thanks =)

all i can guess is that the server sends some form of command to the client to use somthing the client already has (Script, datablock and "object")...i think.

/ponder

#16 If you're not a coder, you can pretty much assume it "just works" and stuff gets networked. :)

#17 A bit of an Old post, but some information that will be of use to People trying to use ghosted objects
That may have stored data about OTHER ghosted objects....

ghostalwaysset does NOT send the objects in the order they are created...

It revereses that order. Thus if you create objects and store information on these objects in the First
created object that is necessary for other Objects that come later, when it ghosts all objects on initial
connection of EACH client, the ghosted objects will be received in an incorrect order.

This is of course IF one of your Objects needs to be Created on the Client FIRST before the latter
objects are sent. This sounds clomplex, and it is...

Heres an example:
A primary object that handles all Terrain Blocks sent to the Client.
It contains a List of these objects (Terrain Blocks) and it NEEDS to be Called ( onAdd() ) before each of
the Terrain Blocks are sent and created on the Client.

It took me hours to figure out WHY the creation was a problem until I spotted the Reveresed Order.
Now once the Manager Object is ghosted (created) on the Client, any changes to terrain can be sent in
any order. And changes to the Manager can occur also, so BOTH needed to be ghosted.

When new Ghosted Objects are Created, the GhostAlwaysSet will change and NEW clients connecting
will most likely NOT get the correct order again.

These type of Issues with ghosted Objects are a few of the things to watch out for.
I worked a way around this by Sending the necessary Objects to each Client on Connect FIRST..
Then Allow ghosting of all the rest as normal from then on.

So IF you need things sent in the correct order on an initial connect, DO NOT DEPEND on ghosting.
Send it manually, then allow ghosting to take over on that connection.

Mythic

#18 Thanks for the writeup, John.

In general the "best practice" for one object referencing another in the context of ghosting is to keep pending the update for the referencing object until the referenced object is properly ghosted. Check out the ShapeBase mounting code for an example of this. But basically, keep returning the relevant flag from packUpdate until the referenced object has a valid ghost ID - and only do your logic that depends on it after its gone through.

#19 Yes, this is good two posts - the answer on my question "ghost replication order". Answer is "No order" and "Yes you have to manually track down their creation on client". How important topic and how short it is! :)) Hope Search Engine now will find this thread for future generations of "ghost replication order" askings.

Thanks again!