Entity/Component System R&D Discussion
by Jeff Raab · in Torque 3D Professional · 07/24/2014 (1:56 pm) · 105 replies
Continued from here: www.garagegames.com/community/blog/view/22794
Per Dan's suggestion, lets move the hardy discussion and specifics into a thread, where the back and forth is more practical
The problem with calling anything directly is for it to correctly type, we'd have to know the specific component's class.
So lets say we want to have a few different colliaion components, to handle various circumstances:
BoxCollider, CapsuleCollider, SphereCollider, MeshCollider.
They have the same basic getCollisionInfo(), hasCollided(), etc calls, but the work they do in them is particular to themselves, right?
So if you wanted your Physics component to correctly call the component's function directly, there's 2 ways to do it.
Either we have an intermediate class 'CollisionComponent' all physics components inherit from, which bulks the heirarchy while still having us use virtual functions anyways, or the Physics component just knowing about every type of collision component we have and checking against them all.
So we have our physics component would have to figure out what component it is to correctly call the function in question. It also means we're adding to the number of headers we have to compile in, increasing compile times. If you wanted to add a new collision component, you'd have to add in the header and typing stuff for the new collider as well.
Then lets say we need to have more than one kind of physics component, such as SimplePhysics, RigidBodyPhysics, SoftBodyPhysics, etc, etc.
We multiply the above considerations for each.
Pretty fast you're going to end up with a ton of excess cross-references just so you can call the functions directly. It'd raise compile times and make adding new components kind of a pain.
With the interfaces as-is, it may be kinda ugly, but none of the above are problems.
You standardize the interface and it's functions, and those virtualize to the component's own interpretation of the interface calls, and does it's work.
This way, the components don't at all care what the other components are, and the same for Entity-Component interactions.
The code may be ugly, but the usage is a lot cleaner and robust.
Unless there's away around the nexus-of-pain of cross-references mentioned above?
Per Dan's suggestion, lets move the hardy discussion and specifics into a thread, where the back and forth is more practical
Quote:
Is there a better place we should discuss this? I was thinking... why are interfaces separate from components? AFAICT, the purpose of an interface is to say 'find me a component on which I can call "getVelocity"'. Could components not do that themselves? Since the whole system relies on dynamic_cast anyway, that is. I guess this would look like components inheriting particular interfaces.
The problem with calling anything directly is for it to correctly type, we'd have to know the specific component's class.
So lets say we want to have a few different colliaion components, to handle various circumstances:
BoxCollider, CapsuleCollider, SphereCollider, MeshCollider.
They have the same basic getCollisionInfo(), hasCollided(), etc calls, but the work they do in them is particular to themselves, right?
So if you wanted your Physics component to correctly call the component's function directly, there's 2 ways to do it.
Either we have an intermediate class 'CollisionComponent' all physics components inherit from, which bulks the heirarchy while still having us use virtual functions anyways, or the Physics component just knowing about every type of collision component we have and checking against them all.
So we have our physics component would have to figure out what component it is to correctly call the function in question. It also means we're adding to the number of headers we have to compile in, increasing compile times. If you wanted to add a new collision component, you'd have to add in the header and typing stuff for the new collider as well.
Then lets say we need to have more than one kind of physics component, such as SimplePhysics, RigidBodyPhysics, SoftBodyPhysics, etc, etc.
We multiply the above considerations for each.
Pretty fast you're going to end up with a ton of excess cross-references just so you can call the functions directly. It'd raise compile times and make adding new components kind of a pain.
With the interfaces as-is, it may be kinda ugly, but none of the above are problems.
You standardize the interface and it's functions, and those virtualize to the component's own interpretation of the interface calls, and does it's work.
This way, the components don't at all care what the other components are, and the same for Entity-Component interactions.
The code may be ugly, but the usage is a lot cleaner and robust.
Unless there's away around the nexus-of-pain of cross-references mentioned above?
#42
01/01/2015 (8:16 am)
Ah, okay, instead of that I went for schedule() to delay the start of the animation slightly, this works fine:function makeM4s ()
{
for (%i=0;%i<6;%i++) {
for (%j=0;%j<6;%j++) {
%temp = new Entity() {
position = (%i*3) @ " " @ (%j*3) @ " " @ "0.0";
scale = "1 1 1";
canSave = "1";
canSaveDynamicFields = "1";
rotation = "0 0 0";
new RenderShapeBehaviorInstance() {
template = "RenderShape";
MaterialSlot0 = "0";
shapeName = "game/art/indieMotion/m4_optimized/M4.dts";
};
new AnimationBehaviorInstance() {
template = "AnimationController";
};
};
%ac = %temp.getBehavior(AnimationController);
%ac.schedule(500,"playThread",0,"zombiewalk");
}
}
}
#43
Yeah, it's something I noted while working on it as well, I'll try and figure out a way to work around that in the next update if possible.
01/01/2015 (8:34 am)
Good to hear.Yeah, it's something I noted while working on it as well, I'll try and figure out a way to work around that in the next update if possible.
#44
Had an interesting crash happen to me though when I naively tried to put these files in the sys/ folder instead of in game where the rest of the guis are, and then tried calling metrics.cs from sys/main.cs. It would run fine if I didn't try to load the FrameOverlayGui file, but trying to load that caused an instant crash on startup, before loading anything or getting to main menu or splash screen or anything. Would this be because a server side script is trying to load a gui that should be client side or something? (Perpetually confused when it comes to server vs client scripts, sorry.)
Tracking down the crash in my debug build yielded the time of death at the following line in GuiControl::onAdd():
I had been calling metrics.cs at the end of the main.cs file, right after this:
Once I moved the metrics code and guis into the game/game folder everything worked fine, so I'm doing great, but just mention this here so the great minds present will know about it - seems like it shouldn't be that easy to make such a hard crash. :-P
01/01/2015 (9:14 am)
Cool, metrics turns out to be mostly just the two files, core/scripts/client/metrics.cs and core/art/gui/FrameOverlayGui.gui, and then yeah also the netGraphGui.gui for networking.Had an interesting crash happen to me though when I naively tried to put these files in the sys/ folder instead of in game where the rest of the guis are, and then tried calling metrics.cs from sys/main.cs. It would run fine if I didn't try to load the FrameOverlayGui file, but trying to load that caused an instant crash on startup, before loading anything or getting to main menu or splash screen or anything. Would this be because a server side script is trying to load a gui that should be client side or something? (Perpetually confused when it comes to server vs client scripts, sorry.)
Tracking down the crash in my debug build yielded the time of death at the following line in GuiControl::onAdd():
mDefaultGuiSB = GFX->createStateBlock( d );
I had been calling metrics.cs at the end of the main.cs file, right after this:
// Initialize all core post effects.
exec("./postFx.cs");Once I moved the metrics code and guis into the game/game folder everything worked fine, so I'm doing great, but just mention this here so the great minds present will know about it - seems like it shouldn't be that easy to make such a hard crash. :-P
#45
More of interest however is the fact that my framerate stays dragged down even when I turn my back on them, which must be because they're ghost always, right? Is that set in stone or will they go back to normal visibility ghosting if I just unset the switch? (going to test...)
EDIT: I appear to have been mistaken here, on further testing I definitely see a significant jump in framerate when I look away from them. There does seem to be a significant overhead involved just in ticking them or whatever, though, even when they are out of visible scope.
01/01/2015 (9:30 am)
Ah more interesting results for you. My purpose with the metrics is to benchmark how many animating characters I can get running. I'm on pretty old hardware here so don't worry about the actual numbers, but I felt pretty good about it for now - 81 fairly highpoly characters all animating drags me down to lower than I want to go, but that's a fair number of guys for what I'm looking for.More of interest however is the fact that my framerate stays dragged down even when I turn my back on them, which must be because they're ghost always, right? Is that set in stone or will they go back to normal visibility ghosting if I just unset the switch? (going to test...)
EDIT: I appear to have been mistaken here, on further testing I definitely see a significant jump in framerate when I look away from them. There does seem to be a significant overhead involved just in ticking them or whatever, though, even when they are out of visible scope.
#46
EDIT: on further research the delete crash happens here, specifically at the _destroySelf() call in the middle of the function:
In release mode when it crashed it said something about calling a pure virtual function, do we need to implement a _destroySelf function for Entity or something?
01/01/2015 (10:09 am)
Hm, more questions... what is the recommended method for getting rid of entities? When I call delete() I get an instant crash.EDIT: on further research the delete crash happens here, specifically at the _destroySelf() call in the middle of the function:
void EngineObject::destroySelf()
{
if( !engineAPI::gUseConsoleInterop )
AssertFatal( !getRefCount() || TYPEOF( this )->isDisposable(), "EngineObject::destroySelf - object still referenced!" );
// Call the internal _destroySelf().
_destroySelf();
// Destruct the object and release it in the pool.
IEngineObjectPool* pool = this->mEngineObjectPool;
void* object = this;
destructInPlace( this );
if( pool )
pool->freeObject( object );
}In release mode when it crashed it said something about calling a pure virtual function, do we need to implement a _destroySelf function for Entity or something?
#47
My concern would be if the GetComponent() happened in a time sensitive area like in Tick().
Just a suggestion. What if?
This should make the overhead for a map very small.
[ PSEUDO CODE ]
// Entity.h
Vector< EntityComponent* > mComponents;
// limits the number of components to 256
Vector< U8 > mComponentMap;
S32 Entity::addEntityComponent( U32 componentId, EntityComponent* component )
{
// already added
if ( mComponentMap[ componentId ] != 0 )
{
AssertWarn( false, "Warning: Trying to add the same component multiple times." );
return -1;
}
S32 index = mComponents.size();
mComponents.push_back( component );
mComponentMap[ componentId ] = index;
mComponents[ index ]->setOwner( this );
return index;
}
inline EntityComponent* Entity::GetComponent( U32 componentId )
{
return static_cast<EntityComponent*>( mComponents[ mComponentMap[ componentId ] ] );
}
And Jeff I was wondering what your opinion of SubDataBlocks for Components?
01/01/2015 (10:39 am)
@JeffMy concern would be if the GetComponent() happened in a time sensitive area like in Tick().
Just a suggestion. What if?
This should make the overhead for a map very small.
[ PSEUDO CODE ]
// Entity.h
Vector< EntityComponent* > mComponents;
// limits the number of components to 256
Vector< U8 > mComponentMap;
S32 Entity::addEntityComponent( U32 componentId, EntityComponent* component )
{
// already added
if ( mComponentMap[ componentId ] != 0 )
{
AssertWarn( false, "Warning: Trying to add the same component multiple times." );
return -1;
}
S32 index = mComponents.size();
mComponents.push_back( component );
mComponentMap[ componentId ] = index;
mComponents[ index ]->setOwner( this );
return index;
}
inline EntityComponent* Entity::GetComponent( U32 componentId )
{
return static_cast<EntityComponent*>( mComponents[ mComponentMap[ componentId ] ] );
}
And Jeff I was wondering what your opinion of SubDataBlocks for Components?
#48
Is this something that is supposed to be happening automatically but isn't, or something you're still getting to?
EDIT: Hm, weird... I can call clearBehaviors() and then delete() on an entity, through the console, but when I try to schedule the same thing I get a crash every time, in fact just scheduling clearBehaviors on its own without the delete also causes a crash. In debug it ends up off in the core compiler zone under "_CrtIsValidHeapPointer" in dbgheap.c, not sure where in T3D the trouble actually starts.
01/01/2015 (10:46 am)
Further results: delete works fine as long as I call clearBehaviors() first to remove the loaded components before removing the entity. Is this something that is supposed to be happening automatically but isn't, or something you're still getting to?
EDIT: Hm, weird... I can call clearBehaviors() and then delete() on an entity, through the console, but when I try to schedule the same thing I get a crash every time, in fact just scheduling clearBehaviors on its own without the delete also causes a crash. In debug it ends up off in the core compiler zone under "_CrtIsValidHeapPointer" in dbgheap.c, not sure where in T3D the trouble actually starts.
#49
I've run into occasionally flakey behavior with the delete stuff as well. I think part of the problem stems from there being a lot of moving parts when initializing or deleting entities with components on it.
In the upcoming update, it's a good deal more streamlined which I hope largely mitigates those problems. If it doesn't fully, it'll at least make it easier to track down the real causes.
I think part of it comes from ghosts being deleted 'out of order'.
I'll give it some testing for the deletion case specifically to see if it's been nipped in the bud though.
@Antony
Interesting idea. However, I would point out that GetComponent is generally intended to fetch a component by a specific type.
So if you wanted a RenderShape component, you'd do your getComponent<RenderShapeComponent>() and it'll iterate through it's list and look for the first component that is of that type.
Just trying to get a component by index is easy and already in where it just fetches via the passed index.
01/01/2015 (4:10 pm)
@ChrisI've run into occasionally flakey behavior with the delete stuff as well. I think part of the problem stems from there being a lot of moving parts when initializing or deleting entities with components on it.
In the upcoming update, it's a good deal more streamlined which I hope largely mitigates those problems. If it doesn't fully, it'll at least make it easier to track down the real causes.
I think part of it comes from ghosts being deleted 'out of order'.
I'll give it some testing for the deletion case specifically to see if it's been nipped in the bud though.
@Antony
Interesting idea. However, I would point out that GetComponent is generally intended to fetch a component by a specific type.
So if you wanted a RenderShape component, you'd do your getComponent<RenderShapeComponent>() and it'll iterate through it's list and look for the first component that is of that type.
Just trying to get a component by index is easy and already in where it just fetches via the passed index.
#50
(Incidentally, I'd favour renaming getComponent to firstComponent :P.)
01/01/2015 (4:58 pm)
Antony - I mentioned this to Jeff on Skype, but this would be my take on it:class SomeComponent {
SimObjectPtr<SomeOtherComponent> mNeededComponent;
void onAddedToEntity(Entity& ent) {
mNeededComponent = ent.firstComponent<SomeOtherComponent>();
}
void processTick() {
if(mNeededComponent) {
// do stuff
}
}
};What are the benefits of mComponentMap in your design? It looks like you could just as easily return an index directly into mComponentList. I guess having the map means you can internally reorder the component list, as long as you don't reorder the map. However, you can do the same with a smart pointer.(Incidentally, I'd favour renaming getComponent to firstComponent :P.)
#51
enum
{
ShapeComponent = 1,
BehaviorTreeComponent = 2,
CollisionComponent = 3,
ControlComponent = 4,
AnimationComponent = 5,
};
This would be the componentId. The map would tell you what index is in respect to the entity.
01/01/2015 (6:03 pm)
I'm sorry I didn't make this clear enough.enum
{
ShapeComponent = 1,
BehaviorTreeComponent = 2,
CollisionComponent = 3,
ControlComponent = 4,
AnimationComponent = 5,
};
This would be the componentId. The map would tell you what index is in respect to the entity.
#52
I forgot to mention I love what your doing Jeff. Thanks for the hard work. And you too Daniel.
01/01/2015 (6:07 pm)
I admit that I might just be misunderstanding things. What are your views on SubDatablocks? I forgot to mention I love what your doing Jeff. Thanks for the hard work. And you too Daniel.
#53
Not really sure what you mean by sub datablocks, but IIRC datablocks are basically unused in the current ECS.
01/01/2015 (6:55 pm)
Ah, I see. The problem there is that you can't have multiple components of a type, and you can't extend that enum with component types from scripts (or from a DLL, or without modifying the engine). Having multiple components of a type may or may not be an issue, but one of the core goals of the ECS is to be able to build your game by extending the engine and adding to it, not hacking in and modifying what's already there.Not really sure what you mean by sub datablocks, but IIRC datablocks are basically unused in the current ECS.
#54
01/01/2015 (8:07 pm)
So, what would the great minds here have to say re: how one would go about attaching one of the physics plugins to the entity/component model? Would you leave the plugin structure as is, and only start modifying at the level of physicsShape? Seems like if you rewrote that to inherit from Entity instead of GameBase it could work?
#55
01/01/2015 (8:37 pm)
Seems like hooking into the github.com/Areloch/Torque3D/tree/EntityComponent_Experimental/Engine/source/comp... end of the pool would be the ultimately cleanest way to go... getPhysicsRep() and the like...
#56
It would seem sensible to me to make a component for a render shape entity that would drive its movement using data from PhysX, and/or feed its movement to the PhysX simulation if it is kinematic, but I'm having a hard time seeing any benefit to rewriting the entire physics plugin system in order to frame it as entities/components/interfaces.
01/01/2015 (11:07 pm)
Hm, well, that would be fine for simple Torque physics, but what I'm interested in is plugging in PhysX 3.3.1 with all its bells and whistles. Just trying to wrap my head around exactly where the Entity/Component system is relevant, and where we just need a bunch of old fashioned "normal" code, such as Px3World etc, for setting up communication between T3D and the PhysX API. It would seem sensible to me to make a component for a render shape entity that would drive its movement using data from PhysX, and/or feed its movement to the PhysX simulation if it is kinematic, but I'm having a hard time seeing any benefit to rewriting the entire physics plugin system in order to frame it as entities/components/interfaces.
#57
01/01/2015 (11:36 pm)
@ Dan - I didn't know we were even considering multiple components of the same type. And something to consider is that the list of components can be generated by a build process. The list of the components can be in a Lua data table for instance and loaded at initialization. Sometimes ugly is faster. Components could be a limiting factor to how many actors we can have in a scene at once. I'm fairly certain that an actor made from components is going to incur more cost then one without components. The major advantage of components is modularity and reuse. Not performance and memory usage. Components help with build time not game performance. And although smart pointers are very awesome, they are more expensive than c pointers. I guess i just want to emphasize the importance of some ugly code. I mostly brought it up as a consideration. And you did that. Thank you.
#58
01/01/2015 (11:42 pm)
The reason, I brought up subdatablocks. Was that I really liked the idea of datablocks for network optimization. with components it makes sense to do something similar. but it is even more costly. i never really decided how i felt about it and wanted to know if Jeff and Dan had thought about it.
#59
01/02/2015 (1:01 am)
@chris. Not quite sure I'm tracking there. Doing a searchdump on mPhysicsRep shows gist.github.com/Azaezel/80402b2e728ac2e926cb, which reads to me as pretty tightly coupled as an exchange hook...
#60
@Antony
The biggest limitation I see to that is what Dan mentioned. While you're probably not going to have multiple animation components or render components, one must consider stuff like gameplay components. Components that exist to handle health and damage, inventory, etc.
As you start adding those, a filtered enum list gets complicated fast to maintain. You wouldn't want just a "GameplayComponent" as you run into 'can't do multiple of the same type', but then you start quickly losing out on having the type ids to quickly filter through components. It gets tricky pretty fast, unfortunately :/
As for datablocks, the E/C stuff pretty much doesn't deal with datablocks. Because components are really granular, each component gets nearly the full mask range for each component. So we can break down what's networked in a very granular way. It also only ghosts down components that designate themselves as relevant to the client, such as render components and the like.
If they don't need to be on the client to function, such as an inventory component, you can just not flag it to ghost down and it doesn't do all the networked business.
@Chris
Yeah, I've really been wanting to try and figure out how to do the physics libraries into components, but it's tricky. The best I've figured is akin to what Az was saying. You basically implement the physicsRep stuff into the component.
You could also go the other way and just specifically write a PhysXRigidBody component and whatnot and directly handle the physx api calls and the like. I haven't done too much with the physics abstraction layer or the physx/bullet apis yet, so if you've mussed with them and have some insight, I'd love to brainstorm a good approach for those!
01/02/2015 (9:35 am)
Woah this thread's gotten fast! Haha.@Antony
The biggest limitation I see to that is what Dan mentioned. While you're probably not going to have multiple animation components or render components, one must consider stuff like gameplay components. Components that exist to handle health and damage, inventory, etc.
As you start adding those, a filtered enum list gets complicated fast to maintain. You wouldn't want just a "GameplayComponent" as you run into 'can't do multiple of the same type', but then you start quickly losing out on having the type ids to quickly filter through components. It gets tricky pretty fast, unfortunately :/
As for datablocks, the E/C stuff pretty much doesn't deal with datablocks. Because components are really granular, each component gets nearly the full mask range for each component. So we can break down what's networked in a very granular way. It also only ghosts down components that designate themselves as relevant to the client, such as render components and the like.
If they don't need to be on the client to function, such as an inventory component, you can just not flag it to ghost down and it doesn't do all the networked business.
@Chris
Yeah, I've really been wanting to try and figure out how to do the physics libraries into components, but it's tricky. The best I've figured is akin to what Az was saying. You basically implement the physicsRep stuff into the component.
You could also go the other way and just specifically write a PhysXRigidBody component and whatnot and directly handle the physx api calls and the like. I haven't done too much with the physics abstraction layer or the physx/bullet apis yet, so if you've mussed with them and have some insight, I'd love to brainstorm a good approach for those!
Torque Owner Jeff Raab
[ghc]games
Currently, the E/C stuff doesn't replace anything in the engine, it exists alongside the normal Torque classes. Eventually I want it to move over and make use of prefabs that replicate all the existing game classes, but as-is, you can mix and match E/C stuff and regular Torque stuff no problem.
As for the GetComponent thing, it loops through the components associated to that particular entity.
Unless you've got some really weird super-entity going on, you're probably not putting more than a handful of components on each entity so running through them in a loop isn't that hard-hitting.
@Chris
Ah, ok. Yeah, my first guess is TorqueScript or networking having a bit of delay and the immediate call is throwing it off. When you create components that do stuff client side such as the render shapes and stuff that modify them like the animation component, it has to be networked down to the clients so the clients can render it.
So my guess is either TS is getting ahead of itself for a few of them by happenstance, or a few hadn't fully networked to your client so the calls would fail.
The next update should simplify that aspect some, it's a bit crazy currently ;)