Entity/Component System R&D Discussion
by Jeff Raab · in Torque 3D Professional · 07/24/2014 (1:56 pm) · 105 replies
Continued from here: www.garagegames.com/community/blog/view/22794
Per Dan's suggestion, lets move the hardy discussion and specifics into a thread, where the back and forth is more practical
The problem with calling anything directly is for it to correctly type, we'd have to know the specific component's class.
So lets say we want to have a few different colliaion components, to handle various circumstances:
BoxCollider, CapsuleCollider, SphereCollider, MeshCollider.
They have the same basic getCollisionInfo(), hasCollided(), etc calls, but the work they do in them is particular to themselves, right?
So if you wanted your Physics component to correctly call the component's function directly, there's 2 ways to do it.
Either we have an intermediate class 'CollisionComponent' all physics components inherit from, which bulks the heirarchy while still having us use virtual functions anyways, or the Physics component just knowing about every type of collision component we have and checking against them all.
So we have our physics component would have to figure out what component it is to correctly call the function in question. It also means we're adding to the number of headers we have to compile in, increasing compile times. If you wanted to add a new collision component, you'd have to add in the header and typing stuff for the new collider as well.
Then lets say we need to have more than one kind of physics component, such as SimplePhysics, RigidBodyPhysics, SoftBodyPhysics, etc, etc.
We multiply the above considerations for each.
Pretty fast you're going to end up with a ton of excess cross-references just so you can call the functions directly. It'd raise compile times and make adding new components kind of a pain.
With the interfaces as-is, it may be kinda ugly, but none of the above are problems.
You standardize the interface and it's functions, and those virtualize to the component's own interpretation of the interface calls, and does it's work.
This way, the components don't at all care what the other components are, and the same for Entity-Component interactions.
The code may be ugly, but the usage is a lot cleaner and robust.
Unless there's away around the nexus-of-pain of cross-references mentioned above?
Per Dan's suggestion, lets move the hardy discussion and specifics into a thread, where the back and forth is more practical
Quote:
Is there a better place we should discuss this? I was thinking... why are interfaces separate from components? AFAICT, the purpose of an interface is to say 'find me a component on which I can call "getVelocity"'. Could components not do that themselves? Since the whole system relies on dynamic_cast anyway, that is. I guess this would look like components inheriting particular interfaces.
The problem with calling anything directly is for it to correctly type, we'd have to know the specific component's class.
So lets say we want to have a few different colliaion components, to handle various circumstances:
BoxCollider, CapsuleCollider, SphereCollider, MeshCollider.
They have the same basic getCollisionInfo(), hasCollided(), etc calls, but the work they do in them is particular to themselves, right?
So if you wanted your Physics component to correctly call the component's function directly, there's 2 ways to do it.
Either we have an intermediate class 'CollisionComponent' all physics components inherit from, which bulks the heirarchy while still having us use virtual functions anyways, or the Physics component just knowing about every type of collision component we have and checking against them all.
So we have our physics component would have to figure out what component it is to correctly call the function in question. It also means we're adding to the number of headers we have to compile in, increasing compile times. If you wanted to add a new collision component, you'd have to add in the header and typing stuff for the new collider as well.
Then lets say we need to have more than one kind of physics component, such as SimplePhysics, RigidBodyPhysics, SoftBodyPhysics, etc, etc.
We multiply the above considerations for each.
Pretty fast you're going to end up with a ton of excess cross-references just so you can call the functions directly. It'd raise compile times and make adding new components kind of a pain.
With the interfaces as-is, it may be kinda ugly, but none of the above are problems.
You standardize the interface and it's functions, and those virtualize to the component's own interpretation of the interface calls, and does it's work.
This way, the components don't at all care what the other components are, and the same for Entity-Component interactions.
The code may be ugly, but the usage is a lot cleaner and robust.
Unless there's away around the nexus-of-pain of cross-references mentioned above?
#22
07/26/2014 (7:55 pm)
Right, editors especially could be a pain point. Also, how does ghosting happen to entities and behavior instances? I assume I had to call scopeToClient because there was some lack of automatic scoping? Or because the client didn't yet have a camera to scope with? Also, even though the entity was scoped to a specific client, it seems that its behavior instance wasn't. Should we look at that?
#23
Any chance you can throw your current build up so I can have a look as well?
The only forcible scopeToClient component I have, I think, is for the camera behavior, in replicating the regular camera's functionality. It's only scoped to the relevant client. Otherwise, it should happen automatically.
07/27/2014 (12:03 am)
Entities should always ghost. If they're not ghosting for you, there's definitely something weird going on.Any chance you can throw your current build up so I can have a look as well?
The only forcible scopeToClient component I have, I think, is for the camera behavior, in replicating the regular camera's functionality. It's only scoped to the relevant client. Otherwise, it should happen automatically.
#24
EDIT: current t3d-bones implementation, featuring a basic camera that doesn't move yet. Also no actual tutorial :P.
07/27/2014 (3:14 am)
Ah, I see now where you're doing scopeToClient in the existing camera template. That makes me feel better. Sorry for not RTFMing initially. I have a strong independent streak :P. I'm going to add a couple more entities now and see how they go. My work on top of your stuff is in my ecs branch.EDIT: current t3d-bones implementation, featuring a basic camera that doesn't move yet. Also no actual tutorial :P.
#25
Of course, you could just as easily do this:
07/27/2014 (4:27 pm)
I made it fancy, so you can now do:UpdateInterface *update;
for(BehaviorIterator b = getBehaviorIterator().with(&update); b.hasNext(); b++) {
update->processTick(move);
}And you can tack as many with()s on there as you like. I like this approach because it's fairly type-safe: you can only call functions that are part of UpdateInterface, so no messing about with stuff you shouldn't.Of course, you could just as easily do this:
SimObject *so;
for(BehaviorIterator b = getBehaviorIterator().with(&so); b.hasNext(); b++) {
so->whateverILike();
}But the point is to prevent unintentional misuse of the interfaces, not deliberate hackery :P.
#26
Of course, we can't prevent ALL bad practices, but I think this is a pretty clean setup here.
07/28/2014 (9:41 am)
Looks pretty solid so far.Of course, we can't prevent ALL bad practices, but I think this is a pretty clean setup here.
#27
However, using the regular interface works fine:
Actually I'm not even sure it's a problem with constructors and move semantics. For example, this guide doesn't have a problem. Though I wonder if it's because the Iterator contains... no I have no idea :/. Maybe I need to store pointers not references. Ugh.
EDIT: while I like the idea of this interface, at least when we start to support C++11, I'm going to stop working on it for a little while. It's a bit self-indulgent, honestly, and will probably be less performant than the current getComponents() that just returns a vector. Not that it'll be a huge difference. But yeah I'm not sure I'd recommend Jeff accept this PR in the end. We'll see.
Next up... not sure. Jeff, what should I work on??
08/24/2014 (6:47 pm)
Component selection interface is getting there. I revised it to eventually support C++11, but I'm having annoying issues with move constructors on the ComponentSelection. Basically, in a C++11 range-based for, the compiler takes a && reference to the object you want to iterate over. For some reason, this results in constructing a new empty selection.However, using the regular interface works fine:
UpdateInterface *update;
ComponentSelection sel = selectComponents().with(&update);
for(ComponentSelection::Iterator it = sel.begin() it != sel.end(); it++) {
update->processTick(NULL);
}Or, until the move constructor stuff is fixed, it's still possible to do this:UpdateInterface *update;
auto sel = selectComponents().with(&update);
for(auto it : sel) {
update->processTick(NULL);
}Just don't do:for(auto it : selectComponents().with(&update)) {because then the bug shows up. Not sure why it's different from above.Actually I'm not even sure it's a problem with constructors and move semantics. For example, this guide doesn't have a problem. Though I wonder if it's because the Iterator contains... no I have no idea :/. Maybe I need to store pointers not references. Ugh.
EDIT: while I like the idea of this interface, at least when we start to support C++11, I'm going to stop working on it for a little while. It's a bit self-indulgent, honestly, and will probably be less performant than the current getComponents() that just returns a vector. Not that it'll be a huge difference. But yeah I'm not sure I'd recommend Jeff accept this PR in the end. We'll see.
Next up... not sure. Jeff, what should I work on??
#28
I'll be honest, I understand basically none of his criticisms. But they were able to achieve a 5x speedup in some of their core routines. One of his big points is 'optimise for the most common case', which I've yet to fully absorb. But one inspiring idea was that instead of writing a single function with branches for different cases, write three functions and operate them each on the applicable objects.
For example, in this slideshow he mentions that instead of checking for a state on each plane, sort the list of planes first and call the appropriate functions over the appropriate ranges in the sorted list.
It's an interesting idea, and one that would make a lot more sense in any other example he could have picked, I think.
Also, he doesn't seem to like references, which is mistifying.
10/08/2014 (2:31 pm)
Just found this article about an OGRE refactor in one of its core classes (a scene node, I think) inspired by a pretty harsh review from Mike Acton, who I don't know anything about but seems to be a big proponent of data-oriented design.I'll be honest, I understand basically none of his criticisms. But they were able to achieve a 5x speedup in some of their core routines. One of his big points is 'optimise for the most common case', which I've yet to fully absorb. But one inspiring idea was that instead of writing a single function with branches for different cases, write three functions and operate them each on the applicable objects.
For example, in this slideshow he mentions that instead of checking for a state on each plane, sort the list of planes first and call the appropriate functions over the appropriate ranges in the sorted list.
It's an interesting idea, and one that would make a lot more sense in any other example he could have picked, I think.
Also, he doesn't seem to like references, which is mistifying.
#29
10/08/2014 (4:19 pm)
I'm unsure why not a list of lists - each list containing the particular case only. Then they are "sorted" only when created instead of worrying about what range meets what case. Just process the first list in the list-list, then the second, then the third. Or am I missing something?
#30
10/13/2014 (2:08 pm)
That sounds like a valid sorting approach!
#31
Note that this is an example of the (current, column-major, possibly bad) model:
Whereas this is the (suggested, row-major, apparently better performing) model:
Or something kind of like that.
I'm still uneasy. Experiments must be done.
10/30/2014 (6:05 am)
Yet another post on Reddit asking about ECSs. Some choice quotes below. I'm really leaning towards the row-major design (a list of all components of a given type) rather than column major (list of mixed components per entity), if we can make it work with scripts.Note that this is an example of the (current, column-major, possibly bad) model:
class Entity {
vector<component*> components;
}
class componentA : public component {}
class componentB : public component {}Whereas this is the (suggested, row-major, apparently better performing) model:
class componentA {
static vector<pair<entity_id, componentA>> instances;
}
class componentB {
static vector<pair<entity_id, componentB>> instances;
}Or something kind of like that.
Quote:The big thing that people miss about components is that they're actually terrible for performance. Like you said, you'd generally have to go through every component, cast it to the desired type and then render it.
Quote:A better approach is to think of each game object as an index into multiple lists of components.
Quote:Is this slightly more complicated than the simple component model? Yes. But most game code works on the component level, not the game object level.
I'm still uneasy. Experiments must be done.
#32
Like we mount entity B to entity A. Logically we'd want B to process after A as there are probably components that would be impacted(like if we went with transform being handled via components).
In cases like that, you could get some weird stuff if you didn't re-order your component lists to have B's components after A's in the lists.
Not technically a super hard obstacle, but it throws one more complication in there to consider.
10/31/2014 (6:53 am)
The only major limitation I can see in that approach would be the odd spots. Like, say, having one entity set to processAfter another.Like we mount entity B to entity A. Logically we'd want B to process after A as there are probably components that would be impacted(like if we went with transform being handled via components).
In cases like that, you could get some weird stuff if you didn't re-order your component lists to have B's components after A's in the lists.
Not technically a super hard obstacle, but it throws one more complication in there to consider.
#33
EDIT: Never mind, found it! (Could be the function called "assignPersistentId()", lol).
12/31/2014 (10:14 am)
So, sorry if this is a stupid question, but where are you guys getting the persistentID field for these entities? Looks like a UUID but are people generating them independently and filling them in or is that part of the system somewhere?EDIT: Never mind, found it! (Could be the function called "assignPersistentId()", lol).
#34
Sorry, didn't spot this earlier.
Yeah, I don't do anything directly with the persistentIDs. At least, I don't remember add anything on there yet.
So that's handled by Torque auto-magically. I think it generally does it when you mount objects.
12/31/2014 (5:03 pm)
@ChrisSorry, didn't spot this earlier.
Yeah, I don't do anything directly with the persistentIDs. At least, I don't remember add anything on there yet.
So that's handled by Torque auto-magically. I think it generally does it when you mount objects.
#35
12/31/2014 (9:17 pm)
Ah, glad you're back! Do you know anything about why I can't seem to find the metrics() function in your build? Don't actually know where that lives in stock Torque, but in your build it's an unknown function so far.
#36
I wonder if you or anyone could shed any light on my current research problem... I'm trying to animate rendershapes. I have a functioning entity with a RenderShapeBehaviorInstance attached, and a working AnimationController that can play any of that shape's animations.
My issue is probably a simple torque script syntax confusion, but I'd like to be able to create these entities and then call functions on them, and my naive attempt looked like this:
This fails dramatically. I also tried using an assignment on creation of the animation controller, ie "%accontroller = new AnimationBehaviorInstance(%acname)" and then "%accontroller.playThread(0,"zombiewalk");" but this also failed.
Could somebody possibly fill me in on how I should be referencing these entities and their components from within a loop?
So happy to see this tech advanced this far however! T3D has needed this for years. =-)
12/31/2014 (11:14 pm)
OMG this is so cool, thank you so much for all your hard work!I wonder if you or anyone could shed any light on my current research problem... I'm trying to animate rendershapes. I have a functioning entity with a RenderShapeBehaviorInstance attached, and a working AnimationController that can play any of that shape's animations.
My issue is probably a simple torque script syntax confusion, but I'd like to be able to create these entities and then call functions on them, and my naive attempt looked like this:
function makeM4s ()
{
for (%i=0;%i<6;%i++) {
for (%j=0;%j<6;%j++) {
%name = %i @ %j;
%acname = %name @ "ac";
new Entity(%name) {
position = (%i*3) @ " " @ (%j*3) @ " " @ "0.0";
scale = "1 1 1";
canSave = "1";
canSaveDynamicFields = "1";
rotation = "0 0 0";
new RenderShapeBehaviorInstance() {
template = "RenderShape";
MaterialSlot0 = "0";
shapeName = "game/art/indieMotion/m4_optimized/M4.dts";
};
new AnimationBehaviorInstance(%acname) {
template = "AnimationController";
};
};
%acname.playThread(0,"zombiewalk");
}
}
}This fails dramatically. I also tried using an assignment on creation of the animation controller, ie "%accontroller = new AnimationBehaviorInstance(%acname)" and then "%accontroller.playThread(0,"zombiewalk");" but this also failed.
Could somebody possibly fill me in on how I should be referencing these entities and their components from within a loop?
So happy to see this tech advanced this far however! T3D has needed this for years. =-)
#37
Are you getting a crash?
It's possible that the call to playThread is happening before everything is done initializing, but the code in general looks more or less correct.
As for metrics, I forgot to copy that and a few other functions over from the old templates to the E/C one.
In the next update I'm trying to wrap up stuff like that carried over.
01/01/2015 (12:26 am)
Can you clarify what you mean by "fails dramatically"?Are you getting a crash?
It's possible that the call to playThread is happening before everything is done initializing, but the code in general looks more or less correct.
As for metrics, I forgot to copy that and a few other functions over from the old templates to the E/C one.
In the next update I'm trying to wrap up stuff like that carried over.
#38
Metrics is in scripts somewhere. I pulled them into their own lib in t3d-bones. Look for filenames similar to these.
Jeff - the processAfter example is an interesting one. I'm not sure what effects you'd see, especially since in that sort of data-oriented design, you don't 'process entity 1 then process entity 2'. You 'process all projectile physics, process all mission area checks, process all collisions' etc. It's split up by system, not by entity, so there's less of a concept of one entity being done before another. In addition, I've seen it discussed that your update logic should actually be entirely pure. So, all your world data is duplicated, and instead of doing destructive updates, your source data for one update is all immutable, and the result is written to a different location. Which means the order is irrelevant, you'll get the same results every time.
Anyway, that's all theoretical, but it's something you could pretty feasibly do if your ECS architecture was decoupled enough. John Carmack talked about doing just that at some idTech talk a while ago. QuakeCon, I think.
I'm interested in how the recent work is coming and how soon I will be able to contribute ;).
EDIT: more reading on data0oriented design, should you be interested. For the record, I'm not saying we should go that way, at least not yet.
01/01/2015 (1:13 am)
EDIT: missed a few posts while internet was down. Reading now :P. Yeah looks like my post is still sane haha.Metrics is in scripts somewhere. I pulled them into their own lib in t3d-bones. Look for filenames similar to these.
Jeff - the processAfter example is an interesting one. I'm not sure what effects you'd see, especially since in that sort of data-oriented design, you don't 'process entity 1 then process entity 2'. You 'process all projectile physics, process all mission area checks, process all collisions' etc. It's split up by system, not by entity, so there's less of a concept of one entity being done before another. In addition, I've seen it discussed that your update logic should actually be entirely pure. So, all your world data is duplicated, and instead of doing destructive updates, your source data for one update is all immutable, and the result is written to a different location. Which means the order is irrelevant, you'll get the same results every time.
Anyway, that's all theoretical, but it's something you could pretty feasibly do if your ECS architecture was decoupled enough. John Carmack talked about doing just that at some idTech talk a while ago. QuakeCon, I think.
I'm interested in how the recent work is coming and how soon I will be able to contribute ;).
EDIT: more reading on data0oriented design, should you be interested. For the record, I'm not saying we should go that way, at least not yet.
#39
I noticed in earlier conversations that to find a given component. GetComponent() would loop through the components. How about if we say you can't have more than 255 components and create a very small map using indices to access the component faster?
Doesn't the idea of components lead to the concept of subdatablocks? to gain the benefits of datablocks for components
01/01/2015 (1:59 am)
Is there anyway we could have components be optional so that we could choose between the two? In some cases components may use more overhead than someone would want.I noticed in earlier conversations that to find a given component. GetComponent() would loop through the components. How about if we say you can't have more than 255 components and create a very small map using indices to access the component faster?
Doesn't the idea of components lead to the concept of subdatablocks? to gain the benefits of datablocks for components
#40
Am going to experiment next with saving an array of ids for all my guys, and then going through and starting the anims in a second loop after they've all been created, see if I'm just having a problem with torque script falling behind...
@Dan, thanks for that, will look there.
01/01/2015 (7:39 am)
@Jeff, sorry, I should have been more specific. No crashes or anything, it doesn't fail that dramatically! It just doesn't do what I want, which is to start animations on all the guys I'm creating. A couple of them will actually start playing the anim, but the rest won't get the message.Am going to experiment next with saving an array of ids for all my guys, and then going through and starting the anims in a second loop after they've all been created, see if I'm just having a problem with torque script falling behind...
@Dan, thanks for that, will look there.
Torque Owner Jeff Raab
[ghc]games
If you look at the behaviorObject and Behavior Template/Instance un/pack functions, they pretty much build a list of things that should be networked, and will keep trying until it works.
It may be cleaner, but without lots of retooling of the loading code in GameConnection stuff, this was the best way to make sure everything passed along in-order(which was a major point of breakage in testing before).
This ties into the issue with making templates option.
Technically, yes. A template will set up the initial fields on a instance(and is used to tell the editor what fields to build in the behavior interface stuff, so that's actually really hard to optionalize if you want editor support), but after the initial setup, the instances pretty much do their own thing.
The problem comes from networking order. If a instance is relying on the template, as the system has now, we NEED to have the template get passed along first just so things run smoothly. So we'd have to either eschew checks that the template got networked first(removing the assurance that everything is networking in a usable order) or some other way I hadn't thought of.
Tl;DR, yeah, we can look at optionalizing templates, but there's some stuff you'd have to rework(especially editors) to make it actually work right.