LuaPlus binding with Tilde debugger.

#21 Hi :)

We have a lot to think and research before T3D SC can take a desission about merge.

I think it's better you dont start any big task are not necesary for make Lua work. We need to try first a implementation with TS MACROs and callbacks.

After holidays I want to try if we can modify callback target ( TS or Lua ) based on Lua functions declaration, not sure if you have tried this.

If we can have this work we can do a safe merge of a Lua integration in T3D, and get time to rethink what we need to clean for T3D 4.0

#22 Antony:

The reason I was asking about integration to the macros is more to avoid needing to build out all those ClassName_FunctionName functions manually.

It means additional workload in adding new functions or classes that are script-accessed, which is why I was figuring it'd make sense to make the Macros automate generating the bindings rather than the manual creation of the scriptBinding functions.

If the macros could be repurposed towards a more general-use means, that'd make things a lot easier for not just adding lua in this manner, but other script languages people would be interested in, right?

#23 Sounds good Luis. I would really like James to weigh in on this, since he has actually made attempts to do this very thing. Maybe we use what he has? Please do look at what I have and try and see where I would like to go with this stuff before you give up on it. I wouldn't want to give up on something, without spending the appropriate amount of time investigating it. That is all. Thank you. :)

#24 Hi Jeff thanks for your input, but my ultimate goal is to not be dependent on those Macros at all. In fact I want to move them out into their own file. If this makes sense. I hope I am understanding what you are saying. I would eventually like those Macros gone. This is what I mean by removing the torquescript bindings. :)

#25 2 clarification queries from Joe User:
1) What technological hurdle does breaking things out from class definitions solve.
2) how do you intend to handle github.com/antonyjones67/Torque3D/blob/development/Engine/source/scene/sceneObje... (script and editor exposure for class variables)

#26 I hope I am understanding you correctly.

The only reason that I can think of at the moment for moving the macros outside of the class definitions is so that eventually we can remove all dependencies of TorqueScript. And use another scripting language in it's place. This could lead to multiple scripting options without TorqueScript overhead.

My answer to your second question is, that I do not have one. My thoughts were that we start moving in this direction and keep using torquescript GUI and eventually replace the torquescript GUI with something else.

Remember this was a long term plan I was mulling over. But I was thinking we could make small steps.

Using a Lua-Bridge is also a great idea. When I started working on this binding I thought what if this could be a stepping stone. TorqueScript makes a lot of sense if it is a proprietary scripting language, but now that it is part of a community I thought that it was a good idea to move things towards a more standardized way of doing things.

Lua is used by a large community. There is a lot of documentation on the interenet for lua including free debuggers.

Anyway I definitely do not have all the answers. That is why I started this thread. By no means am I married to this direction. I really just wanted to try and help.

#27 Perhaps
github.com/GarageGames/Torque3D/blob/69838bdc8c9bc055b9b1ae76f42b0f28d2a33909/En... + github.com/GarageGames/Torque3D/blob/development/Templates/Full/game/art/datablo... would clarify point 2.

#28 This is how you currently create and set fields in SimObjects and SimDatablocks with the bindings I have set up.
Torque3d LuaPlus Documentation

Adding a SimObject.

AL_FormatToken = {}
AL_FormatToken.userData = addSimObject(
{
superClassName = "RenderFormatToken",
className = "AL_FormatToken",

enabled = "false",

format = "GFXFormatR8G8B8A8",
depthFormat = "GFXFormatD24S8",
aaLevel = 0, -- -1 = match backbuffer

-- The contents of the back buffer before this format token is executed
-- is provided in $inTex
copyEffect = "AL_FormatCopy",

-- The contents of the render target created by this format token is
-- provided in $inTex
resolveEffect = "AL_FormatCopy",
}
)

The structure returned by the call addSimObject is of type UserData. You can call functions and refer to properties in a UserData, but it is not intended to be modified. If you want to add functions to this class you can create a new table pointing a field to the userData.

Adding a SimDatablock.

PlayerSplashMistEmitter = {}
PlayerSplashMistEmitter.userData = addSimDatablock(
{
ejectionPeriodMS = 5,
periodVarianceMS = 0,
ejectionVelocity = 3.0,
velocityVariance = 2.0,
ejectionOffset = 0.0,
thetaMin = 85,
thetaMax = 85,
phiReferenceVel = 0,
phiVariance = 360,
overrideAdvance = false,
lifetimeMS = 250,
particles = "PlayerSplashMist",
}
)

The structure returned by the call addSimDatablock is of type UserData. You can call functions and refer to properties in a UserData, but it is not intended to be modified. If you want to add functions to this class you can create a new table pointing a field to the userData.

#29 This is the Lua binding for fields.

All you have to do in Lua is:
local pt = Point3F( 2.0, 4.0, 5.0 )
or
pt.x = 2.0
pt.y = 4.0
pt.z = 5.0

This is not currently setup for all classes.

This is the c-api for the binding

void Point3F_construct( Point3F* classPtr, LuaPlus::LuaState* luaState )
{
classPtr->x = LPCD::Type<F32>::Get( luaState->GetCState(), 1 );
classPtr->y = LPCD::Type<F32>::Get( luaState->GetCState(), 2 );
classPtr->z = LPCD::Type<F32>::Get( luaState->GetCState(), 3 );
}

void Point3F_destruct( Point3F* classPtr, LuaPlus::LuaState* luaState )
{
}

void RegisterScriptVars_Point3F( LuaPlus::LuaState* luaState )
{
LuaObject luaMetaTableObj = luaState->GetRegistry()["Point3F"];
LPCD::Metatable_IntegratePropertySupport( luaMetaTableObj );
LPCD::PropertyCreate( luaMetaTableObj,"x", &Point3F::x );
LPCD::PropertyCreate( luaMetaTableObj,"y", &Point3F::y );
LPCD::PropertyCreate( luaMetaTableObj,"z", &Point3F::z );
}

#30 T2D has an interesting approach where all the DefineEngineMethod calls and similar are in separate files adjacent to the class files they refer to. So you end up with sceneObject.h, sceneObject.cpp, and sceneObject_ScriptBinding.h. Would that make it easier to replace those macros with something that lets you define Lua callbacks more easily? I'm thinking of some replacement to engineAPI.h which instead of generating TS code, generates the Lua code you'e currently got somewhere.

I think the question I still have is if you get rid of the DefineEngine* macros, don't you lose the single point of change that you can make to define scripting stuff? Unless each particular language script binding is autogenerated from the contents of scriptbinding/ which is kept in sync with the engine.

#31 Yes so that is my ultimate plan, but not having the script binding in with the sceneObject.cpp, but instead in a similarly structured set of folders with the classes having their own separate files in the scriptbinding/torquescript folder. So that the binding is clearly separate. The big file is just an intermediate step. It makes things easier for me to dig through stuff right now. But ultimately the sceneObject binding is not in the actual sceneObject class header or source files folder. Dan did you look at my lua_intrusive branch? you will see how i pulled out the ts macros into their own separate file. Just imagine instead of the macros all in one file having seperate files in that folder.

#32 Yep, that branch is what I'm basing my comments on. T2D actually puts these script binding files next to the implementation files they go with - is this a problem? It seems neater to me :/.

#33 it seems like we would want to place all of the torquescripts together for portability. If someone wanted to get rid of TorqueScripts you just delete the folder with all of the TorqueScripts in it. This is true for Lua as well. But it really doesn't matter to me how that part goes down the most important thing to me is that the Macros don't live with the definition of the classes. I believe by separating the bindings it opens doors for us. Of course this is just my opinion. I want to do whatever works best. Ultimately I need to know whether the community understands why I did what I did and that they see it as a beneficial change. I also really need to know that it will get used in some fashion. Not having it integrated into the main branch at some time defeats the purpose of me spending any time on it.

#34 Yo Antony. As a long term tinkerer of TorqueScript and Torque, my thoughts are as follows.

First I should stress these opinions are my own and in no way reflect the Steering Committees views since I am not on the Committee.

I am a bit puzzled why you chose to write your own tool to generate the bindings instead of using something like SWIG which would likely have saved a lot of time and achieved the same result. Also to be frank I think the binding code looks like a mess and unnecessarily duplicates code. Personally I prefer the normal torque approach of having macros and such in one place which is easily modifiable and doesn't require external tools in the build process. Quicker to relink, and also keeps documentation for said functions in one place.

This ends my critique of the code. I could probably go on but lets think about some other things this brings up...

TorqueScript itself is not really a "proprietary" language per se as everything which defines it is open source along with Torque3D. "But nobody knows it!", you might retort. Fair point, though one also needs to remember even if you ditch everything and replace it with lua you've still got to have an API which people need to learn and wrap their heads around. Also the runtime complexity of lua is probably 10x that of TS at the moment (which may be good or bad depending on what you want to do in script).

I do not think the multiple scripting language approach (using different engines) is particularly viable. In particular you end up with a ton of sub-optimal complicated binding code and there is a clear lack of direction. There is also the problem of mapping class types to the correct paradigms in the target scripting language which IMO almost always ends up being awful. You also have the life cycle of objects to worry about. Really you can only pick one language at runtime, so your probably going to have the throw all the editors out of the window to maintain feature parity. IMO It's best to go with a specific language or technology and design specifically around that.

The only viable approach to multiple scripting languages (using the same engine) I've seen has been WLE's .net code.

Now here comes the deal breaker. Are you really going to get people using Lua instead of TorqueScript? Based on my experiences discussing TorqueScript and alternate languages on here.... probably not. For instance I know for a fact I wouldn't be able to get anyone else from mode7 using anything but TorqueScript for torque. Others may think differently however. I know for instance BeamNG really love lua.

I think the core problem is you're attempting to rip out a core part of the engine experience and replace it with something else. Really what you end up with is effectively another engine. In which case... perhaps this would better be suited as a fork?

I don't mean to sound like a party pooper but I haven't heard of many success stories where an Open Source game engine has had its core scripting language replaced with another. Plenty of long forum threads which have gone nowhere though. ;)

@Daniel

Unfortunately with T2D putting the binding code in header files doesn't really help too much since in order to use most of T2D you need the whole console system working since it uses the namespace code for components and such.

As for engineAPI, it could certainty be used to generate lua binding code for functions. The only thing it wont be able to do is provide exposure of object fields which is exposed through the AbstractClassRep system. Also you've got those field accessor functions to consider. ;)

#35

Lua? Doesn't he play football for Real Madrid?

#36 Thanks for the detailed review, James! A couple of points. (It's late, so I'll keep this brief, and focus mostly on the social issues instead of the technical ones.)

Luis has suggested that if we were to try to support multiple scripting languages, the editors should be written as a C++ module. This is obviously a super long-term goal, and might even have to come after making DLL plugins viable. But it does seem like one of the more reasonable ways to maintain total feature parity across multiple scripting languages.

The engine experience you talk about is evidently subpar, or more people would be using Torque. I personally believe TorqueScript is part of what's holding the engine back in the eyes of people who look at it and then move on. So I really don't think it's something we should be holding onto.

Similarly, while the opinions of the people in this community are important to us, it's really not worth our time and effort to maintain this engine solely to serve the interests of the 50 or so regular users that are active here. (That's an off-the-cuff guess; I'm sure there are more than that, but it seems the pool of regulars who I see on the forums is roughly that number.) I myself believe that scripting language availability is a significant factor for people choosing engines. If people here love TS, well, that's great, but apparently nobody outside this community does, and they are, to be honest, the target audience we should be after.

(I'll say again that I'm very convinced that the community-wide love for TS is actually evidence of severe mass Stockholm syndrome ;).)

Maybe if we had other killer features, the scripting language wouldn't be a problem - people used UnrealScript for years because Unreal was the engine of choice, and you put up with what you had to. But really, we don't have those killer features; the question becomes whether we should be focusing on finding some, and whether TS will hold us back from implementing them, and at what point we should just call it a day and use a different engine if our requirements really are such.

#37 I think you hit the nail on the head in your last paragraph dan. Lack of killer features, that is what is holding T3D back. Just for example, it's all but 2015 and we are still stuck using Direct3D9. Let's just say for example T3D used Lua (or C# or whatever super popular scripting language), would that really change anyone's opinion of T3D? Or would a sexy Direct3D11/OpenGL 4 render system pumping out PBS (just an example) change peoples opinion? Personally i think you would find it would be the latter and people would just put up with whatever scripting language T3D uses.

Haha sorry waaaaay off topic here. Probably needs it's own thread

#38 WOW! This feedback is awesome!
James: Thank you for the critique. Yes it is a bit of a mess and it duplicates code. The duplication is there to support redirecting the torquescript binding by moving the macros and pointing Lua and torquescript to the same code script funcs. My intention was to clean stuff up as I go along. I work 40+ hrs with 4 kids so I don't get a lot of time to work on this stuff so I try to work as quickly as I can. Before I went too far though I really wanted to know if this was a worthwhile direction for the community. Let me tell you why I "need" Lua for my project. I have found in my business, that there is nothing faster or as flexible to use for loading data tables. Even if I decide to use TorqueScript for the api, I would use Lua to load data which is perfectly reasonable just to use lua for that. I wasn't really concerned until I had written a behavior tree. Where I had routers and behaviors using snippets of script code. The problem I had with TorqueScript is what made it so cool. The inheritance stuff in TorqueScript which helps to overcome so many issues in other areas was costing me in performance mainly in the casting. Remember I said I like it. Using Lua gave me the option to be more direct. Basically if I don't write the code right, it won't work. I have to manually cast it. But at least I am aware that I am writing code to get around something. Maybe I am being silly, but since this code was getting slammed. Because it was every frame. Another thing I mulled over was the fact that Lua's metatables and dictionaries are very powerful tools. I started this for me. And originally I had considered a bridge( which in my vernacular means using the Torque MACROS to sneak in code to implement a binding ). I went this other direction in a attempt to make this proposal.
Please don't misconstrue me(Steve):) I am not bagging on TorqueScript. I am trying to provide options. The thing is I really need people's vote. One way or another. Because I don't want to invest anymore time into something that will never get used. :) So it doesn't hurt my feelings if the Committee says this is not something we need. I will go a route that makes more sense for me personally.
Man you guys are awesome for the great feedback. You made my day. :)
I really want to help the Torque community, but I am limited in what I can do. For business reasons.
There are some really great people that are a part of this community.
Thank you guys.

#39 @Daniel

"The engine experience you talk about is evidently subpar" - could you please elaborate on this?

I really have to disagree with your opinion on TorqueScript. Saying TorqueScript is holding people back is pure conjecture - I haven't heard anyone specifically say "Well I'd use Torque if it wasn't for that pesky TorqueScript!". Most criticism I've heard about Torque revolves around the bad press generated during the poor mismanagement of the community during the Instant Action era, which to a certain extent I would argue continues today e.g. the seeming lack of interest in properly managing spam on this forum, and your apparent contempt of the current user base. There is also a fair amount of critique of the code being a mess (this is however reasonably normal for a game engine).

From what I have seen, more people like TorqueScript than not despite its limitations. It works well for what its supposed to be used for: glue code.

In the end, bashing TorqueScript is just a scape goat for the more serious problems Torque has.

IMO multiple scripting languages is misguided. It is probably the opposite approach of what an engine should be doing: providing a consistent interface.

@Timmy

Your engine can have all the amazing killer features in the world, but it doesn't mean anything without good PR and a good userbase.

@Antony

I know this stuff is hard, and I appreciate the effort you have put into this. I am sure someone wanting to use lua will be fairly interested in using your code. :)

#40 Historical note:

The EngineAPI was supposed to replace the macro-based binding mechanism with a template-based binding mechanism. This would provide some much-needed type checking along with other things. At the time that they started on it C++ didn't have variadic templates, so most people didn't use the new system (the macro system allows for an effectively unlimited number of parameters to be bound to a generated binding function).

Performance note:

Processing every AI entity's behavior tree every frame is expensive. It might be wise to consider splitting the units into "batches" and processing them less frequently. Of course, your mileage may vary....