LuaPlus binding with Tilde debugger.

#61 Myself and Tim Newell put in a few hours of effort to identify bottlenecks in the main templates caused by torquescript. We couldn't identify any. Torquescript is not used for speed critical code and nor should it be. It's a scripting language, not a programming language.

Example: UE4's blueprints are up to 10x slower than code written in C++. These scripting languages were never meant for speed. It would take a lot less effort to write your speed critical code in C++ than spend all the time replacing TS with a scripting language that is fast enough for what you intend on using it for.

With that said, variety is the spice of life. More scripting languages don't hurt, but I don't think they need to make it into the main repo and I don't think TS needs to be removed. Especially since it will kill all the existing resources, tutorials, etc. Just my opinion =)

#62 My opinion:

Lua is faster and lighter than TorqueScript. It's also more elegant. Replacing TorqueScript completely with Lua would result in a huge amount of work and I don't feel that it would be worth that effort.

That being said, having the option to add it as others have done is great. And adding another set of standing script binding files isn't a big deal in my book. Using Luna it would be simple to do it as it stands now - just grab your arguments off the stack, call the current binding functions and push what they return onto Lua's stack in your binding function. I'm pretty sure you'd have to inject the definitions into the class definitions, but it wouldn't be terrible. The LuaJIT is most likely a faster option. Of course, I'm talking theory here because I haven't actually done this with Torque....

[EDIT]

I guess my real stance is that only you can decide how much work a given feature is worth to you.

#63 Andrew: i would like to emphasize again. I'm not talking about getting rid of TorqueScript. Just rearranging things so there is a layer of abstraction allowing options. I really don't care if this gets killed. I just want to make sure we are killing it for the right reasons.

#64 BTW Thank you all for contributing in a constructive way. I really do like this community. It is very exciting to brainstorm with so many awesome people. LuaJit is such a great direction to go. I heard it is being adopted on lots of different kinds hardware. Exciting stuff.

#65 "Myself and Tim Newell put in a few hours of effort to identify bottlenecks in the main templates caused by torquescript. We couldn't identify any"

We have found quite some. For example TS is called every frame to set some parameters for SSAO. Also moving lights (which we use a lot) were triggering TS execution every frame.

Putting aside the design problems (why set shader parameters every frame?) for a moment. If a faster scripting language was used, then not only we could use it for simple cases but for more complex cases too. In this case, speed is an enabler, that increases the % of the C++ code that would benefit from a move to the scripting side.

#66 Concerning LuaJIT bindings, in our proof of concept prototype we used lua-intf:

https://github.com/SteveKChiu/lua-intf

and also considered LuaJIT FFI:

http://luajit.org/ext_ffi.html

For an example of how FFI can be used, you can look here:

https://github.com/godfat/cubeat/blob/master/cubeat-core/

There is also a nice presentation concerning LuaJIT FFI bindings to a game engine, but i'm not able to find it right now.

Also a very interesting post on this, from the author of LuaJIT (Mike Pall):

http://article.gmane.org/gmane.comp.lang.lua.general/84483

#67 @ Lefteris

We identified those same areas but found no improvement from replacing them with C++. The cost of executing a few TS functions is unfortunately negligible ( I was actually hoping it would be a bottleneck that we could improve. )

The only example I've seen of a direct measurable improvement in speed is with Winterleaf's AI example. Writing the AI in C# showed great speed improvement over writing it in TorqueScript (is this surprising to anyone?). Even in this case I still feel like the example is written to prove the point.

Regardless, as others have said this conversation has happened countless times. The pros and cons are not cut and dry and there isn't a right answer. There is no sentence at this point that can be said that will cause everyone to say "Aww, okay, lets change to Lua then". Winterleaf did everything they could to push C# and it's a completely viable and working option that's there for anyone who wants to use it. Lua will likely have to fall into a similar category because you'll never get a consensus to replace TorqueScript as a whole.

If it's just an abstraction layer that makes it easier to drop in new languages while keeping torquescript in tact, then I've got nothing further to add. That's a perfectly fine idea and I don't see why anyone would oppose that.

#68 Tossing my 2 cents in on the general idea of multiple script languages pretty much jives with what a few others have said here.

I like TS and think keeping it and just improving it is the best route for the core of the engine.

However, I also think that enabling a sane method to let others integrate the language of their choice with minimal pain, such as an abstraction layer or whatever, is a pretty reasonable goal.

Improve the core, and enable others to expand it to fit their needs.

Projects like this help give ideas of the ideal way to enable that. This involves changing the bindings into the engine to make them more language-agnostic, while WLE opted for a separate tool that's run to generate the bindings for the end user.

Which is better? I guess we need more testing to find out ;)

#69 I'm interested in LUA not only for the generic "helper scripting language" concept. I see LUA as a concept that will enable certain architecture choices on the T3D codebase.

For example in the multithreaded scenario that i've previously talked about, each T3D thread will have its own LUA VM running inside it. This architecture choice is nearly impossible with "fatter" VMs.

Another example is the shader templating code that is now implemented on the C++ side. IMHO, using C++ code for text templating, is very suboptimal. I prefer to move the shader templating on the LUA side by using a LUA text templating library.

Both of the above architecture examples require for LUA to become a core part of the T3D engine, meaning that T3D won't be able to function at all without the LUA subsystems.

#70 Also about: "The cost of executing a few TS functions is unfortunately negligible"

It shows in our profilers. It isn't the few instruction that is the problem, but the high context switching cost (between C++ <-> TS).

#71 @ Lefteris

I think at this point it's clear it's not going to replace torquescript and this thread is more about the best way to create an abstraction to support multiple scripting languages in the future. For that reason I think our discussions (or anyones for that matter) on TorqueScript VS Lua are counterproductive and derailing the thread.

I'll let you get back to it Antony.

#72 Lefteris: I have also heard the same thing from Josh Jensen about the context switching. That seems to be a very real problem with scripting languages. It has been very nice hearing your feedback on Lua.

Moving the IMPLEMENT macros into .cpp files and redirecting those to static scriptbinding functions is pretty straight forward.

The parts that made me nervous. And of the reasons I really wanted this thread. Had to do with changes to Torque Source.

I will just think aloud:

1. Since the DECLARE Macros are inside the class declaration. It makes them harder to separate.

2. It would be much easier to access member variables if they were Public instead of Private or Protected.

I might have to create Macros that are much more generic. If you guys have feelings about how this should be done. This is your chance. I would like a way to do things while creating a minimal amount of change. I am in an area that has the greatest potential for creating bugs. So I want to be careful about it.

It would be nice if there was some way for me to know how many people would like these changes.

Or if anyone would be willing to aid in taking things in this direction. James? :)

#73 @Anthony:

if u want to see how WLE got C# and TS both to work in the engine, go to www.omniengine.net. there is a github button at the top of the page.

have fun, hope it helps.

#74

Quote:voila only one set of static functions that 2 bindings are pointing to. if you want later you can have python point to those same statics. voila 3 scripting languages sharing the same code.

What I've never understood is how you arrive at the bindings which point to those static functions, and why the functions need to be static. One, can't the bindings point to member functions? And two, do we have to maintain these bindings by hand? Or do we have to write a C++ parser to look for... stuff? Apparently EngineAPI includes a (half-finished?) system to export the console API as an XML file, which seems like it would be helpful - but doesn't that use the existing macros to determine what's exported?

Lefteris - hi, great to hear from you. I'm really interested to see how BeamNG is going about integrating Lua. I think the points you mention, especially multithreading, are very important. Speed less so, but it's always nice :P. If BeamNG is pushing hard on Lua, then it'll be great to see how it goes, and if it starts to gain traction, could even be a viable way for the engine to continue in the future.

Also, merry Christmas all :3.

#75 @Duion

I made a post about a couple of years ago regarding this which led to the console refactor branch. ;)

@Lefteris

I have to admit I was impressed with the benchmarks of luajit. However there turned out to be a rather major show stopping bug wth regard to OSX x64 support. If the vm doesn't exist within the correct 32bit address space, it will fail to initialize. I couldn't find a solution which reliably worked with torque, and it seemed to be such a bizarre oversight in the first place. The normal interpreter still worked fine, though the speed of that is far less impressive. ;)

Personally while I think lua's state being non-global is great, I haven't found much practical use for it which wouldn't have been better solved by having some dedicated c++ code or compute shader. But of course everyones project is different. For instance we just use it to setup objects and generic game state, not run physics calculations. ;)

I've probably seen as many horror stories as well as good stories about lua littered around the interwebs. Posts by the developers of the godot engine on developing their own scripting language after trying out lua, squirrel, python and others are fairly insightful.

Problems with TorqueScript can be solved without throwing in the towel IMO, though as I mentioned there is the issue of how interested everyone else is within improving it. The response from this thread at least has convinced me there is at least a case for it.

Setting shader parameters was also a problem I noticed. It can pretty much be solved by rearranging the calls in postfx, such as what I did in my cortexgl branch.

@Daniel

See https://gist.github.com/jamesu/3780691 and https://gist.github.com/jamesu/3842450 .

#76

#77 Wow. I thought we concluded that we aren't getting rid of TorqueScript, but providing an abstraction to make alternatives easier to add. You have nothing to fear Steve. :) Something also to consider that nobody has brought up yet is debuggers. An opensource debugger is a good thing to package with an opensource engine. Although I must admit that Torsion is priced very reasonably. I would love to see a visual scripting language in Torque's future. James now that it is understood that TorqueScript is not going away. Do you feel more comfortable about the direction I am suggesting? Or does it still bother you? I know Dan is still confused. I am hoping to make some more progress. And then I'll walk you through it. Ya know If everybody wants this to go away that is fine. I'm not trying to be a bother.

#78 Thanks for all the comments. Especially the godot pointer. James, i think i've found the link you are referring to:

https://github.com/okamstudio/godot/wiki/gdscript

I've carefully read above, and while i understand the pain points that he refers to, i completely disagree with his analysis of the problem and his conclusion (GDScript).

1. "Godot embeds scripts in nodes, most languages are not designed with this in mind"

I cannot really say anything about above because it isn't clear to me what he means, or what the context of "most languages are not designed with this in mind" is.

2. "Godot uses several built-in data types for 2D and 3D math, script languages do not provide this, and binding them is inefficient."

You can bind most C++ math libraries in LUA (we *are* doing it), and using LuaJIT's FFI, JITs the calls making them be near native speed.

3. "Godot uses threads heavily for lifting and initializing data from the net or disk, script interpreters for common languages are not friendly to this."

Like i've wrote before, each thread should have its own VM.

4. "Godot already has a memory management model for resources, most script languages provide their own, which resulted in duplicate effort and bugs"

This is a very real problem. It needs a lot of experience in designing the C++ <-> LUA API to avoid the nasty pitfalls of it. This is why we only created a rudimentary prototype of the bindings. Most of our time went into designing an efficient API.

5. Binding code is always messy and results in several failure points, unexpected bugs and general unmaintainability.

I completely disagree. Binding code, by nature, is in a very frequently passed execution path. Due to that, most of the bugs will be stumbled upon (and solved) quite early in the development phase.

"The language and interpreter for it ended up being smaller than the binding code itself for Lua and Squirrel, and equally as functional"

I consider speed to also have functional characteristics, from an architectural point of view (it increases the % of C++ code that can move to the LUA side). It is extremely hard for any custom VM solution to reach the speed of expertly coded VMs like LuaJIT's one. Persons like Mike Pall of LuaJIT are one in a 1e6.

"With time, having a built-in language has proven to be a huge advantage"

And this is where my real problem with his whole argumentation is. He compares his experiments with bindings to other languages where *he kept them at arm's length* with integrating GDScript at the heart of his engine.

Another way to see his last point, is that it is extremely *valuable* to have whichever support language one chooses, not as an abstract generic binding but as a real implementation language on the same standing as C++ is.

About Andrew's "I think at this point it's clear it's not going to replace torquescript"

I agree replacing TS right away is a huge amount of effort. TS will have to remain there until TS code gets migrated to LUA.

" and this thread is more about the best way to create an abstraction to support multiple scripting languages in the future".

This thread is about LUA bindings. As such, i'm disseminating our thoughts and plans on the subject. To summarize them, we would like:

1. A language with a fast tracing JIT VM. The "tracing" VM property is ideal for dataflow programming, where the same computation kernel gets specialized to the types that it is fed with. Method JITs (like Java's) and C++ *cannot* do that (without runtime compiling). Two VMs that offer this right now are LuaJIT and PyPy.

2. A language with a small VM so as for each thread to run its own VM. This leaves only LuaJIT's VM as a contender.

3. A language that supports coroutines, to be able to compose statefull computation kernels. LUA fully supports them. The godot author had the good taste to add "some support" of them in GDScript.

4. A language with extremely fast calls *to* C++ code. See LuaJIT's FFI.

5. One pass builds. This means no SWIG or SWIG like binding methods.

#79 Okay guys I think I can show you something that I started, of course it has a long way to go. I really am trying to take into consideration transitioning us with out upsetting things.

I have a branch in https://github.com/antonyjones67/Torque3D/tree/lua_experimental_2

In this branch you will find under engine/source you will find something roughly that looks like this.

/ScriptBinding
scriptbinding.cpp
scriptbinding.h
-- the above files contain most of what used to be in the IMPLEMENT macros

/Lua
luaScriptBinding.h
luaScriptBinding.cpp
-- lua files pointing to scriptbinding funcs

/TorqueScript
tsScriptBinding.h
tsScriptBInding.cpp
-- torque Macros pointing to scriptbinding funcs

So for an example.

This is in scriptbinding.cpp
This does all the heavy lifting.

Point3F SceneObject_getEulerRotation( SceneObject* object )
{
Point3F euler = object->getTransform().toEuler();

// Convert to degrees.
euler.x = mRadToDeg( euler.x );
euler.y = mRadToDeg( euler.y );
euler.z = mRadToDeg( euler.z );

return euler;
}

Then in luaScriptBinding.cpp it looks like this.
This binds lua to the static function.

metatableFunctionsLuaObj.RegisterDirect( "getEulerRotation", &SceneObject_getEulerRotation );

Then in tsScriptBinding.cpp it looks like this at the moment.
I'm sure we can do better than this. But it currently looks like this.

DefineEngineMethod( SceneObject, getEulerRotation, Point3F, (),,
"Get Euler rotation of this object.\n"
"@return the orientation of the object in the form of rotations around the "
"X, Y and Z axes in degrees.\n" )
{
return SceneObject_getEulerRotation( object );
}

But you can see how the 2 different bindings are pointing at the static function in scriptbinding.cpp.

Anyway let me know what you think.

But be nice to me. I am very sensitive. :)

#80 James - thanks for the examples. I imagine the benefit of having VMs per thread would be multithreading logic like AI.

Antony - looking forward to being enlightened ;).

EDIT: for those who are still in the dark, this is my list of complaints with TS, and how I'd change the language if I could :P.