Modular Scripting Integration Thread

#21 @Michael,
My reason for replacing TS was to gain access to the thousands of integrated libraries like lxml, twisted web framework, opencl, etc. Integrating and using libraries takes little to no time provided there in one for the scripting language. I use lxml all the time for my commercial coding projects. I cannot think how I would solve of a lot of my data issues without it. Yes, I could integrate xml libraries in C++, but that takes time to learn the API and all of its quirks. Time to work around the issues of the library if any. With a good scripting language the interface to libraries is commoditized and I just need to learn the API relevant to the function I am performing. Most of the time I am not even exposed to the data structures.

I don't agree that a scripting language choice is not really important. It is. V8 for instance has JIT with continued optimization for the most used code paths. Another issue I have found with TS is that there is not a team of VM experts working on it. Most "popular" VMs have this in one form or another. So having the ability to use CSharp, V8, LUA, Python, etc is a good thing.

Greek2me,
I wholeheartedly agree. I went through and converted every opcode to a function and removed any tail calls. I was hoping to take advantage of compiler optimizations. I got it working and was going to use it as a foundation for more advanced improvements for TS. I had read somewhere that getting rid of the case structure most VMs use and moving to function calls will help with later optimizations. I just lost steam on the whole thing as I was not getting anything else done. That is when I figured I would let the VM experts make VMs and I would staple them onto the engine.

#22 Some work that's been done on the Torque 2D side to add Lua alongside TorqueScript, in case anyone is interested in taking this further:

github.com/jamesu/Torque2D/tree/scripting_refactor

James can probably correct me if I'm wrong, but I believe the end-game for this was to lay down the foundation for a plug in framework for scripting languages. Not sure how easy this would be to adapt to T3D, but the Console/TS code is pretty much the same between engines as far as I recall.

#23

Quote:
I don't agree that a scripting language choice is not really important. It is.

Of course it is.

I prefer your idea of a modular console system - so that people can write plug-in language packages and there will be a variety of options.

My main point is that for most purposes you might as well leave TorqueScript as it is (except for actual bug fixes). There is a really bad hands/work ratio here....

As for Lua - if anyone really feels like going down that path I'd recommend looking at Luna (slightly modified to tweak garbage collection). It's pretty simple to essentially place the bindings side-by-side with the Torquescript bindings. I guess I'll give it a whack to provide an example, but it's really pretty easy and you don't have to do it all at once. The one thing I'm uncertain about is how it likes C++ inheritance - haven't walked that far down the path yet....

#24

Quote:My main point is that for most purposes you might as well leave TorqueScript as it is (except for actual bug fixes). There is a really bad hands/work ratio here....

I totally agree with you on that.

Yes, I would make TS another module as well. No reason to throw it out. It is cross platform in that it will compile on any platform T3D will compile on. A lot of VMs cannot make that claim.

#25 @Mike

The initial rationale for the lua integration was to see how well it could be done using the existing console api. My aim also was to make it generic enough so perhaps other languages could be integrated as well. Although realistically in a project you'd have to choose a single language since interoperability cannot be guaranteed between each language, unless you make it so that the bindings are as loose and generic as possible, which would likely be horrible.

Also personally while I think TS is ok as a basic scripting language with objects, its implementation leaves a lot to be desired. There's also of course also the long term issue of maintenance. I do not think at this time there is anyone in the T3D community who is particularly interested in doing any work on TorqueScript.

Although realistically unless there is a clear driving force or project governed by the community which favours the "use a new scripting language" approach, I do not think anything can be improved in this area.

#26 @James,
I think I understand your thoughts on the mix and matching between scripting languages. In Python there is not concept of a "package". So actually providing callbacks that can work with the existing package system forces me to dip into TS to assign the package to the callback function from the Python code. There are a lot of things like this when using my interface. Of course this is partly because mine is stapled onto the console itself.

So how is your Lua interface shaping up? I would be very interested in studying how your interface works. It might give me a better handle on how to improve my stuff. I looked at the code, but I was kind of unsure where the "secret sauce" is.

#27 Going off of what JamesU said, my interests in the VM is to only add things to it that I've wanted in marble blast for so long. I am not a VM programmer, I barely know what the heck I'm doing, I get bye. All I did one day was sat down and traced the VM for hours until I got somewhere.

#28

Quote:Tim Newell and I both tried removing the calls for postFX and moving them into C++ but tests showed no difference at all in performance.

Didn't you guys find a big hit from the per-frame setShaderConstants call? Or was I mistaken? WLE says they've had performance gains from using C# instead of TS - I'm inclined to believe them but you might also consider that gains could come from other changes they've made to the engine.

Quote:I guess my question is what kind of tangible benefits would we expect to see from a change in scripting language? Performance? Ease of use? "Power"?

For me, it's not about performance. Ease of use, definitely, and just pure programming convenience. No need to recompile, quick to type, quick to prototype with, easy to read and share, etc. Popularity is definitely a factor. TorqueScript does tick all those boxes (except popularity)... but some of the ticks are kind of next to the box not in it, and others might be a cross, or just a smudge of somebody's lunch. I guess the question is... why have a scripting language at all? Because it's easier, friendlier for non-programmers, modders, etc. Otherwise we'd just write everything in C++ and enable edit-and-continue :P. We also shouldn't underestimate Frank's point about the vast number of libraries that were available when he started using Python. We can't anticipate what epople will want to do with their games: connect to databases, web services, do special kinds of maths or AI, etc. An existing language, especially a general-purpose language like Python, has libraries for stuff like that that make it easier to implement. C++... kind of has some of this stuff. I guess. But then you have to deal with C++ interfaces, and write your own console interface if you want to access it in TS. It's work you don't want to be doing when you're making a game.

Quote:but speed performance code should be implemented directly in the engine and exposed to the Virtual Machine via EngineMethod/EngineFunction anyways

Yep, sure. Though I still think performance isn't the problem with TorqueScript. And again I'd love to use ECMAscript. It's the bees knees. Lua seems acceptable. Mruby sounds like a fascinating idea, though I'm not sure how practical it would be.

Quote:Language is language

paulgraham.com/avg.html. And oh gosh Richard, that Unreal forum thread has some gold in it:

Quote:No modern language compares favorably to C++, in any respect really. C++ is the most powerful, the most flexible modern language

Hhh.

Honestly the first place to start might be trying to remove the console entirely. I'm tempted, since with C++11 I don't think scripting in C++ would be the worst thing ever. At least I could have my typechecking and lambdas (barely). But that will let us scope the problem, and see where the boundaries of a modular scripting interface might lie. We could also take the opportunity to add a DLL interface for scripting, so we could add C++ code without recompiling.

You know what? Let's rewrite the entire engine in Rust. [/troll........ or am I?]

#29

Quote:However, an obscure language can really be a turn-off for new modders.

That's why I think Epic Games made a smart move to implement Blueprinting and kind of avoid non-coders make a choice based on their knowledge of script languages they do have.

Imo, if you'd want to attract more newcomers, a visual solution like this would be the improvement you're looking for. However, T3D newcomers are pretty scarce, and is T3D not a commercial product anymore.

If you'd stick to TS or any other; a good editor, integrated with T3D, that will guide you in a better way comparing the solutions available today; will surely be welcomed by newcomers and non-coders. The old tooling haven't been updated or upgraded, while everything else has for a decade long...

#30 @Dan,
Can T3D be compiled with C++11? I don't have a clue what that means exactly, but I do know the Polyvox is moving to use more and more C++11. That is one reason I am not pursuing Panda3D. It has issues compiling with C++X or 10 or whatever it is. Thus possibly making Polyvox incompatible.

#31 The current standard is C++98, and yes, a C++11 compiler can compile C++98 code.We're currently not using any C++11 features in T3D thanks to VS2008, though even more recent versions have patchy support of C++11 features. Though I'm pretty sure VS2013 has most of them.

Quote:The old tooling haven't been updated or upgraded, while everything else has for a decade long...

Really good point.

Also, because nobody's mentioned it yet, getting rid of TorqueScript means saying goodbye to the current editors... I'm pretty sure WLE ported them to C#, but I don't imagine that was a lot of fun.

#32 What's preventing us from dropping VS2008?

#33 People still use it. We've had complaints. But we're looking at doing so soon. At least putting a deprecation warning out this version.

#34 So after that (and after changing everything to C++11) the course of action would be to start working on the old engineAPI system, but using the new C++11 features? It's like a half-finished modular scripting system.

#35 Yes, WLE did rewrite the tools.... and yes it IS VERY PAINFUL. We are still debugging some of the more in depth tools.

Trying to stay focused on the scripting side, if I were to start again from scratch (Which you can never rule out.... ask Paul). I would come up with an intermediate library that would allow easy object memory mapping.

Pretty much all languages allow pinvokes, the question always come down to how you interact with the c++ objects.

Ideally, you would want to have native support for things like arrays and class properties.

in a case of SomeObjectInstance.myproperty in a perfect world it would be typed. So if that property was a bool in C++ it would also be a bool in the scripting language. The other question is how do you present that object to the scripting language so that the scripting language knows it is a object and that the object has properties and methods assigned to it.

For the most part, all languages have compatible types. So you can map string to string, int to int, etc. The only time this becomes difficult is when you are dealing with arrays. We worked around it with operator overrides in Omni, but I do not know if this is possible in other languages.

Then you have to deal with function overriding. There are cases in the engine where as objects are inherited, functions are override. The scripting language needs to be aware that.

Take for example you have 2 classes:

Class A has function Foo.
Class B is derived from Class A and override function Foo to some new functionality.

If you have a Class B object, how do you call the Class A Foo? In TS they use Parent::[b] in C# you use [b]base, but do you have that type of functionality in all languages you plan to implement.

Don't even get me started on Packages. Packages are amazing, but do not translate to any other language. There are still some spots where I have to use some torquescript injection to keep the same functionality because C# has no way to do it. Some of the C# code was converted it to class inheritance, some of it we had to come up w/ a custom solution.

Talking about inheritance, TorqueScript allows any type of casting. Take for example our two class example above and add Class C. Class C does not have the function Foo and doesn`t derive from class A or B. But you could call it by calling ClassB:Foo(Class C).

There are many other little features of Torquescript that most people haven't encountered because as far as I know no one has ever tried rewriting the Example and Tools to a different language except us. Aswin and I have encountered many strange connotations in the TorqueScript that has caused us to scratch our heads.


Our solution to the above problems was to write a code parser which reads all the C++ code. After it reads all of the C++ code it then builds a memory map of all enumerations, objects, methods, etc. Once it completes indexing and building this memory map it then generates the C# code.

A C# class called a Model is generated for each C++ simobject based on the object in the c++. We also generate enumerations, etc. An instance of the model is assigned to the C++ object instance at creation and thus all callbacks, properties, methods, etc. are transparent between C# and C++.

This is no small task and if it isn`t planned out from the beginning you will find yourself rewriting several times.

Also, should I mention ease of debugging?

#36 @Demolishun
I know that WLE has one of our coders looking at C++11 and Microsoft STL. There is no "due date" on it but I know that he's already started and said that he underestimated the scope of the job. Ill be checking in with him this Friday.

#37 @Paul,
Is it a naive idea that any translation to C++11 should be done by attrition? I mean as soon as you want to compile does it demand the code be different than it is?

#38 @Demolishun
To be quite honest, im not positive that he's tackled the C++11 yet. I know that he was looking at the STL, but I can specifically ask about the C++11 when I ping him next.

#39 I suspect there isn't going to be much trouble, actually. Especially since we don't use the STL.

#40 @dB,
Thanks for the link. I was just too lazy to look it up I guess. ;)

I am tempted to start looking into a modular console system. I think the first thing I would do is extract and reinsert the TS console as a module. That has the least pain of determining if it is working or not. If that could be made to work then attaching any scripting system would be pretty simple I would guess. Of course all the tools would need to be reworked for those languages "if" TS was not able to coexist.

The biggest issue will be finding all the locations in the C++ that calls TS code and make this abstracted somehow. Maybe a function call to the script module that would be translated to whatever function is needed to be called in script? I just have my doubts about multiple scripting languages coexisting.