Modular Scripting Integration Thread

#41 But it would be nice if we did use the STL - the current container implementations are a little wonky and inconsistent.

As far as multiple script languages coexisting - which language do you fire the callback in? All of them? Just one? Specify in the datablock? That wouldn't get confusing.... I think I'm more on board with the modular swapping of languages than their coexistence.

As for tools - well, that's just a shit-ton of work....

#42

Quote:As far as multiple script languages coexisting - which language do you fire the callback in?

That was one issue bugging me. However, since the tools are still TS they should stay TS unless we develop a TS to <something else> compiler. There will always be a critical mass with whatever language the tools are written in. That is my main motivation in keeping TS and adding 1 other language. So yeah, it is kind of messy any way you slice it.

Then again going to an XML representation for data objects would be a good move. Kind of like T2D did. That removes one reliance on a scripting language right there.

It is possible that the tools could stay TS while the game itself does not use TS.

#43 Hey @Vince and @Paul,
Did you stick with script defining datablocks or have you used something like XML?

#44 Things:

TAML would be a cross-language way of defining 'datablocks', though IIRC TAML replaces datablocks. Lukas seems to have got TAML working in T3D, which is pretty sweet.

Not everyone cares about editors. Case in point, me. And users of frameworks like Pygame, though there are of course external editors if you're using something like that.

Coexistence does sound very tricky to do in a first-class way. On the other hand, let's see how much the existing code relies on scripts. Looking up console variables, for example, won't be an issue. Callbacks may be, though it'd be pretty easy to just make callbacks happen in all languages in which they're defined. Seems like the least surprising solution.

I've been discussing STL with Paul a little. There are tradeoffs, and even UE4, for example, uses its own container classes. One good reason is that it's possible to modify them to fit your needs if they're right there. The inconsistency is a shame, but I don't think the STL is a shining beacon of API design either :P. Historically, the STL has suffered from bad cross-platform support and performance. Though neither of these things are true any more, apparently working with custom allocators is a pain. Not that we're doing anything that needs that at the moment :/.

#45 The main advantage to rolling your own containers is that you can skip error checking if it is determined that this is a bottleneck. The STL has always been intended as a starting point - not the optimal solution for all possible applications, so one should expect to tweak it if necessary.

Thinking about this, I can't decide if it'd be more work to normalize Torque's container behaviors to cause less "astonishment" or to convert to STL containers. I would really like to see Torque's containers (in whatever form) behave more like a newcomer would expect. When you see

vector<int> myVector;

you expect that myVector will behave a certain way - and in Torque it frequently does not.

Firing a callback in every language it is defined in is what I would expect - it would be harder to do anything else. But this raises some interesting bug-hunting scenarios....

#46 Also, another point to consider. Some programming languages are designed to be extended rather than embedded. Python is case in point here. It can be embedded, but its true power comes from being extended. I am not sure there is a happy median between embedding or extending. Python basically turns the engine into a library (on Windows).

I am not familiar with .Net, but I assume it wants to be extended as well. Where as something like Lua I am guessing wants to be embedded?

#47 Yup. It's a good point. Frank, when I was working with ScriptT3D recently I included your swig bindings as a module. I'm wondering if we could make a swig interface that doesn't rely on the TS console. That would deal with extension languages, right? Then we could work on stuff for embedded languages in the other direction. I dunno. I need to learn more about how all this stuff works :/.

#48 Yeah, I don't know. I based upon the console so I wasn't creating tons of class interfaces. I was thinking about letting SWIG try to sort out some of the classes itself. If it can in a meaningful way then that would make adding other languages a bit simpler, at least the ones SWIG supports.

Also consider the purpose of the embedded language. If you want it for user scripting Python is NOT what you want unless you severely limit its scope. Where as something like Lua is better because there is practically zero introspection so the user cannot "escape" back into the rest of the system. V8 is even better as it has execution contexts.

So I would classify 2 reasons for a scripting language: 1 for generic game logic programming, and 2 for user side programming. I am actually fond of user side programming simply because as a user I could make interface changes for different kinds of input devices, or customize a hud.

I think we have been mainly talking about game logic side scripting, but the other may be important if someone wants user side stuff. TS is a bit powerful for user side stuff because there is only one context.

#49 So by user-side programming you mean stuff you do in some games by editing ini and cfg files? Modding? Strictly client-side modding I guess. Interesting, I never really considered that aspect of it.

#50 Yeah, that and possibly being allowed to script AI for their own bots and stuff. I have seen one MMO that allowed limited scripting for your controls and displays.

#51 Huh, interesting. Something to consider. So I guess that was what you were talking about, using V8 in Python? Python being a closed system and the Javascript stuff being user-accessible?

#52 Yeah, because V8 is designed to be confined contexts. In Python I can feed it what it knows via a class definition. That is all that context will know about. I was thinking that would be perfect for AI as all the thinking would be on the client side and it gets compiled and runs fast. Imagine hundreds of user defined AI bots doing "something" in a game. put 50 people in a sim with their own bots. Total chaos. It would be beautiful.

#53 Yeah, that's actually a really cool idea.

#54 Actually, Lua is designed to be both embeddable and extensible. It has a package feature that lets you include and register C dll/so libraries via Lua script so long as those libraries expose functionality to Lua. Of course, you have to find/create these libraries....

#55 Whatever the final implementation may be, if people want this, the first step has to be to abstract the scripting language and move torquescript into its own little section where it could be replaced/improved the same way a GFX device or Physics library can be handled currently.

Who's up for the challenge? I can't jump in at the moment but I can certainly help along the way. We need someone to do an evaluation of the reach of torquescript and identify the most challenging areas to replace/abstract.

#56 Well I just read through the console code comments again, including the engine api comments. It talks about removing the console or the engine api console interface altogether in the future. I am not sure how that would work, or what needs to change in the engine api or the console to make that possible. It looks to me the intent was to abstract the interface via engine api so things could be swapped out. So perhaps part of this work is done.

I know James U is working with it in T2D. I will definitely be tracking how that works out to try and understand better how engine api is supposed to work. What I don't really understand is what the engine api is trying to solve. If it is providing a way to interface to all the functions in the engine to expose to a DLL/so then fine. I can actually do that via SWIG with much less pain. It actually has the ability to parse whole header files and create interfaces.

So I guess the question comes down to: What actually needs to be exposed?

1. SimObjects and those derived from SimObject (Anything lower level than that? Like ConsoleObjects or EngineObjects?)
2. Base functionality like engine init, tick, shutdown for languages that control the main loop (Python, C#?, etc)
3. Maybe just assume all globally called functions from the existing console system are exposed. That might be accessible via macros at this point.
4. Anything else?

Not knowing how to "rip out" the console as it is intertwined into the base classes for SimObjects I am starting to back pedal a bit from a modular version of TS. I just wish we had access to whoever wrote the comments about removing the console system. Even a block diagram would be VERY useful for what was being attempted there. I know my conceptual knowledge of how parts of the engine is supposed to operate it little to none. I have stared at code for hours in the engine not being able to understand what it was supposed to be doing, or why it was supposed to be doing that.

#57 I'd begin by taking a look at console/console.cpp. In there you'll find a large chunk of the console/scripting functions used throughout the engine. My approach would be to keep those functions in-tact and then create a TorqueScriptConsole or something that inherits and overrides the virtual methods just like GFX and Physics.

This way when you call something like Con::execute() it would go to the active scripting language. This raises issue when you execute inline torquescript from inside the engine ( what if the active scripting is python? ) but those seem like bridges we can cross when we get there.

If you run into chunks of code you can't figure out just post about here. Someone will figure it out.

#58 changed thread title to reflect current topic.

I can provide limited knowledge on the internals of torquescript such as the AST, lexer grammar files to define how the scripting works, and how it runs through the op codes and executes a routine of code. However, beyond that my knowledge is very little.

#59 Ideally, all of those specifics of parsing torquescript should be irrelevant to the task at hand. They should be moved into a blackbox of torquescript and not worried about for now. We just want to get a layer of abstraction between the engine and torquescript so a new scripting language could just fill in the blanks and drop in as a replacement.

#60 @Demolishun

From what I gathered engineAPI was at some point supposed to be a replacement for the console stuff, but in the end it seems to have mostly been turned into a way of adding even more docs (most of which are useless or which state the obvious). It also allows for those fancy DefineEngineMethod macros which abstract away the need to parse arguments (at the expense of not being able to have variable length arguments). I also mentioned a while ago you can actually use the information from engineAPI to automatically create bindings, but after working with the information I found it to be actually quite lacking and it was clear the system would need a lot of fixes to work properly in this way (so I would not advise it).

IMO from experience I would say it adds a little too much complexity to the bindings system due to the excessive use of nested templates This is also the reason why I chose Torque2D for the lua integration: it was much simpler to work with.

From what I have seen SWIG is only useful if you intend to turn Torque3D into a library, and essentially requires a separate build step in order to generate the bindings. Sounds a bit like overkill to me, but thats just my thoughts on the matter. ;)