A TorqueScript standard library
by Daniel Buckmaster · in Torque 3D Professional · 04/05/2013 (3:31 pm) · 33 replies
I've been thinking for a while that I don't like TorqueScript very much. The language itself is an issue (being, for all practical purposes, untyped), but I've realised that it's also due to the clunky support for stuff that should be simple. For example, when I need to treat a string as a vector, I have to go through the getWord/setWord functions to read or modify it.
I think we can do better. I'm not sure how many of you are familiar with Bret Victor; he's an inspiration of mine right now, mostly for his article Learnable Programming, which argues for better language design. It's not just applicable to lnguages that are 'kid languages' or supposed to be used as an intro to programming. All languages can benefit from being more accessible. Imagine the first time you read someone else's code - an opaque language can make it a nightmare to understand, even if you're an expert. Code isn't for compilers any more, it's for humans.
So anyway, with me off my soapbox, I'd like to talk about improving TS's libraries to make up for its lack of typefulness. Particularly, I've been thinking about maths. We often have vectors in TS, and as I said before, need to use getWord and stuff to access them. I've been dreaming of another way:
The idea is to abstract all vector operations into the vec namespace, so that we can stop thinking of vectors as strings. It shouldn't matter how the vectors are internally represented; as a programmer I shouldn't have to think about that.
This becomes more powerful when you start applying this to stuff like lists or other data structures. Imagine being able to write
The repetition of %myList = list.something(%myList) is frustrating. Oddly enough, it's very functional, which could be good, but TS doesn't have the syntax to support that without getting parentheses everywhere. For example, see Haskell's $ operator which turns 'print (show (head (namesOf myFriends)))' into 'print $ show $ head $ namesOf myFriends'.
Okay, so, congrats for reading this far. Sorry it's a bit incoherent, but I'm basically wondering who thinks this is a good idea. Better libraries for TS with more expressive naming.
I think we can do better. I'm not sure how many of you are familiar with Bret Victor; he's an inspiration of mine right now, mostly for his article Learnable Programming, which argues for better language design. It's not just applicable to lnguages that are 'kid languages' or supposed to be used as an intro to programming. All languages can benefit from being more accessible. Imagine the first time you read someone else's code - an opaque language can make it a nightmare to understand, even if you're an expert. Code isn't for compilers any more, it's for humans.
So anyway, with me off my soapbox, I'd like to talk about improving TS's libraries to make up for its lack of typefulness. Particularly, I've been thinking about maths. We often have vectors in TS, and as I said before, need to use getWord and stuff to access them. I've been dreaming of another way:
// Create a new vector (10, 20, 30) %myVec = vec.new(10, 20, 30); // Print the x component %x = vec.x(%myVec); echo(%x); // Update the z component %myVec = vec.z(%myVec, -5); // Cross two vectors %c = vec.cross(%myVec, %yourVec);Of course, this is mostly syntax sugar for functions that already exist. I'd like to point out the overloading of the .xyz functions to be both getters and setters.
The idea is to abstract all vector operations into the vec namespace, so that we can stop thinking of vectors as strings. It shouldn't matter how the vectors are internally represented; as a programmer I shouldn't have to think about that.
This becomes more powerful when you start applying this to stuff like lists or other data structures. Imagine being able to write
%myList = list.new(); %myList = list.push(%myList, "Bill"); %myList = list.push(%myList, "Janine"); echo(list.length(%myList));By abstracting the fact that the list is a string, we can do things like represent the list internally as "2 Bill Janine", which gives instant access to the list's length. As long as all the list.x functions expect this format, the user never has to see it. And by using functions that are named what they do (i.e., list.length instead of getWordCount), your intent while writing code becomes easier to understand.
The repetition of %myList = list.something(%myList) is frustrating. Oddly enough, it's very functional, which could be good, but TS doesn't have the syntax to support that without getting parentheses everywhere. For example, see Haskell's $ operator which turns 'print (show (head (namesOf myFriends)))' into 'print $ show $ head $ namesOf myFriends'.
Okay, so, congrats for reading this far. Sorry it's a bit incoherent, but I'm basically wondering who thinks this is a good idea. Better libraries for TS with more expressive naming.
About the author
Studying mechatronic engineering and computer science at the University of Sydney. Game development is probably my most time-consuming hobby!
#2
Unfortunately, I think we're stuck with TS for a little while longer, so it makes sense to see if we can make it any easier to use. PythonT3D notwithstanding, of course - I'm really tempted to look into it.
04/05/2013 (6:26 pm)
Oh, I agree - Python and Javascript are far better languages than TS. One of the things that's fantastic about Python is the sheer volume of libraries that are included by default - something I'm really missing when trying to learn Haskell!Unfortunately, I think we're stuck with TS for a little while longer, so it makes sense to see if we can make it any easier to use. PythonT3D notwithstanding, of course - I'm really tempted to look into it.
#3
With that said, the idea of a lib that standardizes common tasks is interesting. There are a number of non-obvious aspects about TS that understandably confuse new users (even those that are experienced programmers) and a standard lib could go a long way toward bridging the gap.
04/05/2013 (7:39 pm)
There's a little-known TS feature that allows vectors (or really any string with 2 or 3 individual fields) to be accessed via .x, .y or .z. Such as:%myVec = "1 2 3"; echo(%myVec.y); // will print "2" %myVec.z = 500; // %myVec is now set to "1 2 500"
With that said, the idea of a lib that standardizes common tasks is interesting. There are a number of non-obvious aspects about TS that understandably confuse new users (even those that are experienced programmers) and a standard lib could go a long way toward bridging the gap.
#4
04/05/2013 (9:42 pm)
Oh, that's cool - the amount of undocumented features in Torque is staggering! But I think a structure like this would come into its own when it went beyond what's already available - for example, a matrix library, lists as I said, even associative maps. All these things are already available in some way or another, but they're not nice to use or easy to look at.
#5
Have you considered DNT?
www.winterleafentertainment.com/Products/DotNetTorque.aspx
This allows you to use any .NET language instead of TorqueScript
04/05/2013 (9:42 pm)
@DanielHave you considered DNT?
www.winterleafentertainment.com/Products/DotNetTorque.aspx
This allows you to use any .NET language instead of TorqueScript
#6
Also, I just don't like C# :P. I guess I've got nothing in particular against it... it's just too much like Java for my taste.
04/05/2013 (9:49 pm)
Paul: I guess my reluctance to start using both PyT3D and DNT is because they're nonstandard. Having decided TS is bad, I think I can only either wait for the entire community to jump on something like PyT3D or using JS/V8, or just make it easier to use TS.Also, I just don't like C# :P. I guess I've got nothing in particular against it... it's just too much like Java for my taste.
#7
04/05/2013 (9:54 pm)
I am still of the opinion that whitespace should never ever EVER be meaningful (except as separation) to a language. EVER!
#8
as far as it being nonstandard.... thats true. But it works and works well.
and there is a MIT version of DNT out now. All its missing is the DNTC if you ever want to just take a look at it and wander about.
github.com/WinterleafEnterainment/DNT-Torque3D
04/05/2013 (10:00 pm)
well, i guess you could always use Visual Basic with DNT... :Das far as it being nonstandard.... thats true. But it works and works well.
and there is a MIT version of DNT out now. All its missing is the DNTC if you ever want to just take a look at it and wander about.
github.com/WinterleafEnterainment/DNT-Torque3D
#9
(To get back to DNT) doesn't matter that it isn't standard since it works with the standard tools ;) Theres no significant change to your scripts or your engine required to setup DNT.
04/05/2013 (11:50 pm)
Well would be easy enough to write a std library imo. Seems like a 3-day job to lay the foundation.e(To get back to DNT) doesn't matter that it isn't standard since it works with the standard tools ;) Theres no significant change to your scripts or your engine required to setup DNT.
#10
One thing that TS has going for it is that it is automatically cross platform. With other scripting languages you have to make sure the language and the libraries can be be ported readily or use a 3rd party solution for those platforms. That is something I am looking at closely.
As to why I mention Python in particular is because their focus is exactly as you describe. Make the language better rather than find workarounds for the failings. That is why Python 3.x breaks some compatibility with code from Python 2.x. They want to make the next generation of the language better than the first. It is also far beyond being a "learning language" IMO.
04/06/2013 (3:07 am)
I don't think TS is going away anytime soon. It works, it does what it is supposed to. However, for backend, AI, and other things that might require better abstraction, object oriented, or just plain needs resources that are not available then a different scripting language is fine. As for speed DNT had everything else beat. Unless someone injects ECMA into the engine natively.One thing that TS has going for it is that it is automatically cross platform. With other scripting languages you have to make sure the language and the libraries can be be ported readily or use a 3rd party solution for those platforms. That is something I am looking at closely.
Quote:I am still of the opinion that whitespace should never ever EVER be meaningful (except as separation) to a language. EVER!Haha! Although: space and tab have meaning is TS and every other language. It is just Python treats whitespace with a different type of meaning than other languages. I also don't have to hunt down braces. For the most part my it is cleaner than C++.
As to why I mention Python in particular is because their focus is exactly as you describe. Make the language better rather than find workarounds for the failings. That is why Python 3.x breaks some compatibility with code from Python 2.x. They want to make the next generation of the language better than the first. It is also far beyond being a "learning language" IMO.
#11
Lukas: yeah, it doesn't seem like a ton of work. I'm kind of vaguely intending to do it sometime this semester, just wanted to generate a bit of discussion first. People here always have a different angle on things than I do! Good point that DNT is still compatible with TS, that's a big plus.
Frank: TS is only as cross-platform as T3D itself... which is almost inevitably going to be less than any given popular language people might want to use. Of course, there are weird licensing issues for closed platforms (cough Apple cough) to consider.
Richard: you'd love CoffeeScript ;).
Frank again: I should make sure that when I say languages should be easy to learn, that doesn't make them a "learning language"... or maybe I should put it this way: all languages should be "learning languages". It makes it easier to get people on board, easier to pick up someone else's code, easier to pick up your own code six months on! And being a "learning language" shouldn't mean you have to sacrifice power or features. It just means good design.
EDIT: Another one for Frank: have you heard of Nimrod? It's an interesting language that looks like Python but is statically typed and compiles to C.
04/06/2013 (5:14 am)
Paul: I love the idea of using VB with T3D :P. I'll certainly keep the MIT version of DNT in mind.Lukas: yeah, it doesn't seem like a ton of work. I'm kind of vaguely intending to do it sometime this semester, just wanted to generate a bit of discussion first. People here always have a different angle on things than I do! Good point that DNT is still compatible with TS, that's a big plus.
Frank: TS is only as cross-platform as T3D itself... which is almost inevitably going to be less than any given popular language people might want to use. Of course, there are weird licensing issues for closed platforms (cough Apple cough) to consider.
Richard: you'd love CoffeeScript ;).
Frank again: I should make sure that when I say languages should be easy to learn, that doesn't make them a "learning language"... or maybe I should put it this way: all languages should be "learning languages". It makes it easier to get people on board, easier to pick up someone else's code, easier to pick up your own code six months on! And being a "learning language" shouldn't mean you have to sacrifice power or features. It just means good design.
EDIT: Another one for Frank: have you heard of Nimrod? It's an interesting language that looks like Python but is statically typed and compiles to C.
#12
While it definitely could use some improved core functionality(and to be more forward about it, like with the xyz stuff), fact is, there's really actually very, very little I CAN'T do with the language, which is why I like it.
That said, I've actually been accumulating a huge standard library of functions for TS. Currently they're all in TS, though I'd been contemplating moving them to the engine and just have a generic TSSTD::blahblah() type calls to access them.
Currently my 'library' of these functions is about 150. Ranges everything from string manipulation, matrix-quat-euler conversion and utilization, grabbing terrain heights, etc.
It didn't really occur to me that people'd really want these as a big library(though in retrospect, given that I made one should've tipped me off), so I'll look at cleaning it up, engine-porting the sensible ones, and then dropping it as a resource out there for consumption.
Lots of stuff, like AI functionality, if coded smartly, is plenty and perfectly doable in TS, along with some seriously heavy codestuff if you do it right(i'm running a completely script-driven state machine weapon off of the item class for stuff like mounted turrets, and it runs perfectly).
So complaints about TS's performance or utilization probably comes from the fact that it's so loose most people aren't sure what to do with it. Maybe the TSSTD library can help with that.
04/06/2013 (6:46 am)
After getting a feel for it, I think TS is actually one of my favorite languages.While it definitely could use some improved core functionality(and to be more forward about it, like with the xyz stuff), fact is, there's really actually very, very little I CAN'T do with the language, which is why I like it.
That said, I've actually been accumulating a huge standard library of functions for TS. Currently they're all in TS, though I'd been contemplating moving them to the engine and just have a generic TSSTD::blahblah() type calls to access them.
Currently my 'library' of these functions is about 150. Ranges everything from string manipulation, matrix-quat-euler conversion and utilization, grabbing terrain heights, etc.
It didn't really occur to me that people'd really want these as a big library(though in retrospect, given that I made one should've tipped me off), so I'll look at cleaning it up, engine-porting the sensible ones, and then dropping it as a resource out there for consumption.
Lots of stuff, like AI functionality, if coded smartly, is plenty and perfectly doable in TS, along with some seriously heavy codestuff if you do it right(i'm running a completely script-driven state machine weapon off of the item class for stuff like mounted turrets, and it runs perfectly).
So complaints about TS's performance or utilization probably comes from the fact that it's so loose most people aren't sure what to do with it. Maybe the TSSTD library can help with that.
#13
Some people get it in their head that Python is for learning CS. It does not stop at learning, so it is a learning language in that sense, but it does not stop you either. I mean you can do all sorts of really whacked out stuff with it like metaprogramming. That is what I was talking about there. I use some of that to do Memoization
As for cross platform of T3D, I was referring to that if T3D can compile on a platform you have access to TS. I agree, most main stream languages have ways to run on most platforms now.
I had not seen nimrod before. It looks fairly young.
04/06/2013 (6:50 am)
@dB,Some people get it in their head that Python is for learning CS. It does not stop at learning, so it is a learning language in that sense, but it does not stop you either. I mean you can do all sorts of really whacked out stuff with it like metaprogramming. That is what I was talking about there. I use some of that to do Memoization
As for cross platform of T3D, I was referring to that if T3D can compile on a platform you have access to TS. I agree, most main stream languages have ways to run on most platforms now.
I had not seen nimrod before. It looks fairly young.
#14
re PyT3D: I've always seen Python as the goto language for "script-kiddies", hehe, but my real reasons for hating it is besides the point. I made a point (in IRC) awhile back that if PyT3D were to be cleaned up, had a fully working example project, was documented, and presented in a pull request that it would be evaluated openly, honestly, and fairly. Still waiting...
@Jeff Raab: indeed, that would be useful. I've got a whole bunch of "library" functions of my own that I've accumulated over the years and have been looking for a good way of introducing them as standard (yet optional) functionality available to users.
04/06/2013 (10:12 am)
Quote:There is (and has been for several months now) a branch on Github that introduces some basic typing. The steering committee appealed to the community (in a blog and forum threads) to help test and report issues for it before we considered merging it into the development branch. Only one person in this whole time has stepped forward to do so.
The language itself is an issue (being, for all practical purposes, untyped),
Quote:This is always open to improvement... but someone has to do it ;)
the clunky support for stuff that should be simple.
re PyT3D: I've always seen Python as the goto language for "script-kiddies", hehe, but my real reasons for hating it is besides the point. I made a point (in IRC) awhile back that if PyT3D were to be cleaned up, had a fully working example project, was documented, and presented in a pull request that it would be evaluated openly, honestly, and fairly. Still waiting...
@Jeff Raab: indeed, that would be useful. I've got a whole bunch of "library" functions of my own that I've accumulated over the years and have been looking for a good way of introducing them as standard (yet optional) functionality available to users.
#15
04/06/2013 (12:04 pm)
I think that a standard libary should be "standard". It would be good to have ready made functions that would make development much easier.
#16
Frank: I love the idea of using metaprogramming to simulate memoisation... that's pretty insane ;P.
Michael: I actually shouldn't have complained about the types. My idea of typing is strong/static, which I don't expect to see in TS! I keep forgetting to check out the consulefuncrefactor branch, but I thought it was focused on performance (i.e. not performing string conversions where it's not necessary). But if this paves the way for getting more custom types into TS (for example, being able to use Point3F functionality in script), I'm all for it.
04/06/2013 (3:49 pm)
Jeff: I'd be really interested to take a look at that as well. For now, I'll start to work on my own take.Frank: I love the idea of using metaprogramming to simulate memoisation... that's pretty insane ;P.
Michael: I actually shouldn't have complained about the types. My idea of typing is strong/static, which I don't expect to see in TS! I keep forgetting to check out the consulefuncrefactor branch, but I thought it was focused on performance (i.e. not performing string conversions where it's not necessary). But if this paves the way for getting more custom types into TS (for example, being able to use Point3F functionality in script), I'm all for it.
#17
I have cleaned up the code a lot...but the reason there is no pull request is because it forces the created DLL to have a dependency upon the PythonX.X.dll file. This to me is a pain and would require hands on tweaking by the end user to make work anyway. Some of that tweaking is adjusting the compilation to be a PYD rather than a DLL. There was also no major improvements in the engine other than adding Python. Unless someone has a different opinion I felt due to the above reasons that ScriptT3D.
Perhaps I need to spend some more time to see if there is a way to prevent DLL loading by the resulting T3D DLL. I thought about doing a DLL that use the T3D DLL but I am concerned how cross platform that would be.
@dB,
Python is strongly typed IF you use the calls to check type. Otherwise it is pretty forgiving on types. It will try and convert what it can. Now, I know engine commands in T3D can check for type. That is a really nice feature.
The memoization is actually really cool. When a console function is first called from the Python side it determines if that function exists. Then it sets an attribute that "hard codes" the call against the Python to console interface object. The next call does not need to check. Also, another C++ object is created that dives into the call on the engine side and finds the point where the function finds the namespace entry for the function and remembers that. The next call it uses that namespace entry to call the function. The idea there was to reduce the number of string lookups from Python to the engine. I believe this makes the call outside the Python interface slightly faster than the call would be from the console itself.
04/06/2013 (5:49 pm)
@Michael,I have cleaned up the code a lot...but the reason there is no pull request is because it forces the created DLL to have a dependency upon the PythonX.X.dll file. This to me is a pain and would require hands on tweaking by the end user to make work anyway. Some of that tweaking is adjusting the compilation to be a PYD rather than a DLL. There was also no major improvements in the engine other than adding Python. Unless someone has a different opinion I felt due to the above reasons that ScriptT3D.
Perhaps I need to spend some more time to see if there is a way to prevent DLL loading by the resulting T3D DLL. I thought about doing a DLL that use the T3D DLL but I am concerned how cross platform that would be.
@dB,
Python is strongly typed IF you use the calls to check type. Otherwise it is pretty forgiving on types. It will try and convert what it can. Now, I know engine commands in T3D can check for type. That is a really nice feature.
The memoization is actually really cool. When a console function is first called from the Python side it determines if that function exists. Then it sets an attribute that "hard codes" the call against the Python to console interface object. The next call does not need to check. Also, another C++ object is created that dives into the call on the engine side and finds the point where the function finds the namespace entry for the function and remembers that. The next call it uses that namespace entry to call the function. The idea there was to reduce the number of string lookups from Python to the engine. I believe this makes the call outside the Python interface slightly faster than the call would be from the console itself.
#18
04/07/2013 (1:20 am)
@Frank - you don't have to chase down braces, you have to chase down invisible delimiters. Much easier.... ;p
#19
04/07/2013 (3:00 am)
An adequate programming environment renders both your arguments invalid ;).
#20
The tab thing is the main sticking point for me with Python. The rest of the language seems pretty sweet - I just don't want to have to worry if I used the right number of tabs or if my tabs are in the right place. Braces I can see - so I can see when I miss one pretty easily.
Then again, if I had my 'druthers Lua would be my choice. Takes a little trickery to get engine-side object inheritance working correctly but the language itself is pretty sweet.
04/07/2013 (11:00 am)
I concede to that.The tab thing is the main sticking point for me with Python. The rest of the language seems pretty sweet - I just don't want to have to worry if I used the right number of tabs or if my tabs are in the right place. Braces I can see - so I can see when I miss one pretty easily.
Then again, if I had my 'druthers Lua would be my choice. Takes a little trickery to get engine-side object inheritance working correctly but the language itself is pretty sweet.
Torque Owner Demolishun
DemolishunConsulting Rocks!
Your example (and more) above in Python:
# turn into a string appropriate for multi argument strings used in TS myString = ', '.join(myList) print myString # intermediate string not needed # another way myString = "{0} {1}".format("Bill", "Janine") # or create the list this way myList = ["Bill", "Janine"] # serialize the list mySString = str(myList) print mySString # which produces ["Bill","Janine"] # unserialize into object import ast # create the data structure of original myList object myList = ast.literal_eval(mySString) # does not attempt to execute code print ', '.join(myList) # how about an object class NObj(object): x = 0 y = 0 z = 0 nobj = NObj() nobj.x = 10.2 nobj.y = 20.5 nobj.z = -13.9 # serialize to string object # this is kind of tedious to type myString = "{0} {1} {2}".format(nobj.x, nobj.y, nobj.z) class NObj2(object): x = 0 y = 0 z = 0 # override str def __str__(self): # "" (quotes) create a string object that you can call format on to prepare the data. Very similar in function to the printf return "{0} {1} {2}".format(self.x, self.y, self.z) nobj = NObj2() nobj.x = 10.2 nobj.y = 20.5 nobj.z = -13.9 # use new method # automatically invokes __str__ print nobj # many, many, many other ways to store and convert data to many, many, many different formats # you can also do anonymous functions, but they are not as cool as ECMA script anonymous functionsI think it is a good idea. However, the time spent improving the scripting language could be focused on the game rather than the language by using a better scripting language.