Game Development Community

dev|Pro Game Development Curriculum

Extending Python with T3D: Overview 04/23/2012

by Demolishun · 04/23/2012 (12:06 am) · 4 comments

I decided to post an overview of what is finished and what needs work. I wanted to flesh it out in my brain. I know a lot needs to be implemented, tweaked, and polished.

First I will start with a list of what is implemented:

  1. Loop control of T3D loop: init(), tick(), shutdown()
  2. Abstracted callback export system. Allows for callback exports from other languages besides Python. Derived class for Python callback exports.
  3. Abstracted object export system. Allows for object export from other languages besides Python. Derived class for Python object exports.
  4. Simobject import to Python. Allows simobjects to be manipulated directly inside of Python with attributes, arrays, etc.
  5. Sim type functions: FindObject(id), FindObject(str), GetVariable(str) for globals, SetVariable(str) for globals, Evaluate(), Execute(), ExecuteFailed().
  6. Implemented repr() so that it creates a string representation of a SimObject. This would be useful for storing state information of an object so it can be saved to a database or file.

Now I will cover the todo list:

  1. Complete! Finish implementing attribute access for Python objects exports to T3D. I have the get functions implemented, I need to implement the set functions.
  2. Complete! Console output to a Python object. T3D is under control of Python now and I need to have a way in Python to get console output. If Python output is wanted in T3D you can simply export a Python object to T3D and have Python redirect its output to that object. No special implementation needed except for a periodic function to check for messages and echo them to console. It would just be a scheduled TS function.
  3. Solved! Workout a consistent error reporting and decide if T3D code will cause exceptions in Python. This could, if properly implemented, make the interface much more robust by trapping errors in console interfaces to Python. I have decided to let the console output handle this. I can have the function that gets the console output decide to create an exception for warnings or error messages from T3D. The console output callback actually gets a number that determines if it received a message, warning or error (0,1,2 respectively). The Python callback can decide if an exception will be generated. This will allow the programmer to decide if exceptions will be generated for TS called functions. Also, functions called into the engine that are explicitly called from Python will generate exceptions. This is consistent with how Python works. While the previous is consistent with how T3D works as it will by default ignore warnings and errors and leaves it to the programmer to determine what to do with those.
  4. Complete! Look at the API layout and the way T3D looks to Python. Consider cleaning up any messy API kludges. I looked at using a singleton to clean up the Sim interface. However, I am finding there are strange errors when trying to use a singleton with SWIG. It would work partially, but fail on class methods. I decided ultimately to forget it. I want to get the interface working and into the wild. Ultimately if the programmer keeps creating multiple Sim interface objects I just don't care. Thinking about it now, I am guessing it is a static function issue. If I get complaints I will look into this issue again. Also, as part of this task I completely revamped the code to separate out the Python stuff into a separate SWIG file. I also wrapped a #IFDEF SWIGPYTHON around the %include. This will make it so one can easily exclude the use of Python functions if attaching a different language. The module is now called scriptT3D so that it is implied to support multiple languages. All Python specific functions or objects have been expunged from scriptT3D.h and scriptT3D.cpp. Only generic functions remain that are intended to be extended for each language and placed in their own SWIG file. The Python specific items have been placed in pyT3D.i.
  5. Complete! Look at creating helper Python decorators for exporting to TS. I decided I would avoid these as they introduce more function calls. I have simplified my exporting interface so it is not as difficult to use. One simplification was having all exported functions determine the number of parameters by failure. Python will cause an exception which I can catch in C++ and issue a warning that the number of parameters does not match. No need to figure out how many parameters are needed in the export call. Just allow none or unlimited.
  6. Start writing Python code to test out all the interfaces and try to break the code. This will be the beginning of a test suite to ensure code changes do not break functionality.
  7. Start using the library and connect T3D to libraries T3D has never been connected to before...

I am glad to say the bulk of the interfacing is beginning to look a lot simpler and the todo list smaller. When I first started this task I was determined not to take a long time. Well, that was a while ago. However, I have learned so much since my first step on this journey. I have come to respect T3D and Python a whole lot more. I am amazed at how I managed to navigate this far.

So, please comment on things you see that may be missing. I am getting that much closer to releasing this as a resource. I intend for you guys to have fun breaking it even more than I can. I want this code to be a living code base for everyone to draw upon if they wish. I think it will make it better than I could alone. Stay tuned...

About the author

I love programming, I love programming things that go click, whirr, boom. For organized T3D Links visit: http://demolishun.com/?page_id=67


#1
04/23/2012 (1:04 pm)
Has there been any test comparisons between Python and Torquescript?

Great blogs, BTW.
SC
#2
04/23/2012 (1:45 pm)
No tests yet. I really have no idea which one is faster. My main goal is more functionality. Now, when I start playing with the JIT for external Python processes I am very certain that will be much faster.

Also, it really depends up what you are doing. For instance if I import the lxml module into Python then I am certainly going to process xml files way faster than TS ever could. lxml is a wrapper for a very fast xml library written in C. Then again there really is no equivalent of the lxml library in T3D. So it is really not a fair comparison. Obviously you could connect the libraries that make up lxml to T3D on their own, but you have the effort of writing the custom code to make it work in T3D. Hence the reason for connecting Python to T3D. It becomes as simple as an import statement to do the same thing.

Now imagine being able to use 15K+ such mature and well supported Python interfaces to various C/C++ packages. With just an import statement you can now use these packages in T3D. So the development effort to use any of these 15K+ packages is already done.

Now, if you have never programmed in Python it is worth checking it out. I found myself being many times more productive in Python than in C/C++. In the past I have written 10s of thousands of lines of code in C and C++. I am not new to the language. Then I started using Python for a customer and found myself concentrating on the solution rather than the code. I could accomplish more in 1 month with Python that with in 2 to 3 in C++. Other people have found this same phenomenon:
kurtgrandis.com/blog/2010/02/24/python-django-vs-c-asp-net-productivity-showdown... There are a lot more articles like this out there. I only looked for this information AFTER I experienced the same phenomenon.

I know, a long answer, but it really covers a lot of the reasons for using Python outside of execution speed.

I am glad you like the blogs. When the code is made into a resource it will be much more meaningful and help document the interface.
#3
04/24/2012 (5:15 am)
Thanks Frank.

I'll take a look at Python.

Keep up the great work.
SC
#4
04/29/2012 (9:04 pm)
Oh, I found something that may actually improve the speed a bit in the future:
mathema.tician.de/software/pyopencl

I have not compiled and tested this. I believe this version works with both Nvidia and ATI cards. However, I am not keen to use it yet as I would not want to depend on this library for speed.

Although, the idea of accessing 96 cuda processors is somewhat appealing! What is significant is the Python abstraction of the interface. Easy access!