2D Blogs: OS X Rewrite
by Michael Perry · 11/27/2012 (11:42 am) · 55 comments
Torque 2D Blog: OS X, Cocoa, and Platform Work

Kickoff
Greetings everyone. It's another 2D blog day! When time allows, someone on the 2D team will post a blog or resource related to Torque 2D and/or iTorque 2D. The majority of the posts will come from myself, but don't be shocked if someone else takes over for me from time to time.I don't have much time to write a super detailed blog. Melv and I are in the middle of a very intense development cycle. We have taken on a task that is long overdue and requires at least two developers working simultaneously. Compared to previous blogs, this is a less dazzling feature, but it must be done.
Before proceeding, here's the usual disclosure:

FULL DISLOSURE
Do not take the content from these blogs as law. I'm never going to say "this will be in x.x version" or "this is ready to be used right now". There's going to be a lot of R&D discussion. I might even post a discussion I had with Melv that ended up being scrapped. It might be useful for you to understand how we communicate internally, or amusing to see the mad scientists we really are. I will not post timelines. I will not post release dates. I will not commit the team to something we cannot deliver on.

Goodbye Carbon
For the past three years, Torque 2D and iTorque 2D have been getting nailed by Apple updates. More and more code the engine relies on was being deprecated. This primarily affected the desktop editor and runtime, but the iOS side took an occasional hit as well. I'm talking about the use of Carbon. For those who have not been developing Apple products, Carbon is the old framework for building desktop applications. Apple changed gears and has been pushing its new framework called Cocoa.For a while, internal Torque devs like myself would have to swoop in and apply a bandaid patch that kept the engine at least compiling. When Mountain Lion dropped, all of Carbon hit the deprecation list. Suddenly, the engine stopped compiling. Once more bandaids were applied, some features would flat out not work. For example, trying to go fullscreen would result in a Fatal ISV.
In summary, the OS X version of Torque 2D and iTorque 2D is completely borked as long as it uses Carbon. We've put off the inevitable for too long. It was decided earlier in the year that we would block out a large amount of time to "make things right." We started work in the 3 Step Studio trunk two weeks ago. First and foremost, Carbon has been completely removed as a framework. How did that work out?

No Carbon, No Runtime
All stdlib functions still work. Apple would not dare drop support for these functions. I'm mainly talking about basic file I/O and memory functions. However, this is just a small percentage of the OS X layer for our 2D engine. Everything else was littered with Carbon calls. We were left with one option. Scorched earth!As I mentioned previously, Melv and I are attacking this task from both sides. We are both getting our hands dirty with Cocoa (Objective-C), but we are also specializing a bit. Melv has been cleaning up the messiest parts of Torque "Platform Abstraction" code, as well as implementing long overdue systems like unit tests. I have taken the lead on the Cocoa implementation and helping get systems back online.
We started by adding a new platform layer called "osx" and generated a completely fresh Xcode project. This new layer resides right next to win32Platform, iosPlatform, and macCarbonPlatform. Before we could even write new code, we created stubs for every class and function Torque expects. For the first time ever, we had Torque compiling on OS X without the use of Carbon.

Umm, What Next?
It was at this point we kind of just stared at the code for a bit. Torque's startup sequence and general platform layer was a god awful mess. There was no consistency between platforms and a lot of how things worked just did not make sense. While Melv was cleaning up that area, I got us moving by creating a basic Cocoa application that worked with the rest of the Torque source code. It was painful, but a major milestone.
It Lives!
Next, we created a prioritized list of systems we had to bring online one at a time. As an example, we knew we needed memory, file I/O, threading, and network code working before anything else. After much pain and labor, we had a very clean startup process on OS X. We have gotten as far as executing a main.cs script, loading modules, and running the engine in a loop. That doesn't sound flashy, but any Torque developer who has looked at our past Mac layer should be shedding happy tears right now.
Forging Ahead
We are getting into the thick of it now. Before the end of next week, we should have all video, input, and basic events running. From there, it's just marking items off a list. If at all possible, each system will have its own set of unit tests. These unit tests will work on all platforms, so we will know immediately if a change on Windows breaks OS X or iOS. Along with some discipline, this kind of effort should keep all of our platforms synchronized and running into the distant future.Again, UNIT TESTS. I'm sure our new QA lead will love to see this. I know it makes Melv and I very happy. Not only that, Melv's work can transfer over to Torque 3D as well, so we'll be talking to Dave Wyand about it soon.

Other Platforms
Developing a true Cocoa layer for Torque 2D has reaped many other benefits. Torque's general platform code has been cleaned up, with the work propagating to Windows and iOS. Additionally, our in-house knowledge of Apple frameworks will have tripled before we are finished with this task. Finally, I took a snapshot of an "empty platform layer" for future reference. Melv and I have a much greater understanding of what it takes to go to other platforms, which will be quite useful when we target new devices.
Conclusion
This is my least thought out blog yet. I kind of just started writing until I felt like I did not have anymore time. I gotta get back to the code and get our OpenGL setup. Please feel free to ask questions in the comments and I will do my best to go into further details, like "Yes we are using NSWindow and NSView. Yes we are using blah blah". Until next time!About the author
Programmer.
#2
To quote Terry Pratchett on tempting fate; that makes you " the sort to stand on a mountain in a thunderstorm wearing wet copper armour and shouting 'All gods are bastards'. "
11/27/2012 (12:51 pm)
Quote:
Apple would not dare drop support for these functions.
To quote Terry Pratchett on tempting fate; that makes you " the sort to stand on a mountain in a thunderstorm wearing wet copper armour and shouting 'All gods are bastards'. "
#4
11/27/2012 (1:04 pm)
Cool blog. I'm really looking forward to seeing the future of T2D! And, of course, seeing what can be shunted into T3D ;).
#5
11/27/2012 (6:15 pm)
Enjoying the blog posts and look forward to hearing more about the new T2D!
#6
Thank you for keeping us updated, knowing that you and Melv are at the helm convinces me that however long we wait, what we get in the end will be a kickass product.
11/27/2012 (10:10 pm)
Biting down on my tongue, trying to repress the urge to ask 'When can I have intimate time with the new T2D?"Thank you for keeping us updated, knowing that you and Melv are at the helm convinces me that however long we wait, what we get in the end will be a kickass product.
#7
11/27/2012 (11:19 pm)
Congrats!
#8
@Simon - I understand the feeling, but trust me when I say you don't want anything to do with it at this very moment. It's like having a gutted animal sitting in front of you, with your task being how to put it all back in while adding an extra appendage it did not have before.
@Matt - Thanks. You of all people know what Melv and I are attempting to do. You'll be quite happy to see the result.
Update: We just ran a trace of the win32 video initialization. Through years of abuse and quick fixes, it is just a mess. "It" being the video abstraction layer and how each OS has implemented it. Shortcuts, lack of docs, and lack of understanding on other platforms has resulted in a tough decision for us. We'd like to rewrite several things, not just create the new OS X video layer. For now, there just isn't the time to get everything perfect.
We'll get the OS X layer working nicely, then circle back to get the other ones cleaned up. After the OS X layer is taken care of, I'd like to take another swing at cleaning up the iOS layer. More than just supporting the iPhone 5 resolution, the video handling is kind of twisted. House of cards my friends, house of cards.
11/28/2012 (8:41 am)
@All - Thanks for the continued support of these blog postings. Continued responses always increase the chance of the next one happening. @Simon - I understand the feeling, but trust me when I say you don't want anything to do with it at this very moment. It's like having a gutted animal sitting in front of you, with your task being how to put it all back in while adding an extra appendage it did not have before.
@Matt - Thanks. You of all people know what Melv and I are attempting to do. You'll be quite happy to see the result.
Update: We just ran a trace of the win32 video initialization. Through years of abuse and quick fixes, it is just a mess. "It" being the video abstraction layer and how each OS has implemented it. Shortcuts, lack of docs, and lack of understanding on other platforms has resulted in a tough decision for us. We'd like to rewrite several things, not just create the new OS X video layer. For now, there just isn't the time to get everything perfect.
We'll get the OS X layer working nicely, then circle back to get the other ones cleaned up. After the OS X layer is taken care of, I'd like to take another swing at cleaning up the iOS layer. More than just supporting the iPhone 5 resolution, the video handling is kind of twisted. House of cards my friends, house of cards.
#10
11/28/2012 (1:31 pm)
Refactoring like a boss! :)
#11
I know they want to get as much life out of the bad code base that already exist but companies do it all the time.
One request though Pleas make it modular so it not such a pain to implement new features or addon's just saying.
I'm interested in seeing how the rewrite will work out.
11/28/2012 (1:41 pm)
If the code base is such a "hot mess" they burn it down and start from a new code base. No need on to keep band-aiding such a bad code base. That way GarageGames and have a very good and stable engine.I know they want to get as much life out of the bad code base that already exist but companies do it all the time.
One request though Pleas make it modular so it not such a pain to implement new features or addon's just saying.
I'm interested in seeing how the rewrite will work out.
#12
11/28/2012 (1:50 pm)
Quote:If the code base is such a "hot mess" they burn it down and start from a new code base. No need on to keep band-aiding such a bad code base.Pretty much the point of the entire blog. Instead of putting bandaids on our OS X platform, we are rewriting it. iOS and Windows will eventually get the same treatment. We have been doing this for about 11 months now with our major systems: Assets, modules, sprites, physics, animation, and so on.
#13
Do you have a time line for release of the new T2D? Is this possible to be something in my Christmas stocking this year?
11/28/2012 (2:13 pm)
@Michael,Do you have a time line for release of the new T2D? Is this possible to be something in my Christmas stocking this year?
#14
11/28/2012 (2:47 pm)
@ Michael Then you sir will have a paying customer, when the windows refactoring is finished.
#15
@Jimmy - I'll remember that =)
11/28/2012 (3:33 pm)
@Frank - I'm not able to say anything about dates. Well, except December 1st. That's my birthday, but that's the only thing I'm sure of. Dev wise, the platform work should be wrapped up in two to three weeks.@Jimmy - I'll remember that =)
#17
11/28/2012 (5:42 pm)
Because I don't have any other venue to ask, I'll use this. Is the IRC channel down for everyone else?
#18
11/28/2012 (5:43 pm)
@ Michael then let me make another feature request. Pretty Pleaaase add embedded mono scripting engine alone next to but not relying on Torque Script. I'm sorry tbh Torqe script needs to go it's very archaic :(
#19
@Jimmy: Mono has some weird licensing issues. It might not be straight-forward. TorqueScript would be fine if the interpreter wasn't pants-on-head stupid about numbers and strings.
11/28/2012 (5:47 pm)
@Mich: It's pining for the fjords.@Jimmy: Mono has some weird licensing issues. It might not be straight-forward. TorqueScript would be fine if the interpreter wasn't pants-on-head stupid about numbers and strings.
#20
I wasn't gonna say anything, but when the new T2D comes out I am porting ScriptT3D to work with it. I will most likely call it ScriptT2D though. Since T3D has gone MIT I can rip what I need out of there to make it work in T2D. So if you really want a different scripting language there will be Python available at some point. Also, since I am using SWIG, it should not be too hard to add other scripting languages that SWIG supports. I don't really have any interest in working with .NET languages. If you want that WinterLeaf has a very good solution. It might even work with Mono. You might ask them about that.
11/28/2012 (6:17 pm)
@Jimmy,I wasn't gonna say anything, but when the new T2D comes out I am porting ScriptT3D to work with it. I will most likely call it ScriptT2D though. Since T3D has gone MIT I can rip what I need out of there to make it work in T2D. So if you really want a different scripting language there will be Python available at some point. Also, since I am using SWIG, it should not be too hard to add other scripting languages that SWIG supports. I don't really have any interest in working with .NET languages. If you want that WinterLeaf has a very good solution. It might even work with Mono. You might ask them about that.


Torque 3D Owner Ronny Bangsund
Torque Cheerleaders
I'm looking forward to seeing this update loading in my IDEs, so I can poke at it till it breaks. Hurry up and take your time to finish it :P