2D Blogs: OS X Rewrite
by Michael Perry · 11/27/2012 (11:42 am) · 55 comments
Torque 2D Blog: OS X, Cocoa, and Platform Work

Kickoff
Greetings everyone. It's another 2D blog day! When time allows, someone on the 2D team will post a blog or resource related to Torque 2D and/or iTorque 2D. The majority of the posts will come from myself, but don't be shocked if someone else takes over for me from time to time.I don't have much time to write a super detailed blog. Melv and I are in the middle of a very intense development cycle. We have taken on a task that is long overdue and requires at least two developers working simultaneously. Compared to previous blogs, this is a less dazzling feature, but it must be done.
Before proceeding, here's the usual disclosure:

FULL DISLOSURE
Do not take the content from these blogs as law. I'm never going to say "this will be in x.x version" or "this is ready to be used right now". There's going to be a lot of R&D discussion. I might even post a discussion I had with Melv that ended up being scrapped. It might be useful for you to understand how we communicate internally, or amusing to see the mad scientists we really are. I will not post timelines. I will not post release dates. I will not commit the team to something we cannot deliver on.

Goodbye Carbon
For the past three years, Torque 2D and iTorque 2D have been getting nailed by Apple updates. More and more code the engine relies on was being deprecated. This primarily affected the desktop editor and runtime, but the iOS side took an occasional hit as well. I'm talking about the use of Carbon. For those who have not been developing Apple products, Carbon is the old framework for building desktop applications. Apple changed gears and has been pushing its new framework called Cocoa.For a while, internal Torque devs like myself would have to swoop in and apply a bandaid patch that kept the engine at least compiling. When Mountain Lion dropped, all of Carbon hit the deprecation list. Suddenly, the engine stopped compiling. Once more bandaids were applied, some features would flat out not work. For example, trying to go fullscreen would result in a Fatal ISV.
In summary, the OS X version of Torque 2D and iTorque 2D is completely borked as long as it uses Carbon. We've put off the inevitable for too long. It was decided earlier in the year that we would block out a large amount of time to "make things right." We started work in the 3 Step Studio trunk two weeks ago. First and foremost, Carbon has been completely removed as a framework. How did that work out?

No Carbon, No Runtime
All stdlib functions still work. Apple would not dare drop support for these functions. I'm mainly talking about basic file I/O and memory functions. However, this is just a small percentage of the OS X layer for our 2D engine. Everything else was littered with Carbon calls. We were left with one option. Scorched earth!As I mentioned previously, Melv and I are attacking this task from both sides. We are both getting our hands dirty with Cocoa (Objective-C), but we are also specializing a bit. Melv has been cleaning up the messiest parts of Torque "Platform Abstraction" code, as well as implementing long overdue systems like unit tests. I have taken the lead on the Cocoa implementation and helping get systems back online.
We started by adding a new platform layer called "osx" and generated a completely fresh Xcode project. This new layer resides right next to win32Platform, iosPlatform, and macCarbonPlatform. Before we could even write new code, we created stubs for every class and function Torque expects. For the first time ever, we had Torque compiling on OS X without the use of Carbon.

Umm, What Next?
It was at this point we kind of just stared at the code for a bit. Torque's startup sequence and general platform layer was a god awful mess. There was no consistency between platforms and a lot of how things worked just did not make sense. While Melv was cleaning up that area, I got us moving by creating a basic Cocoa application that worked with the rest of the Torque source code. It was painful, but a major milestone.
It Lives!
Next, we created a prioritized list of systems we had to bring online one at a time. As an example, we knew we needed memory, file I/O, threading, and network code working before anything else. After much pain and labor, we had a very clean startup process on OS X. We have gotten as far as executing a main.cs script, loading modules, and running the engine in a loop. That doesn't sound flashy, but any Torque developer who has looked at our past Mac layer should be shedding happy tears right now.
Forging Ahead
We are getting into the thick of it now. Before the end of next week, we should have all video, input, and basic events running. From there, it's just marking items off a list. If at all possible, each system will have its own set of unit tests. These unit tests will work on all platforms, so we will know immediately if a change on Windows breaks OS X or iOS. Along with some discipline, this kind of effort should keep all of our platforms synchronized and running into the distant future.Again, UNIT TESTS. I'm sure our new QA lead will love to see this. I know it makes Melv and I very happy. Not only that, Melv's work can transfer over to Torque 3D as well, so we'll be talking to Dave Wyand about it soon.

Other Platforms
Developing a true Cocoa layer for Torque 2D has reaped many other benefits. Torque's general platform code has been cleaned up, with the work propagating to Windows and iOS. Additionally, our in-house knowledge of Apple frameworks will have tripled before we are finished with this task. Finally, I took a snapshot of an "empty platform layer" for future reference. Melv and I have a much greater understanding of what it takes to go to other platforms, which will be quite useful when we target new devices.
Conclusion
This is my least thought out blog yet. I kind of just started writing until I felt like I did not have anymore time. I gotta get back to the code and get our OpenGL setup. Please feel free to ask questions in the comments and I will do my best to go into further details, like "Yes we are using NSWindow and NSView. Yes we are using blah blah". Until next time!About the author
Programmer.
#22
@Frank - As I just said. I'll be very interested in what you can come up with.
11/28/2012 (6:47 pm)
@Ronny - Ah, so it's simply stunned then? Lame. Guess I'll log in tomorrow.Quote:TorqueScript would be fine if the interpreter wasn't pants-on-head stupid about numbers and strings.Pretty accurate. I don't have much of a problem with TorqueScript itself as a language. It's missing some features I really like in C# and Objective C, but it gets the job done. My big problem is what's under the hood, as stated. Massive string table, constant string conversions, the most horrible switch statement you've ever seen, and so on. I'm very interested in Frank's work with Python and SWIG.
@Frank - As I just said. I'll be very interested in what you can come up with.
#23
Xamarin explicitly state that you must contact them to get a license for closed systems, and the text seems to imply there is money involved.
11/28/2012 (7:21 pm)
@Jimmy: UnityTech use ahead-of-time compilation (basically outputting assembler code) to use Mono safely in Unity3D. Getting that to work nicely for your product can be tricky, as you can't statically link the runtime. Only the compiler itself is dual-licensed.Xamarin explicitly state that you must contact them to get a license for closed systems, and the text seems to imply there is money involved.
#24
11/28/2012 (9:40 pm)
@Mich: Yeah it should have been up by now but apparently I submitted the ticket on the wrong server :( That has been resolved now and hopefully it will be back up soon.
#25
The only part of the DotNetTorque that you don't get source for is the generator. You get all the interface code that it generates and you can redistribute the generated code. They are just protecting the algorithm used to create the interface. The runtime is full source.
@Michael,
I can't wait to see the new T2D code. Since I am nearing completion of my third rewrite I really am preparing to use the code in more places. I also have some tricks up my sleeve for some extended features. The coolest one that I am really looking forward to working on involves: textures, OpenCL, and real-time data conversion. I also like the idea of possibly running both T3D and T2D as part of the same executable. Not sure how useful that will be, but it just sounds like fun.
11/28/2012 (10:44 pm)
@Jimmy,The only part of the DotNetTorque that you don't get source for is the generator. You get all the interface code that it generates and you can redistribute the generated code. They are just protecting the algorithm used to create the interface. The runtime is full source.
@Michael,
I can't wait to see the new T2D code. Since I am nearing completion of my third rewrite I really am preparing to use the code in more places. I also have some tricks up my sleeve for some extended features. The coolest one that I am really looking forward to working on involves: textures, OpenCL, and real-time data conversion. I also like the idea of possibly running both T3D and T2D as part of the same executable. Not sure how useful that will be, but it just sounds like fun.
#26
11/29/2012 (11:57 am)
@ FrankQuote: possibly running both T3D and T2D as part of the same executable.That could be very useful.
#27
11/29/2012 (11:59 am)
Quote:That could be very useful.It's been done before, but not publicly. An old version of T3D and T2D were merged at some point, but it did not last long.We captured some videos of it, however, running materials, lighting, and shadows.
#28
Why didn't it last long ? If you don't mind me asking.
11/29/2012 (6:35 pm)
Can you share ? would love to see that. Why didn't it last long ? If you don't mind me asking.
#29
11/29/2012 (7:58 pm)
Well, not necessarily integrated. They most likely would be separate windows, but part of the same Python app. So it may not be as useful as an integrated package.
#30
Link to video
It didn't last due to all the disruption of the company moving to Vegas, change in management, emphasis being pushed on T3D, and a new project being proposed over T2D R&D. If you go through my older 2D blogs, like 2009, you will see more information. Melv and I were both very sad that work did not move ahead and become the new product, but we are still happy with the direction we are taking the new T2D.
11/30/2012 (4:50 am)
@Jimmy - I refer to that work as "Old T2D R&D". It was the first real attempt to revamp Torque Game Builder into a next-gen engine, with shaders, real-time networking, a new editor, a true component system, and a different programming language. Link to video
It didn't last due to all the disruption of the company moving to Vegas, change in management, emphasis being pushed on T3D, and a new project being proposed over T2D R&D. If you go through my older 2D blogs, like 2009, you will see more information. Melv and I were both very sad that work did not move ahead and become the new product, but we are still happy with the direction we are taking the new T2D.
#31
It seems you were ahead of competition with this tech demo with jiggly physics and 3d lighting.
11/30/2012 (8:44 am)
@Micheal Just got done watching the video. All i got to say is whao, That would of been a great mobile engine to have with today's market. It seems you were ahead of competition with this tech demo with jiggly physics and 3d lighting.
#32
Not trying to be an ass but it tends to be the track record for Torque.
You guys are a great company but with little staff and it seems to keep things from being released in a timely manner.
All these improvements are great and very needed to resurrect the TGB engine. Will we have them before the next refactor has been put into place leaving us holding a shiny new toy that got broken before it was released?
Tech these days is moving faster than ever, what are you doing to keep up with it?
11/30/2012 (8:56 am)
I have a question about this fresh shiny new version you are working on. Will it be released before or after it outdated? Not trying to be an ass but it tends to be the track record for Torque.
You guys are a great company but with little staff and it seems to keep things from being released in a timely manner.
All these improvements are great and very needed to resurrect the TGB engine. Will we have them before the next refactor has been put into place leaving us holding a shiny new toy that got broken before it was released?
Tech these days is moving faster than ever, what are you doing to keep up with it?
#33
Forgive me, but I'm confused by your questions. They don't really make sense to me. These blogs are all about the R&D work going on behind the scenes that could be in a potential release. Whatever gets released publicly will be working and up to date with the latest platforms. Still, maybe I'm missing the point of your question.
11/30/2012 (9:15 am)
Quote:Will it be released before or after it outdated?
Quote:Will we have them before the next refactor has been put into place leaving us holding a shiny new toy that got broken before it was released?
Forgive me, but I'm confused by your questions. They don't really make sense to me. These blogs are all about the R&D work going on behind the scenes that could be in a potential release. Whatever gets released publicly will be working and up to date with the latest platforms. Still, maybe I'm missing the point of your question.
Quote:what are you doing to keep up with it?A core engine team working more than standard hours to keep up with the pace, another team using the tech daily to create games and templates. Are you looking for more details?
#34
What are you doing as far as the rebuild for the engine: bringing it up to current tech levels or making it a step ahead?
11/30/2012 (11:19 am)
hmmm, let me think of a different way to phrase this.What are you doing as far as the rebuild for the engine: bringing it up to current tech levels or making it a step ahead?
#35
11/30/2012 (11:21 am)
@Greg - Before I write out a lengthy reply, have you read through my previous 2D blogs? I've been covering the various system revamps and ways we are elevating our 2D technology
#36
Mich has posted numerous blogs about what they're doing. The big failing is only speed. GG work at snail's pace, and always have. But in their defence: Have you looked at the source code?
Every subsystem isn't so much a subsystem as intertwined blocks of code with sometimes duplicate, even triplicate, functionality, only a vague notion of why and how certain #defines should be used, dead code that never, ever gets called and a cross-platform graphics initialisation written by 20 blind people who didn't know how to communicate.
Other than that, it's all fine. I'm just hoping that this massive rewrite for the main two platforms makes the subsystems actual subsystems, resulting in an API which can actually be documented and then is easier to subdivide among more than two people.
Rewriting input shouldn't be taking you to the rendering code, through initialisation code, and back again. Adding a platform shouldn't require copying and pasting bits of the other platforms' code. And so on.
So what's it like now? If I wanted to add an input manager for Android, could I get away with adding just the C++ implementation for the code, and keep using the existing header, plus add any necessary Dalvik stuff to set up things? Or would I need to duplicate the existing headers and make small changes for a new platform? The former is clean code, the latter is the Torque way normally ;)
11/30/2012 (11:33 am)
Bloody L, Greg - why are you trolling blog posts now? Did you get sand in your clam?Mich has posted numerous blogs about what they're doing. The big failing is only speed. GG work at snail's pace, and always have. But in their defence: Have you looked at the source code?
Every subsystem isn't so much a subsystem as intertwined blocks of code with sometimes duplicate, even triplicate, functionality, only a vague notion of why and how certain #defines should be used, dead code that never, ever gets called and a cross-platform graphics initialisation written by 20 blind people who didn't know how to communicate.
Other than that, it's all fine. I'm just hoping that this massive rewrite for the main two platforms makes the subsystems actual subsystems, resulting in an API which can actually be documented and then is easier to subdivide among more than two people.
Rewriting input shouldn't be taking you to the rendering code, through initialisation code, and back again. Adding a platform shouldn't require copying and pasting bits of the other platforms' code. And so on.
So what's it like now? If I wanted to add an input manager for Android, could I get away with adding just the C++ implementation for the code, and keep using the existing header, plus add any necessary Dalvik stuff to set up things? Or would I need to duplicate the existing headers and make small changes for a new platform? The former is clean code, the latter is the Torque way normally ;)
#37
glad to see that gg now considering to rewrite a whole engine.
hope someday u guys will add 3d part also.
we will get an engine to do both(2d 3d)
" Carbon has been completely removed as a framework"
is not this line Contradictory with this one:
" This new layer resides right next to ................, and macCarbonPlatform." ?
11/30/2012 (10:02 pm)
happy Birthday Michael.glad to see that gg now considering to rewrite a whole engine.
hope someday u guys will add 3d part also.
we will get an engine to do both(2d 3d)
" Carbon has been completely removed as a framework"
is not this line Contradictory with this one:
" This new layer resides right next to ................, and macCarbonPlatform." ?
#38
11/30/2012 (11:05 pm)
No, it's not contradictory. Cocoa is an additional platform, and the existing Carbon can now be removed. Carbon only works on shitty old systems anyway.
#40
@ahsan - Ronny is correct. To provide more details, I was referring to a folder. We have the following:
platform: The base folder all platforms will use.
platformWin32: Our current Windows implementation.
platformiOS: Our iOS implementation
platformMacCarbon: Our old Mac implementation
platformOSX: Our new OS X implementation
The platformMacCarbon folder will simply be deleted once we are finished with platformOSX.
12/01/2012 (5:21 am)
Ha. Wasn't actually trying to get birthday wishes, just throw a distraction at a request for a date. Thanks guys =)@ahsan - Ronny is correct. To provide more details, I was referring to a folder. We have the following:
platform: The base folder all platforms will use.
platformWin32: Our current Windows implementation.
platformiOS: Our iOS implementation
platformMacCarbon: Our old Mac implementation
platformOSX: Our new OS X implementation
The platformMacCarbon folder will simply be deleted once we are finished with platformOSX.

Jimmy R Armes
@ Frank I've been following your blog and it sounds intresting but I never got into python, but I do know a few people who would swear it ;)
I'm intrested in DotTorqe but with out full source access that would defeat what I have planned for this product when it is released. I guess if worse come to worse I'll just pay someone to do. But really hopping not to go that route.