Torque Vector vs std::vector

#1 I could be wrong, but to answer your question why reinventing the wheel seems common in this engine was because T3D was made off of TGE/TGEA which ran on multiple platforms besides Windows. I think it was to keep maximum computability between all operating systems to have its own universal standard library right inside of it, although I'm probably wrong on this...

#2 AFAIK the engine was written around 1998, before C++ really had a decent standard library. Which sounds crazy, but they were wild times. These days the standard library is actually standard, I understand. It's similar to why TorqueScript exists - back then there weren't off-the-shelf helpful scripting languages you could just integrate.

#3 Hmm, dunno. After looking at the implementation, I seriously doubt it was written before STL; it's clearly intended to work exactly the same (at least the Vector class).

I'd still like to know specifics of why it was done, but I guess my real question is: Can it all be ripped out and replaced with STL containers? Should it be ripped out and replaced?

(The reason I'm asking all this is that I'm building out some custom code that needs containers and I'd like to know if there's a reason to go with T3D's implementation or stick with STL.)

(Edit: Wait a minute; there were STL implementations as early as 1994. So obviously there was already an implementation available when Tribes was created. Hmm. Maybe one of the "old school" Torque devs will wander by and shine some light on things. *crosses fingers* )

#4 It was all written as it is original for maximum memory efficiency. I'm not sure if current Torque works the same way, but originally Torque overwrote the new operator to use it's own memory management scheme. The memory manager was designed to allocate new memory from the system if needed, but when freed, that memory would be held onto an reused for the next allocation.

Since Torque was replacing the new operator, you couldn't use standard STL containers. If you do a search in the forums you can probably still find old discussions about this. At one point there was a #DEFINE that you could use to not use the Torque memory management, but I think even with that there were still issues with STL.

I don't know how much of this still applies to MIT Torque, but I would imagine removing the existing container classes to use the STL ones would be a good chunk of work.

#5 The reason is pretty much exactly what Daniel Buckmaster stated that during Starsiege Tribes game engine development C++ wasn't standardized across compilers yet and the Dynamix developers used Visual C++ 5.0. Before Visual Studio 2005 the std headers were terrible and the compiler was rather awful in terms of C++ compliance. The standard template library wasn't truly standardized until 1998 along with the actual C++ language, hence why the earliest C++ standard is called C++98.

The STL copycat implementation that, unfortunately, Torque 3D still uses to this day is very similar to the templates in the std:: namespace, but have issues and are incompatible with the real STL. There's nothing more memory efficient about them over STL. And you can write your own stl allocator template class to manage memory for containers like std::vector if you want a memory manager. Not to mention that Torque hasn't had a working memory manager since at least TGE v1.2 and it was disabled because it was unstable as it was supposedly the source of many sudden and unpredictable crashes that plagued TGE back in the early versions.

#6 I thought TGE 1.5 still used the memory manager - that's why I never got Recast working in TGE, it didn't like the overridden new operator.

Either way, I think there's a principle involved here: scores of people spend a lot of time designing, implementing, and improving the STL. I highly doubt a replacement written fifteen years ago by a small game team will compare.

I have heard arguments that suggest the STL is designed to be general and user-friendly, not implement performance improvements that are needed for, for example, a game engine. I'm not convinced, though. I know someone's starting to use the STL in Torque2D. I've also used it in my own code. Pretty sure Walkabout is using it in places, and my AI code definitely is.

I'm very tempted to do some profiling next time I have an hour or two, though I doubt it's straight-up insertion and deletion performance that will dominate in an app as complicated as a game engine.

#7 I don't think std::vector <> had an overridable allocator back then, so they had no choice if they wanted it to get memory from the MemoryManager.

We actually replaced String and Vector with their std:: counterparts, and I can't honestly remember if more time was spent in those afterwards, or before. But since I can't remember, the difference must've been tiny.

#8 Stroustrup himself points out that if a standard library implementation is available it is probably in your best interest to go ahead and use it. He also points out that the STL was designed to be as useful as possible in as many use cases as possible and is therefore not guaranteed to be the best implementation for your specific needs. At this time it is probably safe enough to use the STL implementations, but back then the performance of the STL was a crap shoot.

Torque's versions make certain assumptions, frequently bypassing bounds checks and the like - probably for the sake of speed when running on late 486 and early Pentium class machines. On today's hardware this is probably not as necessary, but there could be performance issues if the wrong thing is done in an inner loop somewhere (we all know that occasionally Torque does some funny stuff). We could replace them and see what happens....

#9 Thanks everyone for the input! Exactly what I was looking for.

I wasn't planning on going through the engine and replacing the Torque containers with STL containers (at least not while I have better things to do), but for my own code, it sounds like the STL containers will work fine.

@Daniel: Please let us know what you find. My understanding is that the STL container classes, particularing the vector, are just about as optimized for speed as they can get, given all the functionality they provide. But it would be interesting to see some benchmarks. . .

#10 I should probably delete that gfx-stl branch. It strayed into replacing the linked lists that occur in resource management and didn't make it out. feature/resources did better and made it through to a stable 64-bit build in osx, That build is tagged 64bitOSX.

I'm currently tearing out the Stream classes and replacing it with stl::stream objects - t2d disabled networking, so I can stub a lot of that and concentrate on cleaning the replacement.

I think that stl::vector shouldn't cause performance issues, but I fear for my stl::list replacement, and will probably go to adding intrusive lists (http://www.codeofhonor.com/blog/avoiding-game-crashes-related-to-linked-lists) if I see particle systems eating up resources.