Mac glDrawElements Performance Issue

#1 That is very odd. There is a better way to draw, however you would need to move Torque over to using vertex buffer objects, or interleaved arrays. Both would be a lot of work.

#2 Why is it odd? Are you responding to the long thread that I linked to or what I typed here? Perhaps I'm wrong and there is something else that is causing the performance problem. Would it help you to see the profile log?

Are you saying that I shouldn't be seeing that kind of drop in frame rate? And are you saying that there is nothing that I can do? I'm stuck with this terrible performance?

Please tell me you can help.

#3 Well there are a few things that are odd. First we have a lot of games running on Torque on the Mac and I haven't heard of this issue yet. Secondly, glDrawElements shouldn't be a slow call with respect to other drawing methods. It shouldn't even be one of those SNAFU's about going to the Mac either because that is a pretty typical call. I would think this would have been a huge problem that would have been addressed by now. There are always faster methods but it uses draw elements on Interiors too, so I am wondering if this is a shape issue.

#4 Hi Joe,

Few notes. First, Pat is right in that this is weird since it hasn't come up before. But to dig in...

I read the previous thread, and looked into this a bit. Presently, I think the problem is as follows (with solution).

The Windows implementation of Torque uses calls to glFinish in order to support the d3d layer. On Windows, it's structured such that the glFinish calls aren't too problematic.

However, the calls to glFinish persisted in the Mac implementation, and this was an oversight. The next release of Torque will rectify this, and it's a really easy change to make. To fix it, just grep for glFinish and ifndef for OSX.

I'm at home so I don't have access to my Mac right now, but I'll do some tests Monday, I hope. If you don't mind, it'd be great if you want to test too. If you make the above suggested change, do you notice any performance improvement. Theory of course being that since Torque's scenegraph doesn't have a super-advanced shape material / state sorting or caching scheme, state changes and glDrawElements calls are somewhat frequent... throw glFinish into the mix and you get blocking, which can kill performance.

Let me know how it looks if you test this weekend, and I'll investigate some more. Thanks much for the report.

#5 Josh and Pat,

Thank you for the responses and follow up.

I commented out the glFinish call a while ago. I saw the resource from the guy at 21-6 or bravetree, don't remember his name. The only place I find glFinish is in the guiCanvas.cc in the renderFrame method. In XCode I did a find in project and that was the only hit that appeared. So the frame rates you see are with that call commented out.

In regards to testing, compiling, etc I don't mind helping. I am a professional programmer, although I have to admit that my C++ is very rusty at best, it's been about 10 years since I worked with it. I work mostly with Java these last number of years. With that being said, I really need to fix this problem so I won't let that stop me.

By the way I'm working with the lighting pack. If that will cause any issues please let me know and I'll work with the stock version then port the changes, if any. I did the tests reported in my thread against the stock version, and I get the same frame drops.

In regards to it being a model problem, to investigate the theory, I've done two things as documented in my posts. I tested the mission by replacing all objects with your flagpole from the orc demo and in another instance with a simple box. I added the objects after the mission loads with the following script code.

I call this function a bunch of times from another function.

function addTSShape(%fileName,%position)
{
	%obj = new TSStatic() 
	   {
	      shapeName = %fileName;
	   };
	   %obj.setTransform(%position);
	   transformObject(%obj);
	   MissionCleanup.add(%obj);
	   //MissionGroup.add(%obj);
	   return(%obj);
}

As a follow up to the error problem, what kind of errors can a simple box with 12 polys have? I can send you the box if you'd like. Or if you want, you can create a simple box with a texture and send it to me and I can report any changes.

To build my models I've used the DTS exporter with the Mac version of Blender.

I added profiler points to the code and directly wrapped the glDrawElements and that is where the bottle neck lies. At least that is how I, and I stress I, interpret the profiler. Perhaps I didn't run the profiler long enough. If you'd like, I can send you the profile dump.

Finally, to rule out any setup issues, I had a friend run this on his new iMac g5. With the newer NVidia 6600 he gets about the same frame rates. I don't remember exactly but if anything it was only off by 1 or 2 fps.

One final question, how come glEnableMetrics isn't enabled for the Mac?

Thank you for your time on this and helping me resolve this. It is greatly appreciated.

#6 The glEnableMetrics utility isn't available on the Mac because I guess whoever did the Mac platform layer didn't port it over. ;) I haven't looked at it myself, so don't know if there was any better reason for that. If there's not a good reason, we'll look at porting it over for the next update.

As a quick aside, have you used the glProfiler and Shark which are available on OS X? I highly recommend these profiling tools, they rock. And the glProfiler has some really cool updates planned for the release of Tiger. For instance, the GLProfiler can be used, in stock TGE 1.3, to tell ya that calls to glFinish take about 5.5% of time spent in GL when flying around the demo starter.fps mission, and this number goes up with the number of shapes being used. Note as well that that's just time spent in GL, since glFinish blocks the CPU it's total processing usage is greater than 5.5%. So, this seemed like a logical suggestion, but it seems you've already handled that issue. :)

So, I don't know exactly what's happening then, and can't test until Monday when I have my Mac back. It certainly would seem that we'd have heard of this issue by now if it affects all DTS rendering, and personally I've never observed it. For instance, in the RTS Starter Kit, we can have way more than 70 characters (all with bones and blended animations) in a mission, or on-screen, at once. Yet, you seem to be experiencing problems with simple boxes or the default flag.

So, I'm not sure what's up. If I could get you to zip up and send your mission files to me via email, I'll try to take a look at this Monday night. Please understand that I have some really high priority stuff on the plate right now, so I might not get a chance to dig into this in detail for a few days, but I'll do my best.

Thanks for the detailed reports here, always helps. :) Again, we should be able to get this figured out. I highly doubt it, for the basic reasons Pat states, but if this is a general performance issue on OS X that has simply gone undiscovered to this point (and seems inconsistent with our results in other projects), then we'll certainly try to resolve it right away. The next release of Torque already has lots of Mac performance enhancements planned anyway, so it'd be great to throw in a solution to this issue, if it's necessary.

Anyway, I'll try to reproduce these results as soon as I can, and if I can't, I'll report so. If I can, I'll try to figure out what's going on. So, update ya soon Joe! :)

#7 @Joe - you might try looking at the bools in tsmesh which controls how polys are loaded: as a list of triangles, as a bunch of triangle strips, or one big uber triangle strip. In turn, try setting TSMesh::smUseTriangles and TSMesh::smUseOneStrip to true (these control how polys are loaded out of the dts file, so you have to set them on start-up). In some implementations of ogl, calls to gl can be slow, so joining all the polys into one big strip or one list of triangles will reduce the number of calls to the bottleneck call to glDrawElements.

With that said, I agree with Pat and Josh and don't think this is the issue. We have upwards of 100 dts' in ThinkTanks and have had no problems running it on macs. So how much of the cpu time is spent in glDrawElements inside tsmesh?

BTW, if you do comment out glFinish and want to use d3d still, you will have to make a call to d3d->BeginScene someplace else. What I did is put it into glDrawBuffer (a bit of a hack, but so was putting it in glFinish).

#8 They key thing here, IMHO, is that the framekiller exists even when he replaces the models with 'official' ones. Others of us have used those same models with no problems.
Joe, is there any way you can compile your exact code on windows? And run those same areas/models?

#9 Yeah I'm confused about that too. We just do not see this performance hit ever, and we have Marble Blast on like every new Mac that is shipping and no reports of this issue.

#10 Thanks for the thoughts,

Clark, I had smUseTriangles set to true but not the useOneStrip flag. Setting that to true helped the frame rate a bit, but only by a few fps. I'll do some more testing before I say for sure by testing it against my real data later and see what happens. Thanks for the tip on that one.
I tried think tanks 1.1, very cool game by the way. I get nice performance playing the game. However, I have nothing to which to compare. I seems smooth but not overly fast. So I'm not sure what to make of this.

Erik,
I can't compile this on windows as I don't have access to a windows machine. Besides, this is stock torque and torque with the lighting pack. I plan to get my hands on a windows machine to see if the performance is the same by trying it against the executable from the torque demo.
In my last post you mentioned the model problem and I believe you are the one who looked at the models. I though gosh, perhaps it's a bug in the exporter or my model process. That's why I tried the stock models and I got the results mentioned.

Josh and Pat,
In regards to the shipping products on Mac, are any of the products shipping based upon 1.3 or 1.2 of torque? Could this be something introduced in 1.3? Finally, how stable is 1.4? Can I grab a somewhat stable copy of 1.4 and try it against that to see what type of results I get? How far along is it?

Josh,
I tried glProfiler and I got a mess of data back. I couldn't quite figure out what to do with the results. I got easier results from your profiler. Do you want me to send you the profile log? Would that help?

Last night I planned on putting together the least of what I need to replicate the issue but I didn't get time. I will do that today so you can download the mission and see for yourself. This will allow me to do more testing and see where the bottle neck really is.

So in short, I'll double check what glProfiler does and look at shark. In the mean time, I will create a really really simple version of my mission for download and perhaps josh can try it out. I'll download the rts starter kit and try out the demo to see what I get.

Thanks for all your replies.

Joe

#11 Did some more tests with my real data after implementing the changes that Clark mentions and I get about 26 fps. So that's about a 6 fps increase. Without the dts objects I get about 60 fps.

In the chart below the second to last column is the % app time glDrawElements is using as per Apples OpenGLProfiler and the last is per Torques. I put a Profiler_Start and End around the glDrawElements function is TSMesh. Everything else if very small. What's interesting is the discrepancy between the Torque profiler and Apple's.


Objects | With DTS | Without DTS | GLProfiler % | Torque %
-------------------------------------------------------------------------
Real Objects | 26 (fps) | 60 (fps) | 17.27 | 26.013
Simple Box | 44 (fps) | 60 (fps) | 11.84 | 10.045

I don't know if this helps but my player starts out by facing away from the objects. When I turn to face them it's really slow and jumpy. Then once it's viewed its more consistent.

#12 Joe, got your email, thanks. I'll take a look.

Quick note re: different numbers in the glProfiler vs Torque, that's expected as the glProfiler only tracks time spent in OpenGL. The numbers shouldn't bear any relation to each other, except when compared to other gl calls (the ratios between gl call usage times should be very similar). On sending the glProfiler log, thanks, but it wouldn't help as I have a beta / preview copy of the next edition, and I don't believe there's a log importer.

#13 Josh,

One thing you missed in regards to the difference is that I wrapped the glDrawElements in torque with the profile statements like so in tsmesh.render():

PROFILE_START(TSMeshGLDrawElement);
glDrawElements(drawType,draw.numElements,GL_UNSIGNED_SHORT,&indices[draw.start]);
PROFILE_END();

Would the % still be different?
Finally, I meant if you would like the log from Torque not glProfiler.

Joe

#14 Josh,

Where you able to take a look at my level? Did you notice a drop in the frame rate?

Joe

#15 I got a hold of a Windows laptop this weekend with a 1.4 ghz pentium 4 512 mb and a Nvidea go5200. No idea how much video ram.

Without the dts I get 61.8 fps with the dts I get 44 fps. This is with real data. I get a drop but not as severe as on the Mac. My Mac has a ATI 9000 pro so perhaps it's a difference with the video cards. I have no idea the difference between the two in terms of rendering power.

Not sure if this helps, but I'd really like to know why the Mac gets such a huge frame drop.

#16 Are you running in debug mode? A 1.4 pentium with 512mb of ram and any decent video card should not be getting 44fps. I have a Pentium M 1.6 with 512 mb of ram and a 9600 in a GG laptop, and it runs the new terrain of TSE at over 100 frames per second. There is something very wrong.

#17 It's with the executable that comes with the lighting pack and the torque demo. I don't have a development environment for Windows so I'm using the exe's from the demo's. I own the products but I do my work on the Mac and so I had to borrow the machine.

In regards to speed this is a 5200 go mobile graphics chip not a 9600. Not sure how they compare in power. By the way the resolution was 1024x768. It's a dell inspiron 8600

I think there is a problem when there are lots of small objects on the screen. Let me ask you a question. Does torque occlude objects that are hidden by interiors? I get these drops with or without my interior which hides many of the objects. I put profiling code in the Mac version and it doesn't seem to the material state set, it really seems to be with the glDrawElements function.

I sent Josh a sample environment but I have yet to hear back what results he found.

#18 I just tried the dojo and car shader engine demo's on this laptop. I get about 5.6 fps. The car said I should update my drivers so perhaps direct x 8 isn't updated. I didn't install the update since it's not my machine. Perhaps it has low video memory as well.

#19 Joe, I did try this and didn't notice an immediate big drop in frame rates (not much % difference than on the PC). I haven't had time though to really dig in, or to use the files you sent via email. I am crunching on Torque 2D right now, so it will take me a few more days, my apologies about that. I'll give this personal attention as soon as possible, but I just wanted to let you know about the delay.

#20 Josh,

Thanks for pinging me about this, I was beginning think you forgot.
My files are simple to try, just unzip them into the directory that contains an app, the contents are a full game. If you have a high end video card you may be masking the problem. If you can, please try this on a low end card, I have a 9000, this way drops will not be as hidden.

Thanks