Torque 3D Community Edition Discussion Thread

#181 I've added the old terrain code back in inside a #ifdef - you guys are right, #defining engine modifications is probably a good idea. In this case, I made the new code default, and you can enable a define to switch back to the old code.

Mike: I definitely agree with goal-oriented commits, no committing disparate changes 'cause they were just what you happened to be working on. But IMO, branches are precisely for enabling changes to be made that are likely to break things for a while, allowing you to commit freely and in a way that maximises your insurance.

In any case - I think I agree with your model here, so I'll use that when working on the CE!

Alfio: I used to comment all my changes in your style as well, but once I started using version control I felt it was no longer necessary. Not trying to dissuade you, just justifying my choice! ;P In addition, if we all set up some sort of communication between SVN and trac, all the commits related to an issue can be seen on that ticket, which makes finding changes and such even easier.

#182 Hmm, I seem to be unable to commit. Alfio, can you check permissions for me?

#183 @Guy: you should be good to go now, give it a try.

#184 Yep, all good, thx :)

#185 @Guy
I'd suggest you add in your AI smooth, configurable, rotation resource.

#186 @Steve, I'm considering it. Any idea if the way I use an AIPlayerData datablock interferes with other stuff like UAISK?

#187 Nope, no idea. :P

#188 I threw out my implementation of the UAISK a while back -- no defamation intended to the kit -- but from what I recall that fix/resource shouldn't cause any serious problems with it. At worst would be the necessary change to the new Datablock name/class.

It's a valid fix and includes a useful enhancement... I'd say go for it.

#189 I have made some progress on the TS VM.

I found that the call to CodeBlock::exec by default treats the parameters provided to that particular call as a stack storage of sorts. Not sure the correct way to describe this. Anyway, I cannot just place the parameters to the exec function in a class instance and expect it to operate correctly.

When the VM calls exec inside itself this results in the parameters to exec being modified from that point on. However, the parameters to the original call to exec remain unchanged. When I put those values in the class as variables it messed up this paradigm. It needs this recursion to keep the parameters to the exec function straight. The main issue is that all the opcodes are now void op_function(void) calls on the CodeBlock class. So I am going to have to take the parameters to each call to exec and setup a stack with the parameters to exec to keep this straight.

One benefit to this will be the ability to actually look at the exec function parameter stack for recursive calls. This will actually make it easier to debug the VM operation. I think once I get this part sorted out I can start to determine if I am really going to get any performance gains from optimizing the VM. At the very least I now have a more in depth understanding of how the VM actually operates. This will probably come in handy if I decide to try and use another VM in the future and/or turn the TS VM into a JIT compiler. Either way I have found this to be a fascinating journey. Seeing the inner workings is really cool.

#190 Woohoo!

I think I am through the last of the functional bugs and am now into issues like stack overflow. This is actually a good sign. You see the chain calling of opcodes in a non-optimized compile will generate a stack overflow. Apparently I actually see this after about 7000 chain calls or so. It is hard to tell what this number actually is.

Anyway, now I am finally needing to tell the code to optimize to actually do the tail call optimization which will get rid of the stack overflow. So hopefully the VM is now working well enough that it is now a matter of getting the optimizations to work. Once I get the VM booting the engine all the way to a window and can run a mission I will know I have the VM in pretty good shape.

#191 Alright progress!

% - Initialized Core

--------- Initializing Directory: scripts ---------
Executing scripts/client/init.cs.
Executing scripts/server/init.cs.
sfxStartup...
'BenchMarkTesting_DEBUG.exe': Loaded 'C:\Windows\SysWOW64\wrap_oal.dll'
'BenchMarkTesting_DEBUG.exe': Loaded 'C:\Windows\SysWOW64\imaadp32.acm'
'BenchMarkTesting_DEBUG.exe': Loaded 'C:\Windows\SysWOW64\msg711.acm'
'BenchMarkTesting_DEBUG.exe': Loaded 'C:\Windows\SysWOW64\msgsm32.acm'
'BenchMarkTesting_DEBUG.exe': Loaded 'C:\Windows\SysWOW64\msadp32.acm'
'BenchMarkTesting_DEBUG.exe': Loaded 'C:\Windows\SysWOW64\l3codeca.acm'
'BenchMarkTesting_DEBUG.exe': Unloaded 'C:\Windows\SysWOW64\wrap_oal.dll'
..\..\..\..\..\Engine\source\sfx\openal\sfxALDevice.cpp(43) : Fatal - Failed to create OpenAL device and/or context!
Error, a DecalManager (92ad6d0) isn't properly out of the bins!
'BenchMarkTesting_DEBUG.exe': Unloaded 'C:\Windows\SysWOW64\D3DCompiler_43.dll'
HEAP[BenchMarkTesting_DEBUG.exe]: HEAP: Free Heap block 4e59e9c8 modified at 4e59e9f0 after it was freed
Windows has triggered a breakpoint in BenchMarkTesting_DEBUG.exe.

The good news I have successfully initialized the core scripts using the modified VM using tail call optimization. The bad news is I have no idea why OpenAL is freaking out. I am hoping someone might have seen an error like this before and can help point me in a the right direction. I know this is related to the modified VM as it does not do this with the normal VM. Anyway, it is coming along!

#192 I often have that problem with my sound card. Blame the driver locks up completely and requires the closing of all applications that use the sound card. Only after i closed all programs like Skype/Opera/etc etc, the driver starts again itself and i can start the client.

Maybe it's not your same case. A callstack might help us to help you,

#193 Thanks Alfio,
It was not the code changes, well it was sort of. It was the fact that I set the stack size for the app during the link to 10MB rather than the default 1MB. I thought it was interesting that it would cause a failure on a library interface for both the original VM implementation and mine.

Once I got rid of that OpenAL error by getting rid of the /STACK:10000000 command line option everything started working for both VM versions. I can now run the FPS demo using the new VM implementation. Woohoo! I have in no way done any extensive tests, but it did feel smoother while running the FPS. However, I do have optimizations turned on as well. One down side is I have to have optimizations while using the new VM version. If there is a way to turn on optimizations for just the VM that would be a good thing. Otherwise it will make it hard to debug. One way around this would be to have the old vm active when optimizations are off, and the new vm active when the optimizations are on. This would at least allow some decent debugging.

I get one error on exit related to smCurrentCodeBlock, but I can deal with that easy. smCurrentCodeBlock is actually put on a temporary stack when I exec other codeblocks from inside the VM. The VM can call other codeblocks other than the current so certain variables must be maintained for when it returns to the frame of execution. So I maintain the smCurrentCodeBlock as it is needed to be maintained for certain calls for when the code comes back to execute the original code block.

#194 Great work! I will confess to not really understanding what you're doing, but it sounds exciting. I actually am quite interested in all the low-level VM/compiler/language definition stuff, after having a bit of a muddle through the flex/bison files for TorqueScript. I'm just not very knowledgeable! Anyway, we're rooting for you, even if we have little to contribute ;P.

#195 @dB thanks.
Right now the modified VM starts up slightly slower than the default. It still has some strange bugs in it as well. I am going to rethink the variable handling for the VM. Stuff is getting out of whack because the variables were supposed to be local to the exec function not to the class itself. I also suspect some hidden exec calls in the VM op codes could be messing up stuff.

I really hope this pays off. Right now it is not any faster. On the other hand I will be the VM guru by the time I figure this out.

#196 I concur with Daniel, the VM work your doing Frank sounds extremely exciting. And I'm sure will really help reduce some bottlenecks within T3D.

#197 T3D.. so many unnecessary goto's :(

I Might remove the ones that aren't needed and don't affect speed..

#198 10 print "Basic is awesome!";
20 goto 10;

#199 TS VM status:

1. VM is working for the most part. There are some strange errors on a few functions that I need to track down to determine what is causing them. There are some strange crashes as well.
2. It is as fast as the normal implementation on startup time with or without optimizations turned on. Yes, it will run without optimization as long as you don't try and debug it with VC++. Otherwise you will get a stack overflow.
3. Everything is segmented out to individual functions as opcodes.
4. All functions share state with the class itself. There is a stack in the class that is static to all class instances. This is used to keep track of variables that must be maintained individually between calls to exec. So basically it simulated variables local to the exec call in the normal implementation. It is not pretty, but it does the job.

This is a starting point for further optimizations for the VM. It is not really production code, but it lays the foundation for further work. I need to put this in a branch of the CE. That way other people can dabble with this. I am starting to get burned out on it. So I am going to take a break from this for a while. I have other projects to attend to.

I am hoping it will allow the VM to be further extended and possibly used with a JIT library. I have also learned quite a bit how the actual VM works. I don't really have a clue as to how the TS is turned into opcodes, but I can see how the opcodes play out. I am also starting to see how you could hook a different VM into T3D. I am not all the way there on this, but I can see the light at the end of the tunnel. I think you could derive or add an exec into the CodeBlock class that calls a different VM. This other VM would have hooks to call functions in the engine much like the current VM does. Not sure it would be completely worth it though. The TS VM does its job well.

#200 How do I go about creating a branch that includes the changes to the VM only? I don't want to clutter the head with partially working code, but I do want to give people something to play around with.

Edit:
I am using Toroise SVN in Windows and I have downloaded the entire repository.