T3D RTT and Audio Capture

Note:
This resource is a combination of a couple of things. To show how to do simple RTT and to show how to capture audio data (in Windows Vista and up). You can pick and choose what you want out of this resource. Also, this should be considered a WIP or a living resource. I wanted to get this into people's hands so they can play/use pieces or all of it.

Background:
I was looking for a way to have the engine respond to external audio sources. I wanted to build something similar to www.vsxu.com/. The idea was to make it so I can put some music on and the source can be from anything. That way T3D does not need to know how to play the media and it can be an internet radio station, youtube, audio player, midi player, movie player, microphone, anything.

I found out that in Windows Vista and above there is an interface that allows you to capture audio from the sound output device (before it actually amplifies this sound, although the volume from each program can amplify the sound). So I took the approach of using loopback capture. So yes the loopback capture portion is Windows Vista and above specific. However, there are ways to do this in Windows XP, but the API is different. It is a similar story with Linux and I am guessing Mac OSX. So if someone needs those platforms please research and share your findings with this resource.

Then I needed to be able to isolate frequencies. So my first inclination was to look for FFT code. I have no idea how to write FFT equations. I know it has to do with wavelet theory and is related to s domain math. I took some in college, but has since been flushed. I found a great little library as a consequence: KISSFFT. It is not the fastest FFT library available, but it is simple, small, and has a BSD license. I also see that the author is active on Stack Overflow and helps people use the lib. There is another equation I will use down the road that is actually better at isolating specific frequencies. I think it is a Gabor Wavelet based equation. So that takes care of detecting the frequencies bands using FFT.

I did some testing early on and found I can use script to manipulate many objects. However, I wanted to create data that C++ objects could use so the reaction to the music could be faster. So that prompted me to explore doing RTT (render to texture) and making objects that have generic access to the raw and converted data. This all had to be done fast without slowing down the main loop of the game. So all capture and conversion are done in a separate thread. That way the main app only has to deal with reacting to the data. I also constructed the processing objects to be able to get their data from other sources besides the loopback capture. This way if someone wanted to create a different data capturing object they can. Something like a capture object that gets the audio stream for a specific character(s). That way you can tie the moving of their lips or mouth to do lip syncing to a speech sound byte. That would be just one example of how this can be expanded.

How it works:

1. A loopback capture object runs in another thread and grabs audio data continuously. The starting and stopping of this object is controlled through script commands.
2. Audio processing objects: LoopBackObject and FFTObject are created as SimObjects in script. These are added to the processing queue by using a script command.
3. When I processing object is in the processing queue they are accessed every 100mS (can be tweaked by a #define statement). When they are accessed their buffers are locked using thread locking mechanisms. The data from the loop back capture will be copied to the objects and the object will perform any data processing on the newly acquired data. For the LoopBackObject there is no processing, and for FFTObject it does a FFT algorithm. I have created and run 500 FFTObjects at once all in the queue. I was still able to interract with the engine. Although I could imagine doing that much processing could slow down the engine. YMMV
4. Data is extracted from the objects using a preset function calls defined in the LoopBackObject. Each object based upon LoopBackObject must create their own version of the getProcessedOutput function. This is a C++ function that can be used anywhere you need the data. The raw audio is retrieved using getAudioOutput. Both function return a Vector(F32). I need to add a couple of console functions to get this data. Right now it is recommended to only get raw audio from a LoopBackObject and not an FFTObject. The reason is the FTTObject "tweaks" its internal data using a "windowing" (math term) function.

Usage:

1. renderRTTExample.h/.cpp are not required for the audio loopback. It is an example of RTT.
2. processedMaterial.cpp and gfxTextureHandle.cpp are changes to the engine files of the same name. The Generic enabling of texture targets describes the same engine file changes. So these files are not needed if you already made those changes.
3. audioTextureObject.h/.cpp are the SceneObject based objects that visualize the audio data provided by the LoopBackObject for now.
4. loopbackAudio.h/.cpp are the loop back capture object and processing objects: LoopBackObject and FTTObject. It might be a good idea to separate the processing objects into a separate file so it is not dependent upon the Windows only code used for the loop back capture.
5. test.cs is a script file for testing using the code. It assumes the existence of certain gui elements. You can pick the names of those elements out of the script code. It should be fairly self explanatory.
6. usage.c shows how to setup the render to texture targets using my persistent NamedTexTargetObject SimObject. Make sure you use two values in texDims (not texSize) attribute (ie texDims = "1024 1024";) I will fix that, but I inadvertently used an older usage.cs file. The texSize attribute is not needed. Put the NamedTexTargetObject somewhere that it will get created before anything in the mission/gui objects that use it. It can be created at startup, just remember that it does create a texture of the size specified in texDims. So it can eat up memory on the graphics card if you create too many. Use wisely. This object basically solves the issue for creating a named texture target from being loaded in the wrong order if the texture was defined in a SceneObject. The NamedTexTargetObject is used by the AudioTextureObject.

Code:
Links to code:
Loopback Audio Version 1.0
Background render example

RTT required engine changes:
Generic enabling of texture targets

Why is this resource so crappy? How do I use this?
I am being purposely vague how to use this. It WILL require skill in C++ and T3D codebase to install. When I get some time I may add some data to show how to install this code and use each piece. However, I know that a large number of Torque developers are very capable of doing this. I just am behind on my other work. This started out last year as a simple feature and has grown like a vine. It also required me to take the time to learn a significant number of things in the engine, in the OS (threading), and in Windows. Basically this "simple" feature took me 4 or 5 months to piece together. Gah! Has it really been that long?! Anyway, I promised to get this out there so people can get their hands on pieces of this. That is the main reason I am posting now.

So, if you have question about installation please post. Then I can construct a step by step from where people had issues.

Why give this away?
Simply put: Because the GG community is awesome. They have made it possible to use a very good engine to make things I simply don't have time to build from scratch. Everyone here is so eager to help so any code I contribute back just makes this community that much better. I know that YOU are like me and have only so much time in the day. I also know that nothing of significance is done by individuals. So I want to enable you and your dreams. In return I ask that you just be your "awesome self".

Thanks for being so great!