Online video sound for TGEA

#1
As is, the SFX system in TGEA needs some custom code modifications to be able to do this. If you only want to support this on a single SFX device, you can pretty quickly hack this in.

As you have a Torque 3D license, you could also use the SFX changes from Beta 3 and bring them over to TGEA. It has a couple of dependencies that need to go with it (the platform/threads/ and platform/async stuff plus some more), but all in all this should be a painless thing.

It has an SFXPacketStream class that can do exactly what you need. Just feed it the raw sample packets you are receiving from your video feed and it will push them to the sound device as needed.

BTW, video playback in Torque 3D will hopefully be back by B4. This may also be useful to you.

#2 Rene, Thank you very much.
Does it mean in TGEA1.7.1, audio stream from video is NOT supported currently, am I right?

So I have to port Torque 3D related stuff to TGEA1.7.1's SFX system, right? Are there any tips or tricks to do a mod with TGEA1.7.1's SFX system directly to support the audio stream of video?

Thank you very much, very appreciated.

#3
TGEA's SFX system has no streaming support at the moment, yes.

If you wanted to hack something in, chose one of the devices (FMOD or OpenAL if you want cross-platform support; OpenAL is thread-safe and its queuing is easier to work with, though there is a number of issues and all in all queuing tends to be trickier to work with than the simple and straightforward wrap-around buffers that FMOD and DirectSound use) and then start working your way up from the voice and buffer classes to create a data path for your sound stream. This can be hacked together pretty quickly.

If you think about porting Torque 3D's SFX changes, I'll recommend waiting a bit (at least until B4 if not until final) so the kinks will be out before you start porting. Otherwise you'll probably have to run a few merge passes.

Instead of going through all this pain, you could also do your audio/video streaming completely separate of the SFX system. People have done this before by linking other audio/video libraries in and creating a few console functions to control them.

#4 Thanks again, Rene,

Your suggestion is very helpful.

#5 Well, it seems that it's far more complicated than what I've imagined. Especially about the synchronization between video and audio stream. Very difficult and I have to give it up with the method.

I'm eager to know if anyone else has implemented such audio stream in TGEA1.7.1 or others, it will be very appreciated for any tips or tricks are given. :)

Thanks a lot.

#6
Yes, synchronizing the frame and the sound stream takes a bit of work.

While some form of sound streaming can be hacked together pretty quickly, the rest will take more work.

If you really need this, especially the streaming from online channels, I'd definitely look into existing APIs that do most of the work for you and integrate them into Torque. Apple's QuickTime API would be one way, for example.

Nonetheless, this remains a non-trivial undertaking but given a nice API, it should be quite doable.

#7 Rene, Thank you very much and we're very appreciated if you can do this for us.
In fact, we need this very eagerly/urgently. We have tried it for several days and got no results.

We are very appreciated for your help, expecting your progress. :)


Thanks a lot.

#8
I'm sorry, Nabarro, about my rather unfortunate wording. Stumbling time and again over not being a native speaker.

What I meant is that in your case, I recommend looking integrating QuickTime into Torque.

Unfortunately, doing online video streaming in TGEA will probably require quite a bit of custom coding on your end.

BTW, where do you want to stream from and in what format?

#9 Hi, Rene,
Thank you very much for your suggestions. Our video stream is from our online 24-hour TV broadcasting video stream, and it's format if the general flv format.

Currently we don't have the basic method of doing it, if you think QuickTime is ok, then we can try to do the customization for it. While it will be very appreciated if any references can be taken. Because you are the expert in the area, and we're sure you can give us some instructions with it.

Thank you very much.

#10
Unfortunately, my expertise here is kind of limited. I haven't myself used any existing video codec library except for Theora.

AFAIK you can't play FLV through the QuickTime API, though I may be wrong.

The following requires quite a bit of work and a working C++ knowledge, but here's how I think I would go about it in your case if I had to strictly tick to TGEA:

* Create a sound streaming path by deciding on a single device and change SFX to create me an SFXBuffer on that device that I then use to directly access and feed the device-specific info in it.
* Compile and link in ffmpeg as a library (given that you can live with its licensing issues and the mess this thing is)
* Write the code to connect to and read from the online stream
* Pipe its data stream into libavcodec
* Pipe the output stream into a texture and the SFXBuffer (depending on the device used, you'll need access to an associated SFXVoice, too)

As you see, this is quite a non-trivial undertaking and will require some serious coding work.

However, there probably is no quick and easy way to do this with the codebase as it is.