Streaming Video, Audio, and I/O
by Nathan Bowhay - ESAL · in Torque 3D Professional · 10/15/2009 (6:07 pm) · 2 replies
Ok so I know this has been talked about in the past and with services like OnLive and others emerging I know it is sort off a hot topic.
I was wondering about solutions for streaming the video and audio for something developed in T3D to a web browser and be able to send I/O back.
I know many people suggest just sending the packets and then have a special Torque Client simply use those packets like a advanced spectator mode, however this is not exactly what i would want to do cause I am thinking about using it on different devices.
The idea is closer to the OnLive reason that you could have less powered devices like a smaller pc with maybe a poor graphics card or even a small device like an iPhone that you could stream to. And then have a beefy machine to handle the rendering.
I thought about remote desktop like solutions on a smaller pc, but this wouldn't work for smaller devices like an iphone. The tricky part with devices like this is that streaming video through the web browser on it is non-existent, but doing something with I/O and screen captures may work. I was also wondering if it could be much more efficient and quicker to just grab the video feed from the video card for only my Program and sending that that out and only sending limited I/O. I was thinking of some sort of library to grab each frame after being rendered and then something else for the I/O sending receiving maybe even just writing a separate interface just for this.
Keep in mind as well what we are developing is less of a game and more of an application. So using this for things like environment creation or using guis or 2D items and then just videos like a spectator, so the quickness needed for things like an FPS isn't as big of a deal, the main focus is multiple platforms which is why I was thinking web based.
My main question is would grabbing a each frame from only my app and only specific I/O be that much better than a more broad already developed app like remote desktop or a video capture program (my experience with video capture software is it is slow and tends to use a lot of resources that should be being used for the game) in conjunction with something to handle I/O? There has to be a way to do it much better if you are integrating it into the application and have that sort of access.
PS: this is sorta a messy post, but I really just want all sorts of discussion about solutions or ideas and any existing libraries out there to do some of these things that could be brought into torque.
I was wondering about solutions for streaming the video and audio for something developed in T3D to a web browser and be able to send I/O back.
I know many people suggest just sending the packets and then have a special Torque Client simply use those packets like a advanced spectator mode, however this is not exactly what i would want to do cause I am thinking about using it on different devices.
The idea is closer to the OnLive reason that you could have less powered devices like a smaller pc with maybe a poor graphics card or even a small device like an iPhone that you could stream to. And then have a beefy machine to handle the rendering.
I thought about remote desktop like solutions on a smaller pc, but this wouldn't work for smaller devices like an iphone. The tricky part with devices like this is that streaming video through the web browser on it is non-existent, but doing something with I/O and screen captures may work. I was also wondering if it could be much more efficient and quicker to just grab the video feed from the video card for only my Program and sending that that out and only sending limited I/O. I was thinking of some sort of library to grab each frame after being rendered and then something else for the I/O sending receiving maybe even just writing a separate interface just for this.
Keep in mind as well what we are developing is less of a game and more of an application. So using this for things like environment creation or using guis or 2D items and then just videos like a spectator, so the quickness needed for things like an FPS isn't as big of a deal, the main focus is multiple platforms which is why I was thinking web based.
My main question is would grabbing a each frame from only my app and only specific I/O be that much better than a more broad already developed app like remote desktop or a video capture program (my experience with video capture software is it is slow and tends to use a lot of resources that should be being used for the game) in conjunction with something to handle I/O? There has to be a way to do it much better if you are integrating it into the application and have that sort of access.
PS: this is sorta a messy post, but I really just want all sorts of discussion about solutions or ideas and any existing libraries out there to do some of these things that could be brought into torque.
#2
It is definitely good you brought up the fact the machine would have to be faster than normal to handle the compression of the video and data recording. This is one of the key things I was worried about as I know from using a capture program that simply streams a window in in windows has low frame rates and lags the machine (server) down quite a bit. That is why this will be my focus on how exactly to handle it, I know lots of people who even do something as simple as recording a tutorial sometimes use a separate computer to capture the video or a piece of hardware specifically made to capture and record/stream video & audio.
Luckily for now I wouldn't have to worry about audio as we don't really use it and definitely wouldn't need to for the web based interface we where thinking about.
So my focus is really on the video and from the sounds of it there isn't any more control you have by being the one writing the app you need to stream vs just streaming a fully compiled app. The only advantage (from the sounds of it) that I could possibly handle the guis differently since they are 2D either with a different compression/streaming, since we have windowed viewports then this would cut down quite a bit on having to stream a large resolution 3D viewport, we could probably even look at the device and scale the viewport down to be smaller before streaming. But those are close to the only advantages we would still be streaming the video after it had been rendered and it wouldn't be any different really than using a video capture program.
Again thanks for the input and if anyone else has anything to add feel free.
Oh and as far as OnLive that is kinda what I though (casual games and other very simple things), but it seems like they are focused on full 3D games with fast input like Gears Of War or Modern Warfare (FPS's)... So either they have some magic protocol and algorithms for handling video or they are planning on piping fiber to every costumers house and having some of the most expensive beefy servers ever. In concept it can be done cause as of now you can stream full HD video (1080p) and audio over most High speed connections, but at that point the pipe is pretty full, so things like I/O could suffer. Not to mention if a video takes a while to get to you as long as it is consistent it isn't an issue, but with the interactivity a game has if you get that you are moving left late you will move more left than you want, by the time you moved out of the way of something you could already be dead. Guess we will see when they start opening it to the public to use (instead of a very closed beta program), I am definitely as skeptical as you though.
10/22/2009 (2:02 pm)
Thanks for the input. I knew that the reaction times would be unacceptable for something where you are constantly having to provide input, which is why when/if we end up implementing something. We are definitely going to have limited input. It will probably be something as simple as clicking a button and then changing to another camera and simply streaming the video from that camera (like observer). If we have anything interactive it will probably be all 2D and would use a good compression if not have it all on a separate client (web based) and then send it all as an entire thing once the user was done for example a web based 2D level editor where objects could be translated along the terrain (top down) and then they hit save and the data would be sent to torque and the level would be updated, but all the editing would be in a 2D web app (just an example).It is definitely good you brought up the fact the machine would have to be faster than normal to handle the compression of the video and data recording. This is one of the key things I was worried about as I know from using a capture program that simply streams a window in in windows has low frame rates and lags the machine (server) down quite a bit. That is why this will be my focus on how exactly to handle it, I know lots of people who even do something as simple as recording a tutorial sometimes use a separate computer to capture the video or a piece of hardware specifically made to capture and record/stream video & audio.
Luckily for now I wouldn't have to worry about audio as we don't really use it and definitely wouldn't need to for the web based interface we where thinking about.
So my focus is really on the video and from the sounds of it there isn't any more control you have by being the one writing the app you need to stream vs just streaming a fully compiled app. The only advantage (from the sounds of it) that I could possibly handle the guis differently since they are 2D either with a different compression/streaming, since we have windowed viewports then this would cut down quite a bit on having to stream a large resolution 3D viewport, we could probably even look at the device and scale the viewport down to be smaller before streaming. But those are close to the only advantages we would still be streaming the video after it had been rendered and it wouldn't be any different really than using a video capture program.
Again thanks for the input and if anyone else has anything to add feel free.
Oh and as far as OnLive that is kinda what I though (casual games and other very simple things), but it seems like they are focused on full 3D games with fast input like Gears Of War or Modern Warfare (FPS's)... So either they have some magic protocol and algorithms for handling video or they are planning on piping fiber to every costumers house and having some of the most expensive beefy servers ever. In concept it can be done cause as of now you can stream full HD video (1080p) and audio over most High speed connections, but at that point the pipe is pretty full, so things like I/O could suffer. Not to mention if a video takes a while to get to you as long as it is consistent it isn't an issue, but with the interactivity a game has if you get that you are moving left late you will move more left than you want, by the time you moved out of the way of something you could already be dead. Guess we will see when they start opening it to the public to use (instead of a very closed beta program), I am definitely as skeptical as you though.
Torque 3D Owner Marc Dreamora Schaerer
Gayasoft
A 200ms+ reaction time is for anything but RPG and "casual games" completely worthless and on devices like the iphone its even worse as the input itself is already slower, the network is slower and as such you end with reaction times outside the acceptable. It must never be forgotten that the other machine must not only handle the rendering but the whole world update.
Even at best, the application will feel more sluggish than lets say "Second Stutter" where the input reaction is that awkward that I and many others can't use it for more than a few minutes before I go nuts (I guess only people with crap slow systems can use it as the input there is on the same performance as the rest)
Your actual experience in such an environment on realistic network connections (not the connection OnLive used when they showed it) is in the range of the effect that you get with a high end graphic card on an engine that don't compensate for the input lag caused by prerendering, where your input does not take effect until 200-400ms later, just with the difference that you are already that much further at that point in time with your input again.
But if you really want to go that way, you basically need three things:
1. An image snapshot / video stream from server to "client"
2. An input data stream from client to server
3. An audio execution command stream that tells the client which local audio to play (if possible that would be optimal)
1 and 3 become one if you don't want local audio data but you would lose any support for more than 2 audio channels practically.
The second one is pretty simple I think. Even basic data compression is likely not needed due to the low sampling rate.
Your primary problem will be 1, because the common image formats etc are not really an option for this purpose due to the massive data amounts you would need to send. Your best option is looking into the research on video stream if you want to develop an own format or look into the existing integrateable technologies you could integrate for this purpose.
In any case, it must be clear, that your "server" must be significantly stronger than a said computer on the client side would need to be for the same quality and performance because the compression and data recording comes at massive costs.
T3D has the benefit that it is already basically a Client - Server architecture, this makes this approach simpler for the server end.
The client end could either be a totally cut down and rewritten T3D based core or an own development.