Designing games for the Oculus Rift - a success story

(Got the above 2 images from Google image search, no copyright infringement intended)



The Enterprise sitting in space dock, with Earth orbiting in background.

Hello all,

Some of you may have seen the forum posts from Brian M and myself lately, as we integrate Oculus Rift support into our custom engine build - This has been a challenge on a number of levels, but we have made great progress and thought it would be useful to share our findings with the community here.

ORIGINAL BUILD

In the interest of full disclosure, here is what we had working prior to integrating the Rift:

• T3D MIT 3.0
• AFX 2.0 (for special effects and object selection)
• Custom object selection code (assign a button command to any TSStatic)
• QT Webkit support (for web-enabled objects)
• Pathshape resource (for moving platforms/elevators)
• Climbable objects resource (for ladders, etc)
• Paul Dana's turret resource (AI- and player-controlled turrets)
• The Universal AI Starter Kit (controls our AI crew and enemies)
• Custom SnapTo functionality in World Editor (select two objects and line them up)
• GenEpak code for packaging art assets into secure PAK files for release

With this package of features, we completed building the entire original starship Enterprise, and have been giving it away on our web site since Paramount is not interested in licensing us to sell it (that's another blog story)... We then started working on some other projects, using our engine and what we learned working on the Star Trek game. As fun as it was to keep working on it, we decided to shelve the Enterprise now that it was complete - time to actually work on some stuff we can sell down the road.

But then the Oculus Rift came out - Here was a chance to take things to the next level. Finally, a viable VR solution that went beyond the 80's-era Lawnmower Man look. So we purchased two Rifts and started learning how to use it in our games.

You know where this is going - yes, we pulled out the Star Trek game AGAIN and began tweaking it for Rift support. We had to walk the Enterprise in Virtual Reality, and be able to interact with all the consoles and buttons, etc, that we had built into our final release... It was a moral imperative!

INTEGRATING THE OCULUS RIFT - PROBLEMS ENCOUNTERED


First crack - looks great, but the AFX highlighting on Sulu is an issue... And the 2D GUI player names have to go.

The actual engine integration of the Rift was pretty straight forward - following the Wiki instructions quickly gets you a Rift-capable engine build... It did not conflict with any existing resources we had, and we were able to pretty much start using it right away on our existing game.

NOTE: The first time we built the solution with the Oculus support enabled, we forgot to add the Extended Move class support. That is done in the project.conf file, and is explained on the Wiki. Without this, the Rift's head tracking features will not work properly, so make sure you don't miss this step or you'll have to compile a second time like we did.

So, with Extended Move support and the Oculus SDK hooked up properly, we got a successful build and were excited to give it a try...

But that is when we encountered the issues I would like to share and discuss here - namely, how do you build interactivity into a game where you have no usable mouse cursor and no easy way to have popup GUIs? The Rift works with a stereo-rendered view of the 3D scene, and GUIs are 2D elements - when you pop a GUI up on-screen, it will get split across both eyes since it is being overlaid on top of the stereo-rendered scene. And without being able to use the mouse, it wouldn't matter even if we could successfully duplicate the GUI in both eyes - the mouse cursor will only be in one eye at a time as you move it around, since it is not stereo-rendered.

Our QTWebkit objects were not usable in the Rift either, since they just used the standard AFX mouse cursor (which as I mentioned is not usable in the Rift).

Also the nice highlighting that AFX does, when you mouse over an object, was appearing only once and offset in the Rift (instead of being stereo-rendered) and did not match up with the object in question. This left a ghost image on-screen offset from what was selected, distracting to the user.

To top it all off, when we tried to mount a shuttle to fly it around with the Rift on, the game kept crashing.

So, we were left with the following issues at this point, when trying to play the game with the Rift:

• Vehicles unusable
• Web objects unusable
• Selectable objects unusable
• AFX highlighting not working properly
• GUI elements unusable

Creative solutions were required here, this was not quite the turn-key solution we had hoped for - We had to rethink the entire way a user interacts with the game if we ever wanted this VR headset to be usable to explore the Enterprise.

Here are the various ways we handled each of the problems:

VEHICLE MOUNTING


Wanting to drive the Cheetah on CESTUS III, but game would crash every time. And there's that highlighting offset again.

We got lucky with this one - within a week of me posting about this issue, we had Michael Reino answer that he had succeeded in getting the mounting working with a Cheetah in the Rift. He posted all his code and we integrated the changes. Sure enough, we could now mount our shuttles and fly them around. Not only that, but now we could look around inside the shuttle using the head tracking while we flew straight forward using the keyboard and mouse to steer. What an immersive experience!

WEB-ENABLED OBJECTS

It was apparent that we needed a new web object solution, QT Webkit was no longer going to cut it - Extensive research brought us to the Awesomium integration pack by Stefan Lundmark... GG user NiN-NiN had posted about how he had achieved success with Awesomium web objects in the Rift, and when we watched the demo we saw a stereo-rendered cursor on top of web objects, exactly what we were looking for.

So, we ripped out all the QTWebkit code from our build, and swapped in Awesomium. Converted all our existing web objects to AwShapes and they all came back with a usable mouse cursor in the Rift, one problem solved!

There were some issues with web objects not always coming up properly, but we have since upgraded to Stefan's latest release of the Awesomium pack and that seems to have improved things greatly.

So, to everyone looking for a web-enabled objects solution, Awesomium gets our vote - superior to the QT Webkit in several ways, and easier to integrate into the source code. And it works perfectly in the Rift.

SELECTABLE OBJECTS


Those helm buttons are clickable, but with no mouse cursor how the heck does that work?

Now, if only that Awesomium dual mouse cursor could be used in the rest of the world, so we could regain the clickable objects functionality we had that gets lost when using the Rift... But Awesomium sticks the cursor to the web object and then only shows it when your field of view is looking at the web object - so you can pan around the content of a web object fine, but when your view passes away from it the cursor disappears... It is only usable on the objects themselves.

In the Rift, the weapon will fire at whatever you are looking at, just as it would in normal non-Rift mode. Without the Rift, the crosshair tells you where your weapon is pointed, but in the Rift the crosshair does not work because it is a 2D GUI element in the center of the screen.

So what we needed was some sort of way to produce a crosshair in the Rift - something that would be centered on the user's view and rendered in stereo.

First we tried to see if the AFX highlighting code could be fixed up to render in stereo, and thus we could use that to indicate to a Rift user what they are pointed at and about to click on... However this didn't get much traction and seemed to be a major issue in terms of AFX engine fixes required.

But we were able to combine the AFX RolloverRayCast function and the Awesomium raycast check, thus making it so what you pointed your view at would get preselected if it was a clickable object. In non-Rift mode, this achieves the same effect as if a user has used the mouse to click on an object - they can also now just point their view at something to preselect it... Half the battle solved, we could now look at objects to preselect them, but without the AFX highlighting working we also needed a way to tell the user exactly what they are about to click on (think of a panel of small buttons close together, we needed to be able to fine-tune the aim and point it at a specific button in order to use what we already had in place all over the ship... Impossible to do without some sort of aiming cursor/reticle).

We looked at various custom crosshair resources and couldn't find anything that would really work the way we wanted, until we got lucky again - Gibby posted a resource in response to our question, that used AFX to mount lights on our weapons. With a little adjustment, this light could be made into a laser pointer and tweaked to fit right in the center of the crosshair (ie. Pointed right at what the user is looking at). So when in Rift mode, the laser dot is stereo-rendered and wherever the user points that dot, the object underneath it will be preselected if it is clickable... All the user needs to do at that point is right-click to action the preselected object.

This was a major coup - now we did not have to alter any of the existing small clickable buttons we had in the various bridges and control rooms of the ship. We could pinpoint the button we want to click using the laser pointer, then just right-click the mouse to action it.

Thanks so much to Gibby for solving that one for us, and to Jeff Faust as always for his tireless work supporting AFX... Everyone should have that product integrated into their T3D builds, there is just so much useful stuff in there.

AFX HIGHLIGHTING

Since we were unable to get this feature working properly in the Rift, and since we now had a laser pointer to act as a reticle, there was no need for object highlighting in the Rift anymore. So a simple change to the engine now checks for a global variable to see if the Rift is turned on or not. If the Rift is being used, the engine just bypasses the AFX highlighting code... So now we no longer get ghosted images in the Rift when our view passes over selectable objects.

GUI ELEMENTS

_
Old popup GUI-based map on left, new in-game Web-based map object on right.

This turned out to be the easiest one - there is no way to make GUIs play nice in the Rift, so when you load up in Rift mode you get a HUDless PlayGui, with no GUI elements at all. Michael Reino's Cheetah/Rift fix included some useful functions for removing the HUD elements and toggling between Rift and non-Rift mode, so we make use of those.

We also disable the 2D GUI Hud that shows the player names, since they are all offset in the Rift and only render in one eye at a time.

The only thing with turning off all the GUI elements is that our turbolift system used a popup GUI map system for the player to choose which deck and destination they are going to... So we converted the map system from GUI-based to web-page-based, and put Awesomium objects in all the turbolifts - Now a user enters any turbolift and sees a web page panel with an image map of the ship, and they can select a deck and then select a destination on that deck...

The Awesomium kit includes a Javascript to Torquescript bridge (and vice-versa) which is just about the coolest thing ever created, period... Anything you click on in a web page can execute Torquescript commands in the game. So we use this to have image map areas in the web page (turbolift destinations on an overhead map of a deck) be clickable and they actually teleport you to that location when you click them.

I didn't believe it until I actually got it working and was stunned. The possibilities with this are endless, think about it...

Stefan you are a miracle worker for creating the Awesomium pack, well done!

TWEAKS AND UPGRADES

After getting this far, things were really humming along... So much fun to be completely immersed when exploring the Enterprise. But there were still some intermittent problems with Oculus head tracking and point of view getting messed up when entering/exiting vehicles, as well as our Awesomium shapes were not always coming up 100% of the time.

We were lucky again with two updates - GG posted some Oculus fixes and related updates for T3D MIT 3.5, and the Awesomium integration pack was updated with bug fixes and optimizations.

Applied both of these updates, as well as upgrading our Oculus SDK to the latest version (required for latest GG changes), and now we really have a well-oiled machine... The game is working so well, and it is an unbelievable experience being able to walk the whole ship in Virtual Reality.

FINAL THOUGHTS, AND THE FUTURE

The things we have learned in this process will help us greatly moving forward - we now have an idea of how games of the future may look and work, once Virtual Reality becomes more widespread...

Never really thought that adding the Oculus Rift to our list of peripherals would result in a complete overhaul of our game's user interface, but that's what was needed to really make it work. Those of you who are interested in having this kind of support in your games now or in the future, will need to confront these issues as well...

Our hope is that this blog gives you a foundation to work off of, and that maybe we can all share our experiences with designing games for the Oculus Rift.

If anyone has a Rift and would like to try this out, please contact me... This latest updated build with Rift support is not available on our website, but I would be happy to package up a binary installer for any Rift users who want to help test it.

For all non-Rift users, if you have not tried our free Star Trek Enterprise exploration game, please head on over to our website and download it, and feel free to leave us any feedback here... It's a pretty good example of all the things you can do with T3D.

Cheers and thanks to all who helped us figure this out.

Would be happy to answer any questions you all may have... Let the discussion begin!


ETA: Resource / software / forum thread references:

Vehicle fixes for the Rift (Michael Reino) - www.garagegames.com/community/forums/viewthread/135827

afxWeaponLight resource (Gibby) - www.garagegames.com/community/resources/view/22561

Awesomium integration pack (Stefan Lundmark) - www.garagegames.com/community/blogs/view/22068

Snap To functionality in mission editor (Davis Ray Sickmon, Jr) - www.garagegames.com/community/resources/view/5880/2#comment-195821

GenEpak for T3D (Crispy Software Development) - www.crispysoft.com/prod_genepak.html