Zworldo.com Blog 2 - Anatomy of a Web Installer
by David Wyand · 10/14/2010 (2:36 pm) · 11 comments
Zworldo.com is a free-to-play web site featuring games/worlds built with Torque 3D. Its first game, Greenwood Faire, is currently in private alpha testing.
INTRODUCTION
My goal for Zworldo.com is for the site to support a number of games, and for these games to be playable in the browser. I wanted all common files to be downloaded once and all game specific files to be downloaded only when required. And everything needs to be patched on demand as new revisions are released.Torque 3D includes the technology to deliver an entire game and have it play within the browser, but its philosophy in how that game is delivered differed from mine. Time to dive in and get coding.
BABY STEPS
My approach was to have a very minimal download and install up front. There are already enough hurdles to get someone to download new web content, and making it take longer than a few seconds is a killer. This means the user is only downloading the web plug-in itself. The plug-in would then be responsible for downloading the game content as required, and in a more web friendly manner. If it could resemble how Flash works, all the better (minus the requirement Flash has of always downloading the game content).To begin with, I integrated libcurl into the Torque 3D plug-in. This went fairly smooth as I initially went with a single threaded design. I didn't want the hassle of debugging multiple threads while learning how all of the technology had to come together.
Next, I came up with a file manifest document that could be downloaded by the web plug-in. This XML file details all files, their file sizes, their CRC values, whether they are base files or game specific, etc. This manifest is used to both download files for the first time, as well as patch them later. I've opted for patching entire files rather than trying to patch with binary deltas that are tracked against various releases.
To build the file manifest, I've written a Python script that goes through my game files directory tree. There are a number of hints that I've built into the script so it knows which files to include and which to ignore, which are site-wide files and which are game specific, etc. The Python script also copies all of the files into a clean directory that I can upload to my file server (currently my web server) for the user to patch from. I should also note that the script automatically ZIPs each file and it is this compressed version that is uploaded to the file server.
When the user navigates to a game's page on Zworldo.com, the web plug-in is told which game it should patch and it immediately downloads the file manifest. The plug-in then compares the required files list with the user's storage directory on their computer, and downloads the necessary files.
This method worked well and was a great proof of concept. Unfortunately it was too slow. The bottleneck at this stage was the downloading of each individual file. Having a whole bunch of small files to download doesn't take advantage of the user's bandwidth. The startup and shutdown for each file was too much overhead.
BUNCHES ARE BEST
When the user wants to start their first game on Zworldo.com that is when they have the most to download. There are all of the common files, as well as all of the game specific ones. So if these files could be grouped together into much larger ZIP collections, that would greatly help the download speed.Of course, the solution was to have a ZIP containing all of the common files between games, and a ZIP for each game specific file list. The process then, was to download a couple of ZIP files and decompress their catalog to the appropriate directories. This indeed gave a huge speed increase in the overall download and install process.
But how to handle users that have already downloaded the common and game content and only need to patch a few files? The solution is to have both available. The plug-in determines if any files for that game have been downloaded at all. If not, then use the ZIP collection files. If any files exist, then download the individual files required for that patch.
But now a new bottleneck showed up. Downloading the ZIP files was fine, but they needed to be decompressed and copied to the appropriate directory. And this process takes time. This version of the web plug-in would download and then decompress each file, one after the other. Time to allow these two steps to happen concurrently and move to a multithreaded plug-in.
MULTITHREADING -- ALL THE COOL KIDS ARE DOING IT
This is where I'm at today. The web plug-in downloads the first file from the manifest, and once that is complete, it passes it off to another thread that does the decompression. The plug-in then continues on with downloading the next file and queues it for decompression. This continues until all files have been downloaded and decompressed. Depending on the speed of the user's computer, the decompression may take longer than the actual downloading, so the plug-in waits for everything to complete before allowing the game to run.My final web plug-in workflow looks something like this:
From the user's perspective, while the files are downloading they are presented with a progress bar. It displays the number of bytes currently downloaded vs. the number still remaining, as well as the percentage complete. Once the download is done, if there is still decompression work going on, a second progress bar is displayed with the number of files remaining.
Download game files using the Zworldo.com web plug-in.
Something important to note that is not web plug-in specific, but user retention specific, is to allow the user to do something while the files are downloading. For my Greenwood Faire game, I allow the user to choose a gender and a name for their character. Down the road they'll also be able to choose outfits and different looking avatars.
Once everything is downloaded, the web plug-in notifies the web page and the game's Play button becomes active. To handle cases where there is a large delay between the files being patched when the user lands on the game's page vs. when the user actually presses the Play button (could be days), the patching process is automatically started again. It usually goes quite fast if there are no changes (did I mention I use a local file manifest to track patched files to speed up this check?) and then the game starts up right away.
Now the user is happily in the game and playing with their friends.
All blogs in this series:
Zworldo.com Blog 1
Zworldo.com Blog 2
Zworldo.com Blog 3
Zworldo.com Blog 4
- Dave
About the author
A long time Associate of the GarageGames' community and author of the Torque 3D Game Development Cookbook. Buy it today from Packt Publishing!
#2
I certainly hope that you are thinking of commercializing what you have done. I think a lot of T3D teams would love to utilize it to deliver their games through the browser.
10/14/2010 (4:24 pm)
Awesome stuff Dave. I had a chance to go into Zworldo a week ago and I have to say that the world is impressive and the technology that you have enhanced to make this happen is really awesome and provides a great experiance.I certainly hope that you are thinking of commercializing what you have done. I think a lot of T3D teams would love to utilize it to deliver their games through the browser.
#3
Even if you're not launching your game through a web browser, your way of delivering content is definitely one of the best solutions I've seen done with Torque.
Perhaps binary delta patching will also make it in one day, it would do miracles to download times. We're using a binary patcher, and I love how it can speed up the whole process. Granted, we've got a lot to enhance on the initial download size and process.
I really like the idea of character creation / customization while you download. Ingenious!
10/14/2010 (9:34 pm)
Very impressive, Dave. It looks like you're ahead of everyone with web plugins in Torque 3D. They are a great feature but with a design that is lacking (unfortunately, due to technical constraints as I understand).Even if you're not launching your game through a web browser, your way of delivering content is definitely one of the best solutions I've seen done with Torque.
Perhaps binary delta patching will also make it in one day, it would do miracles to download times. We're using a binary patcher, and I love how it can speed up the whole process. Granted, we've got a lot to enhance on the initial download size and process.
I really like the idea of character creation / customization while you download. Ingenious!
#4
@Logan:
I have thought of possibly commercializing my work. I would certainly need to clean it up some if others were to take a look. :)
But, honestly, right now I'd like to focus on games. I did tech before starting with GarageGames as a hobby (TST Pro), and now I spend all my time doing tech with IA (Torque). On my own time I'd like to work on the whole reason I got into this industry -- to make games.
Of course, if someone is willing to throw enough money at me, I'm sure I could be persuaded. I've got three kids to feed, after all. ;)
@Konrad:
Yes, my methodology could certainly transfer to non-browser based games. The one piece that would be missing is patching the patcher. This is handled in the browser world just by downloading a new plug-in.
As for binary delta patching: that made my head spin. Keeping track of what revision the user is at and then pulling down the correct delta to bring them up to the current release doesn't sound like a fun thing to tackle on my own.
Back in the day I saw how these two systems -- full file patching vs. binary delta patching -- were implemented for Everquest and Anarchy Online. EQ took the file approach, and AO the binary delta approach.
If you kept up to date with AO's patches (and it had some really big ones at its start) then it seemed to save some patching time by going with a delta. But if you ever missed a number of patches, it could potentially take a long time as you patch from revision A -> B -> C -> D. In these cases, just downloading a single file instead seemed to be quicker, especially if the file isn't all that large.
Of course, there are ways around this by keeping multiple revision combinations around, and perhaps other methods. But it wasn't something I wanted to spend time on right now. Especially as most of the patching happens with small files for Torque (i.e. script files). Unfortunately, when I do patch the game binary, the user does take quite a hit.
Hmm, maybe some sort of hybrid system would work...
- Dave
10/14/2010 (11:02 pm)
Thanks, everyone!@Logan:
I have thought of possibly commercializing my work. I would certainly need to clean it up some if others were to take a look. :)
But, honestly, right now I'd like to focus on games. I did tech before starting with GarageGames as a hobby (TST Pro), and now I spend all my time doing tech with IA (Torque). On my own time I'd like to work on the whole reason I got into this industry -- to make games.
Of course, if someone is willing to throw enough money at me, I'm sure I could be persuaded. I've got three kids to feed, after all. ;)
@Konrad:
Yes, my methodology could certainly transfer to non-browser based games. The one piece that would be missing is patching the patcher. This is handled in the browser world just by downloading a new plug-in.
As for binary delta patching: that made my head spin. Keeping track of what revision the user is at and then pulling down the correct delta to bring them up to the current release doesn't sound like a fun thing to tackle on my own.
Back in the day I saw how these two systems -- full file patching vs. binary delta patching -- were implemented for Everquest and Anarchy Online. EQ took the file approach, and AO the binary delta approach.
If you kept up to date with AO's patches (and it had some really big ones at its start) then it seemed to save some patching time by going with a delta. But if you ever missed a number of patches, it could potentially take a long time as you patch from revision A -> B -> C -> D. In these cases, just downloading a single file instead seemed to be quicker, especially if the file isn't all that large.
Of course, there are ways around this by keeping multiple revision combinations around, and perhaps other methods. But it wasn't something I wanted to spend time on right now. Especially as most of the patching happens with small files for Torque (i.e. script files). Unfortunately, when I do patch the game binary, the user does take quite a hit.
Hmm, maybe some sort of hybrid system would work...
- Dave
#6
For the moment, Zworldo.com's Greenwood Faire game is in private alpha testing, and is only available to TorquePowered Associates and Employees. I'm not quite ready to open it up to everyone just yet. But I'd like to open it up to a public beta over the next few weeks.
I'll post another blog here at torquepowered.com once I begin the public beta. I'd like to squeeze in another blog about installing the web plug-in itself before then, but we'll see.
- Dave
10/15/2010 (6:28 am)
@Shenyijun:For the moment, Zworldo.com's Greenwood Faire game is in private alpha testing, and is only available to TorquePowered Associates and Employees. I'm not quite ready to open it up to everyone just yet. But I'd like to open it up to a public beta over the next few weeks.
I'll post another blog here at torquepowered.com once I begin the public beta. I'd like to squeeze in another blog about installing the web plug-in itself before then, but we'll see.
- Dave
#7
I have globally the same system (manifest file server-side and client-side) to make a list of updated files, containing name, path, version and size.
It was made for a Adobe Director project.
Later, I ported it easily on Mac and then on iPhone.
I have as a project to port it to T3D using libCurl of course.
It's very simple structure and processing is what makes its force.
It's quite efficient, easily portable, and can work with different types of servers : local directory on the same computer, on the lan, on HTTP server, on FTP server and so on…
I nearly always think that simple techs are the best techs.
As a note, the small difficulty was : how to remove a file when it is no more needed after an update. The simple solution I found was : do not list it in the new server-side manifest, and the client will know that it has to delete it.
Nicolas Buquet
www.buquet-net.com/cv/
10/15/2010 (6:56 am)
Hi David,I have globally the same system (manifest file server-side and client-side) to make a list of updated files, containing name, path, version and size.
It was made for a Adobe Director project.
Later, I ported it easily on Mac and then on iPhone.
I have as a project to port it to T3D using libCurl of course.
It's very simple structure and processing is what makes its force.
It's quite efficient, easily portable, and can work with different types of servers : local directory on the same computer, on the lan, on HTTP server, on FTP server and so on…
I nearly always think that simple techs are the best techs.
As a note, the small difficulty was : how to remove a file when it is no more needed after an update. The simple solution I found was : do not list it in the new server-side manifest, and the client will know that it has to delete it.
Nicolas Buquet
www.buquet-net.com/cv/
#8
I'm sure my approach to patching is quite common -- I drew inspiration from years of playing MMOs. In fact, my server-side manifest format drew inspiration from Everquest 2. If you're interested in what my manifest XML format looks like, you can see a version of it here.
I also took the same approach as you did to deleting files. If the file isn't listed in the downloaded manifest, it is deleted. However, the manifest does have a DeleteExceptions section where I can list local files that should not be deleted.
- Dave
10/15/2010 (2:59 pm)
@Nicolas:I'm sure my approach to patching is quite common -- I drew inspiration from years of playing MMOs. In fact, my server-side manifest format drew inspiration from Everquest 2. If you're interested in what my manifest XML format looks like, you can see a version of it here.
I also took the same approach as you did to deleting files. If the file isn't listed in the downloaded manifest, it is deleted. However, the manifest does have a DeleteExceptions section where I can list local files that should not be deleted.
- Dave
#10
10/19/2010 (5:26 pm)
Cool info Dave. Stirred up quite a few ideas in my mind after reading :) 
#11
01/12/2011 (1:17 pm)
Dave I still hope you will sell this someday :-) 
Torque Owner 000
-