Deploying Indie Games
by Konrad Kiss · 06/09/2011 (6:33 am) · 27 comments
Whether it's a game or another kind of desktop application, deploying it to your users can be a pain if you can't afford to or don't want to pay the price for the high-end tools. There are alternatives, however. And there are pitfalls.
This has been posted on my personal blog but I thought it would hopefully benefit the community if I re-posted it in my GG blog. Hope it saves you the headaches I had.
Since the game my studio works on is an MMO, we had to keep all of the game logic strictly on the server side. This means that the game server is authoritative and the client is merely a "screen" to display what the server is computing.
Still, protecting game assets in the client version of Xenocell is a very important task. Not so much because people can steal our stuff, but also because we use assets that assume such protection and we agreed to abide by their license agreements - such as an adequate level of protection of these assets against 3rd party use.
So after taking a good look around, we decided to go with Molebox VS to pack our game's contents. This solution has the benefit of being transparent to the game code. It functions as a virtual file system so we did not have to make any changes to the existing code, which was a huge benefit.
Alas, as I had to learn, virus and malware authors also found this packing solution elegant. Although Teggo Software - the studio behind Molebox - ensures its customers that it has a "unique executable template for every customer. This helps preventing antivirus false positives since applications of different users look less similar." - it doesn't actually work as advertised, unfortunately. Interestingly enough, I don't think it's Teggo's fault.
Most users can't tell the difference between a false positive and malware.
You see, anti-virus software needs to be fast to stay ahead of the competition. Almost all of these solutions default to marking your executable suspicious if it was packed with a packer that has been used before for creating malware. They can't always look behind the protections employed in the packer to actually know if your game executable is indeed dangerous or not, so they assume it is. There's simply no time for the AV to crack the protection and take a peek at your binary data.
It is very bad practice to assume that anything that has been packed with Molebox (or any other packer for that matter) is dangerous. But it's safe for the user, and I'm not sure I could come up with a better solution. I'm sure however that these alerts hurt many businesses that become aware of the issue only when it's too late.
I don't dare to assume that I could stop dedicated crackers from getting to what they want, and it would be a mistake to think I ever could - but if I can keep our assets safe from 99% of the users then it's a compromise I can live with. For an online game with a privately hosted authoritative server, the decision was easy: be as transparent to anti-virus software as possible. So I opted to not use Molebox's advanced encryption and obfuscation mechanisms, and - it worked! I did a few quick checks to see how the executable did and the results were great!
But this wasn't the end of the story. The executable still bears the marks of our packer, and during the installation process it again gets packed into the setup executable. I assumed that AV programs will be able to see inside the installer binaries, perhaps check the individual files, but they will - again - not bother checking them thoroughly. I was right, unfortunately, because AV software reported the setup .exe as a false positive right at the moment we tried to download it during our deployment tests.
For patching and installation we use Putchisoft's Dispatcher - it is a very powerful tool with binary difference patching capabilities for a very indie-friendly price. Not only that, but Robin, the guy behind Dispatcher offers one of the the best customer support I've ever seen, and he is also available for custom modifications to the software. (We worked together with him to find a way for Dispatcher to be able to handle patches published via Amazon's CloudFront service while avoiding its caching mechanism to make patches available around the globe instantly for everyone to download.)
So with Dispatcher, I managed to go another route. Since installation (NSIS) is a part of Dispatcher's deployment process, I changed our setup executable to not include any of the game files, but download them instead as a part of the install process. Dispatcher has a built-in way to automatically download the latest version if the installation is damaged beyond repair. Not having any of the game files in the installation directory pretty much satisfied this condition.
This approach had a number of advantages. First, what mattered most to me was that the game executable was being transmitted in a fashion that was again transparent to AV software - it could easily confirm that the game executable is not harmful. On the other hand, it was less than 200 kilobytes in size, which was a great side-effect. The third best thing was that running it would always get the latest version of our game. The only drawback was that it required an Internet connection at install time. Since our game assumes a constant Internet connection, this didn't bother me much.
I guess that's it. False positive alerts from anti-virus software could have hurt our business significantly, but in the end instead of more obfuscation, transparency proved to be the right way to go.
As always, thanks for reading - any comments are much appreciated (either here or on my blog). I hope to continue my AWS series that I started with my previous blog post with an example on how to host your game downloads in the cloud and how to make it available with mirrors all over the world.
@konradkiss
KonradKiss @ GitHub
konradkiss.com
This has been posted on my personal blog but I thought it would hopefully benefit the community if I re-posted it in my GG blog. Hope it saves you the headaches I had.
Since the game my studio works on is an MMO, we had to keep all of the game logic strictly on the server side. This means that the game server is authoritative and the client is merely a "screen" to display what the server is computing.
Still, protecting game assets in the client version of Xenocell is a very important task. Not so much because people can steal our stuff, but also because we use assets that assume such protection and we agreed to abide by their license agreements - such as an adequate level of protection of these assets against 3rd party use.
So after taking a good look around, we decided to go with Molebox VS to pack our game's contents. This solution has the benefit of being transparent to the game code. It functions as a virtual file system so we did not have to make any changes to the existing code, which was a huge benefit.
Alas, as I had to learn, virus and malware authors also found this packing solution elegant. Although Teggo Software - the studio behind Molebox - ensures its customers that it has a "unique executable template for every customer. This helps preventing antivirus false positives since applications of different users look less similar." - it doesn't actually work as advertised, unfortunately. Interestingly enough, I don't think it's Teggo's fault.
Most users can't tell the difference between a false positive and malware.
You see, anti-virus software needs to be fast to stay ahead of the competition. Almost all of these solutions default to marking your executable suspicious if it was packed with a packer that has been used before for creating malware. They can't always look behind the protections employed in the packer to actually know if your game executable is indeed dangerous or not, so they assume it is. There's simply no time for the AV to crack the protection and take a peek at your binary data.
It is very bad practice to assume that anything that has been packed with Molebox (or any other packer for that matter) is dangerous. But it's safe for the user, and I'm not sure I could come up with a better solution. I'm sure however that these alerts hurt many businesses that become aware of the issue only when it's too late.
I don't dare to assume that I could stop dedicated crackers from getting to what they want, and it would be a mistake to think I ever could - but if I can keep our assets safe from 99% of the users then it's a compromise I can live with. For an online game with a privately hosted authoritative server, the decision was easy: be as transparent to anti-virus software as possible. So I opted to not use Molebox's advanced encryption and obfuscation mechanisms, and - it worked! I did a few quick checks to see how the executable did and the results were great!
But this wasn't the end of the story. The executable still bears the marks of our packer, and during the installation process it again gets packed into the setup executable. I assumed that AV programs will be able to see inside the installer binaries, perhaps check the individual files, but they will - again - not bother checking them thoroughly. I was right, unfortunately, because AV software reported the setup .exe as a false positive right at the moment we tried to download it during our deployment tests.
For patching and installation we use Putchisoft's Dispatcher - it is a very powerful tool with binary difference patching capabilities for a very indie-friendly price. Not only that, but Robin, the guy behind Dispatcher offers one of the the best customer support I've ever seen, and he is also available for custom modifications to the software. (We worked together with him to find a way for Dispatcher to be able to handle patches published via Amazon's CloudFront service while avoiding its caching mechanism to make patches available around the globe instantly for everyone to download.)
So with Dispatcher, I managed to go another route. Since installation (NSIS) is a part of Dispatcher's deployment process, I changed our setup executable to not include any of the game files, but download them instead as a part of the install process. Dispatcher has a built-in way to automatically download the latest version if the installation is damaged beyond repair. Not having any of the game files in the installation directory pretty much satisfied this condition.
This approach had a number of advantages. First, what mattered most to me was that the game executable was being transmitted in a fashion that was again transparent to AV software - it could easily confirm that the game executable is not harmful. On the other hand, it was less than 200 kilobytes in size, which was a great side-effect. The third best thing was that running it would always get the latest version of our game. The only drawback was that it required an Internet connection at install time. Since our game assumes a constant Internet connection, this didn't bother me much.
I guess that's it. False positive alerts from anti-virus software could have hurt our business significantly, but in the end instead of more obfuscation, transparency proved to be the right way to go.
As always, thanks for reading - any comments are much appreciated (either here or on my blog). I hope to continue my AWS series that I started with my previous blog post with an example on how to host your game downloads in the cloud and how to make it available with mirrors all over the world.
@konradkiss
KonradKiss @ GitHub
konradkiss.com
About the author
http://about.me/konrad.kiss
#2
06/09/2011 (9:59 am)
Ditto the thanks, I haven't seen much of anything listed on deployment issues before. I have a great fear of making a great product only to have it ruined on launch, and you've given me several things to consider! 
#3
Question: by downloading your files with Dispatcher, dont you loose the compresion that could have the package by the initial method?
(I dont mean the Molebox packing of course, I mean rar, zip or any other compression format).
06/09/2011 (12:08 pm)
Excellent read Konrad!Question: by downloading your files with Dispatcher, dont you loose the compresion that could have the package by the initial method?
(I dont mean the Molebox packing of course, I mean rar, zip or any other compression format).
#4
If you are worried about AntiVirus you are going to have bigger problems, there are about 6 out there at the moment whom find the compiler inside of T3D to be a false-positive (deletes your .exe on launch).
You are right on par with how to deploy an MMO through web deployment though. Small sized patcher, and have it download the game through the patcher. You'd be even more on par if you looked into streaming download, AKA download while you play. It will save you big money in the end on your bandwidth. Not to mention give you a better launch, as everyone will not be downloading the whole game on launch day, but rather bits and pieces as needed.
06/09/2011 (12:09 pm)
I have a question for you Konrad, are you going strictly web deployment?If you are worried about AntiVirus you are going to have bigger problems, there are about 6 out there at the moment whom find the compiler inside of T3D to be a false-positive (deletes your .exe on launch).
You are right on par with how to deploy an MMO through web deployment though. Small sized patcher, and have it download the game through the patcher. You'd be even more on par if you looked into streaming download, AKA download while you play. It will save you big money in the end on your bandwidth. Not to mention give you a better launch, as everyone will not be downloading the whole game on launch day, but rather bits and pieces as needed.
#5
@Novack: No, Dispatcher keeps all your files intact. There's a 7zip compression option that you can use during file transfer, but Dispatcher is pretty much transparent in how it works. It does not have or introduce any dependencies.
@Richard: I've been in contact with AV companies and they are a lot more flexible than I thought. Having said that, it would be a good idea to see why T3D is treated as a false positive. I've seen many false-positive alerts on innocent files mainly by the smaller AV companies who push updates out once a month and don't have the manpower to keep up with the big guys.
I think a line must be drawn somewhere - it won't be possible to satisfy everyone, but since the market penetration of these smaller AV programs is negligible, I'm not worried about being identified as a false positive on those. Out of a 100 AV programs at least one will identify notepad.exe as harmful. It's not exact science. :)
06/09/2011 (12:17 pm)
Thanks for the comments, guys! I'm glad this is useful to some of you.@Novack: No, Dispatcher keeps all your files intact. There's a 7zip compression option that you can use during file transfer, but Dispatcher is pretty much transparent in how it works. It does not have or introduce any dependencies.
@Richard: I've been in contact with AV companies and they are a lot more flexible than I thought. Having said that, it would be a good idea to see why T3D is treated as a false positive. I've seen many false-positive alerts on innocent files mainly by the smaller AV companies who push updates out once a month and don't have the manpower to keep up with the big guys.
I think a line must be drawn somewhere - it won't be possible to satisfy everyone, but since the market penetration of these smaller AV programs is negligible, I'm not worried about being identified as a false positive on those. Out of a 100 AV programs at least one will identify notepad.exe as harmful. It's not exact science. :)
#6
Also, you have a personal blog? Now bookmarked with the others I watch ...
That's the look of a negative! Did I ruin my life? No! Huzzar!
:P
06/09/2011 (12:19 pm)
Fascinating stuff, Konrad. Also, you have a personal blog? Now bookmarked with the others I watch ...
That's the look of a negative! Did I ruin my life? No! Huzzar!
:P
#7
06/09/2011 (12:23 pm)
@Richard: Sorry, forgot to answer your question on web deployment. I am hoping for us to appear on Steam eventually and we are working with Gamespy on Direct2Drive publishing. I also intend to pray the full install up onto as many game magazine dvds as I can - once we launch of course. So no, not only web deployment. I will work my way around it somehow.
#8
06/09/2011 (12:24 pm)
@Steve: Thanks! About that blog.. yes, just started this month as a hobby. One that doesn't feel like betraying the game. Ah.. who am I kidding..
#9
06/09/2011 (12:31 pm)
You got my question the other way. My point is: If you download your files with Dispatcher, you loose the option to compress the files before downloading. It probably wont matter for patches, but on the initial download, you could make a lot of difference by compressing the files. Have you considered it?
#10
But that 260 Megs is never "extracted" since Molebox's virtual file system takes care of it. We have only a few bigger data files with .gap extensions and the executable basically. Everything is in these gap files, compressed and separated by function. So if we need a gui update, then only one of these .gap files is patched - using binary differential patching, compressed for delivery. So we compress a patch file that patches the compressed data file. :) There's compression all around.
06/09/2011 (12:40 pm)
@Novack: Oh I see what you mean. We do compress them with Molebox. It does a great job even on compressed formats - the uncompressed game right now is 480 Megs, while the installed, final game is 260 Megs.But that 260 Megs is never "extracted" since Molebox's virtual file system takes care of it. We have only a few bigger data files with .gap extensions and the executable basically. Everything is in these gap files, compressed and separated by function. So if we need a gui update, then only one of these .gap files is patched - using binary differential patching, compressed for delivery. So we compress a patch file that patches the compressed data file. :) There's compression all around.
#11
06/09/2011 (12:43 pm)
@Konrad - don't you find that setting the encryption too high, slows down the loading process of the virtual file system? I've not tried molebox yet, but other similar ones and on slower systems it grinds.
#12
As for speed, your data needs to be decrypted, and that definitely takes time, but if the decryption is not too complex then you can walk away with some speed improvement thanks to no further file fragmentation in the one file virtual data storages. If I had to decide whether Xenocell was slower or faster with Molebox, I'd probably say faster, but I really can't tell the difference. It's not an issue at all, fortunately.
06/09/2011 (12:51 pm)
@S2P: Well, one way to get rid of the false positive alerts in AV software is to not set it too high. So we have a pretty low encryption setting set with trial and error. There is some encryption, but it's simple.As for speed, your data needs to be decrypted, and that definitely takes time, but if the decryption is not too complex then you can walk away with some speed improvement thanks to no further file fragmentation in the one file virtual data storages. If I had to decide whether Xenocell was slower or faster with Molebox, I'd probably say faster, but I really can't tell the difference. It's not an issue at all, fortunately.
#13
06/09/2011 (12:52 pm)
Cool, thanks Konrad =)
#14
06/09/2011 (1:01 pm)
Ok thanks Konrad, I'll give Molebox a go at some point and see what the differences are speed wise.
#15
For Xenocell to generate new patches for previous versions and a new installer it only takes the click of a button now - based on an SVN revision. And it happens in the cloud too, on a dedicated server started only for the duration of generating the patch (meaning that costs "mount" to somewhat less that $2 (!) a month if we do one patch a week). Then it deploys the patch to an Amazon S3 bucket and through that - using CloudFront - it appears everywhere around the world within about 15 minutes of pressing that button. Pretty convenient. :)
06/09/2011 (1:16 pm)
@S2P: No problem. You can try Molebox VS for free. I don't especially like the user interface, but both Molebox and Dispatcher can be used via command line, so it's easy to automate the whole process.For Xenocell to generate new patches for previous versions and a new installer it only takes the click of a button now - based on an SVN revision. And it happens in the cloud too, on a dedicated server started only for the duration of generating the patch (meaning that costs "mount" to somewhat less that $2 (!) a month if we do one patch a week). Then it deploys the patch to an Amazon S3 bucket and through that - using CloudFront - it appears everywhere around the world within about 15 minutes of pressing that button. Pretty convenient. :)
#16
06/09/2011 (1:36 pm)
Outstanding reasearch Konrad!
#17
06/09/2011 (1:38 pm)
Thanks, Novack! :) 
#18
06/09/2011 (1:55 pm)
Thank you konrad for sharing your reflexion and your experience. ;) 
#20
I wanted to use Molebox as well as I got a licence years ago. I also tried encrypted zip files but they were soooo sloooooow to load during game run (1 minute to load is indeed slow).
On the end I decided to leave all my files uncompressed and open. A very difficult decision but after looking at so many different games I found that lots and lots (even high end ones) have open files. Even Magic The Gathering has all its assets open.
In my case, packing plays against me as in Warscale I plan to release item after item in micro updates. If I start posting small new files for each new object everthing will become caothic really soon.
Maybe I will try again with unencrypted zips later. All the support code is just there so I still can try...
Very good reading!
06/09/2011 (3:17 pm)
Excellent post. I think I will ditch my SVN based updater and go for Puchisoft.I wanted to use Molebox as well as I got a licence years ago. I also tried encrypted zip files but they were soooo sloooooow to load during game run (1 minute to load is indeed slow).
On the end I decided to leave all my files uncompressed and open. A very difficult decision but after looking at so many different games I found that lots and lots (even high end ones) have open files. Even Magic The Gathering has all its assets open.
In my case, packing plays against me as in Warscale I plan to release item after item in micro updates. If I start posting small new files for each new object everthing will become caothic really soon.
Maybe I will try again with unencrypted zips later. All the support code is just there so I still can try...
Very good reading!
Associate Logan Foster
perPixel Studios