Improved Zip Support for Torque 3D
by Tim Newell · 11/01/2012 (3:25 pm) · 28 comments
This is a small modification to the Torque 3D 1.2 zip support that lets you lay out your file structure inside the zip file the same way you would on disk. Only 2 changes are required for this.
1. in platform/platformVolume.cpp in the function bool MountZips(const String &root) you should see:
just change that true to false:
That would be the only change but there is a bug that stops fonts and possible other stuff from working correctly out of a zip file. To fix this you need to do step 2.
2. in platformWin32/winFileio.cpp in the function bool Platform::isFile(const char *pFilePath) in the section:
change that to:
You'll need one more change in this file. At the top add the header:
UPDATE: Found another bug with Platform::compareModifiedTimes which is used by imposter generation as well as the terrain
3. In the file platform/platform.cpp in the function S32 Platform::compareModifiedTimes( const char *firstPath, const char* secondPath ) you will see the code:
replace this with:
You also need to add the include at the top of platform/platform.cpp
Note: This change is assuming that the files its comparing are both in a zip or both on disk. So it may not work for half and half.
That should be it. Now you can package all your assets into zip files.
Note: This only fixes windows, but you can make the same changes in the Platform::isFile implementation in the other platform layers as well.
1. in platform/platformVolume.cpp in the function bool MountZips(const String &root) you should see:
mounted += (S32)Mount(root, new ZipFileSystem(zipfile, true));
just change that true to false:
mounted += (S32)Mount(root, new ZipFileSystem(zipfile, false));
That would be the only change but there is a bug that stops fonts and possible other stuff from working correctly out of a zip file. To fix this you need to do step 2.
2. in platformWin32/winFileio.cpp in the function bool Platform::isFile(const char *pFilePath) in the section:
if (handle == INVALID_HANDLE_VALUE) {
return false;
}change that to:
if (handle == INVALID_HANDLE_VALUE) {
//check in zip files too since it was not found on disk
return Torque::FS::IsFile(pFilePath);
}You'll need one more change in this file. At the top add the header:
#include "core/volume.h";
UPDATE: Found another bug with Platform::compareModifiedTimes which is used by imposter generation as well as the terrain
3. In the file platform/platform.cpp in the function S32 Platform::compareModifiedTimes( const char *firstPath, const char* secondPath ) you will see the code:
if ( !getFileTimes( firstPath, NULL, &firstModTime) )
return -1;replace this with:
if ( !getFileTimes( firstPath, NULL, &firstModTime) ) {
//The reason we failed to get the file time could be because it is in a zip file, lets check.
return Torque::FS::CompareModifiedTimes(firstPath, secondPath);
}You also need to add the include at the top of platform/platform.cpp
#include "core/volume.h";
Note: This change is assuming that the files its comparing are both in a zip or both on disk. So it may not work for half and half.
That should be it. Now you can package all your assets into zip files.
Note: This only fixes windows, but you can make the same changes in the Platform::isFile implementation in the other platform layers as well.
#2
11/01/2012 (5:46 pm)
Does this support passwords for protecting assets? Probably not fool proof as you can get the password out of the exec, but it would prevent casual snooping.
#3
11/01/2012 (6:57 pm)
I am not sure if the built in zip code supports passwords. All this small change does it make the way you setup your assets with zips less wonky. 
#4
@Tim: Quick glitch in your resource, change isFIle to isFile (note the capital I)
@Frank: I'll write up a resource for you that encrypts your ZIP using AES and then decrypts it on load. That should accomplish what you want to do!
11/02/2012 (9:41 am)
Thanks a ton for this Tim. I've been looking for a while to get .zip working to deploy my assets in a much more effective way.@Tim: Quick glitch in your resource, change isFIle to isFile (note the capital I)
@Frank: I'll write up a resource for you that encrypts your ZIP using AES and then decrypts it on load. That should accomplish what you want to do!
#5
Would this be the same way the zip format does encryption? That way you could use an external utility to zip files up as needed.
Edit:
I just looked at the source and it looks like there is support for encrypted zips. It has a default password called "changeme" in the source code:
So maybe it is already there.
11/02/2012 (1:46 pm)
@Robert,Would this be the same way the zip format does encryption? That way you could use an external utility to zip files up as needed.
Edit:
I just looked at the source and it looks like there is support for encrypted zips. It has a default password called "changeme" in the source code:
// in zipArchive.h #define DEFAULT_ZIP_PASSWORD "changeme"
So maybe it is already there.
#6
11/02/2012 (1:48 pm)
@Robert: Thanks, just updated it. 
#7
a note if your going to use the zip password, try make your passwords long, don't use recognisable words, use lots of random characters, this makes it harder to crack and more difficult for hackers to find.
@Robert, I'm interested in the .Zip added encription layer as well :-)
11/02/2012 (6:15 pm)
@Tim, Nice. I'll have to have a play around with this, always wondered why Torque's zip support was a bit odd.a note if your going to use the zip password, try make your passwords long, don't use recognisable words, use lots of random characters, this makes it harder to crack and more difficult for hackers to find.
@Robert, I'm interested in the .Zip added encription layer as well :-)
#8
Go go Robert!
11/02/2012 (7:17 pm)
I should have poked around more in that same file:/// <h3>Encrypted Files</h3> /// /// Preliminary support for Encrypted/Passworded files is included in TGB Pro only. /// Currently, only Zip 2.0 encryption is supported by the stock code. AES support /// exists and may be released as a resource in the future. /// /// To set the password used for zips, you need to modify the #DEFAULT_ZIP_PASSWORD /// define in core/zip/zipArchive.h. This password will be used for all zips that /// require a password. The default password is changeme. This may be used by /// TGB Binary users to test encrypted zips with their game. Shipping with the /// default password is not recommended for obvious reasons. /// /// The intended use of encrypted zips is for preventing casual copying of your /// game's assets. Zip 2.0 encryption has known weaknesses that allow an attacker /// to decrypt the contents of the zip. AES encryption is significantly more secure, /// but as the password must be stored in the executable it will not stop a /// determined attacker. /// /// A script accessible mechanism for setting the password does not currently exist. /// To use encrypted mod zips, if the password was in script then the password /// would be clearly visible to anyone that cared to poke around in your scripts. /// /// Encrypted zip support will be improved in a future version. For now, a more /// secure method of storing the password is left as an exercise for the reader. ///
Go go Robert!
#9
removing that code will cause the imposters to load properly out of a zip. My guess is the compareModifiedTimes needs to be modified to work with zip files. I'll update my resource when I have a proper fix for it.
11/02/2012 (8:39 pm)
Just a heads up, I found another bug with zip support. If you use something that uses imposters like the forest tool, the imposters will fail to load from a zip file. The code preventing it is in ts/tsLastDetail.cpp in the function void TSLastDetail::update(bool forceUpdate) . the Code:if (Platform::compareModifiedTimes(diffuseMapPath, shapeFile ) < 0 )
{
Con::errorf( "TSLastDetail::update - Failed to create imposters for '%s'!", mCachePath.c_str());
return;
}removing that code will cause the imposters to load properly out of a zip. My guess is the compareModifiedTimes needs to be modified to work with zip files. I'll update my resource when I have a proper fix for it.
#10
11/02/2012 (9:22 pm)
OK I have a fix for compareModifiedTimes posted above now as #3. Like I noted above it really only works if both files are in zips or both files are on disk.
#11
Once I have that done, we should be golden.
11/03/2012 (7:54 am)
I'll try to have this done sometime later today, I just need to add a quick rewrite to mountZips that can detect the AES encrypted zips and properly decrypt them before loading it in.Once I have that done, we should be golden.
#12
11/03/2012 (9:33 am)
Found another bug in zip support this morning. It seems to have issues streaming ogg vorbis files from zip files. I havent found an issue/fix yet, will post when I do.
#13
11/04/2012 (1:15 am)
@Robert, maybe we could ask GG if we can get that feature ported from T2D to T3D. This is one reason I wish there was more parity between the code bases. T2D has things T3D does not, and T3D has things T2D does not when it comes to the core codebases. At least now T3D things can be put into T2D without any issues. The reverse is not true however.
#14
11/04/2012 (1:09 pm)
Yeah sorry guys, I've been really busy this weekend, with grad. school applications to worry about, fixing problems over emails and other school work coming up, this has been a busy time for me, so I might not be able to get to this for a few days. I'll keep you up to date when it's done though.
#15
Maybe when its all working we can ask GG to add it to the MIT build?
11/08/2012 (2:09 pm)
@Frank & Robert, isn't adding this feature to T3D what Robert is doing?Maybe when its all working we can ask GG to add it to the MIT build?
#16
I was hoping we could get permission from GG to use the existing encryption code from T2D. That way it does not have to rewritten. If Robert already has it figured out then never mind.
11/08/2012 (3:21 pm)
@David,I was hoping we could get permission from GG to use the existing encryption code from T2D. That way it does not have to rewritten. If Robert already has it figured out then never mind.
#17
11/26/2012 (10:13 am)
I would be like can encrypt my 3d models files (ultimate unwrap3d can read .dae and torque .dts) zlib is a opensource library and will be can used for commercial apps as a commercial game (www.zlib.net) is posible use zlib for this, and if this is posible, how? thanks.
#18
11/30/2012 (2:22 pm)
I hate to say it but I don't think you can encrypt ZIP files. I tried to accomplish this earlier, but it only results in corruption of the ZIP file. Sorry folks, looks like we'll need to try something else.
#19
12/01/2012 (3:21 am)
I don't understand this. The ZIP file format supports encryption as part of the format. Or is the library used to open zip files just not support that?
#20
12/01/2012 (7:38 am)
I ran the zip through my AES File Encryption code and it handed back a smaller file (considerably smaller). Trying to decrypt it only corrupted the archive which prevented it from being loaded. 
Associate Steve Acaster
[YorkshireRifles.com]
Having previously to use the actual zip as a named folder in an address was kinda wacky.