Zip Objects
by DragonSix · in Torque Game Builder · 04/02/2007 (9:05 am) · 26 replies
After looking a little into the documentation (which promise to be really good from the look of it), I found something called Zip Object with function to read, write, compress and extracts files within a .zip. Thats really promising for further .zip functions.
I have two question about Zip Object :
Does it support reading image and sound through it ? (or will do ?)
Does it support passworded archives ? (or will do ?)
I have two question about Zip Object :
Does it support reading image and sound through it ? (or will do ?)
Does it support passworded archives ? (or will do ?)
#2
04/02/2007 (10:30 am)
Thats why I have interest into Zip Object : I don't own a pro license. 
#3
04/02/2007 (10:31 am)
You can use zip archives without a pro license, but the encryption resource for passwords requires source modifications. 
#4
Either with the 1.5 ZipObject, or with the resource David mentioned -- which, btw, do you have a link to this resource available?
04/02/2007 (10:55 am)
With the new ZipObject in 1.5 -- can we use it to create a VFS and store all of our data in this VFS ... then read images and level files from the VFS, without extracting them (even temporarily) to the users workstation?Either with the 1.5 ZipObject, or with the resource David mentioned -- which, btw, do you have a link to this resource available?
#5
To clear up any confusion...
I completely rewrote the zip code for 1.5. Existing zip related resources won't work as the only thing left from the old code is ZibSubStream.
The new zip code's main features are:
* Reading of zip files from both the resource manager and your own C++ or script code
* Writing of zip files from your own C++ or script code
* Support for many compression formats (only Deflate in the shipping version)
* Support for many encryption algorithms (only standard Zip2.0 shipping version, see below for caveats)
* Support for entire zip format, including zip comments and file comments (old zip code would break if you tried using zips with comments)
* It's also pretty extensible if it doesn't support something you need from the zip format. You'll need to hit C++ for that, though.
* Extensive unit tests
The main reasons behind writing the new zip code were to improve support for reading files and to add support for writing files. While I was at it I took the opportunity to massively clean it up - the Zip code is now pretty much it's own self contained library in it's own namespace. The only affects on core Torque were to integrate it with the resource manager and to add the script classes.
Support for encryption (aka passworded zips) is technically only available to Pro users. However, it is also in binary builds, you just can't change the password. The reason for this is the password is set in a #define in the C++ code (I forget where, it should be doc'd). Put simply, if you encrypt all your mod zips, then the only place left to store the password is either a plain text script file or the C++ code. Thus, it would be completely pointless as anyone could just open up the script and read the password and thus get at your zips.
The default password for a binary build is "password" (without the quotes, I may be misremembering, it should be doc'd) so you can try out the crypto. However, you will need a pro license if you want to change the password or add things like more secure encryption algorithms such as AES. Currently, encryption support should be thought of as a bonus, it was not the main reason behind writing the new code, and I added it mainly because I could.
Re: VFS support. There is something resembling VFS support in there. This was something that I added mainly for grins. It is probably not particularly useful as it stands, as most everything is ifdef'd out. The main purpose behind VFS support was running everything from a single .exe with no external files. Assuming nobody removed the code from main.cc, you can see an example of using the VFS from there. However, VFS related stuff is entirely undocumented and you are pretty much on your own for now if you choose to attempt to use it. Again, it was not the main purpose behind the new zip code.
I would suggest that you all read through the docs and play around with it. If you want script examples, you can find them in the packaging utility and handling of the core.zip in the level builder (I forget where, Justin wrote that code). If you want C++ examples, see the unit tests ... they cover everything except for VFS, I think. They may not cover encryption either.
A word regarding ZipObject, StreamObject and FileStreamObject. ZipObject is a wrapper around Zip::ZipArchive and reading the docs for ZipArchive will probably help you more then reading the docs for ZipObject. StreamObject is a script wrapper around the Stream class and can wrap any instance of a Stream derived class. FileStreamObject is a script wrapper around the FileStream class and is intended to replace FileObject. FileStreamObject should be significantly more reliable then script file io with FileObject, and can also be passed to ZipObject methods that take streams.
Note that StreamObject can only be instantiated in C++, so FileStreamObject is the only way you can create a stream in script. However, it would be incredibly simple to make a wrapper for MemoryStream or any other stream in Torque if it was needed to create them in script.
If you have any questions, feel free to ask. Just don't be offended if I don't reply for a while, I am horrendously busy on the next project and thus may be slow to answer.
T.
04/02/2007 (3:42 pm)
Hi,To clear up any confusion...
I completely rewrote the zip code for 1.5. Existing zip related resources won't work as the only thing left from the old code is ZibSubStream.
The new zip code's main features are:
* Reading of zip files from both the resource manager and your own C++ or script code
* Writing of zip files from your own C++ or script code
* Support for many compression formats (only Deflate in the shipping version)
* Support for many encryption algorithms (only standard Zip2.0 shipping version, see below for caveats)
* Support for entire zip format, including zip comments and file comments (old zip code would break if you tried using zips with comments)
* It's also pretty extensible if it doesn't support something you need from the zip format. You'll need to hit C++ for that, though.
* Extensive unit tests
The main reasons behind writing the new zip code were to improve support for reading files and to add support for writing files. While I was at it I took the opportunity to massively clean it up - the Zip code is now pretty much it's own self contained library in it's own namespace. The only affects on core Torque were to integrate it with the resource manager and to add the script classes.
Support for encryption (aka passworded zips) is technically only available to Pro users. However, it is also in binary builds, you just can't change the password. The reason for this is the password is set in a #define in the C++ code (I forget where, it should be doc'd). Put simply, if you encrypt all your mod zips, then the only place left to store the password is either a plain text script file or the C++ code. Thus, it would be completely pointless as anyone could just open up the script and read the password and thus get at your zips.
The default password for a binary build is "password" (without the quotes, I may be misremembering, it should be doc'd) so you can try out the crypto. However, you will need a pro license if you want to change the password or add things like more secure encryption algorithms such as AES. Currently, encryption support should be thought of as a bonus, it was not the main reason behind writing the new code, and I added it mainly because I could.
Re: VFS support. There is something resembling VFS support in there. This was something that I added mainly for grins. It is probably not particularly useful as it stands, as most everything is ifdef'd out. The main purpose behind VFS support was running everything from a single .exe with no external files. Assuming nobody removed the code from main.cc, you can see an example of using the VFS from there. However, VFS related stuff is entirely undocumented and you are pretty much on your own for now if you choose to attempt to use it. Again, it was not the main purpose behind the new zip code.
I would suggest that you all read through the docs and play around with it. If you want script examples, you can find them in the packaging utility and handling of the core.zip in the level builder (I forget where, Justin wrote that code). If you want C++ examples, see the unit tests ... they cover everything except for VFS, I think. They may not cover encryption either.
A word regarding ZipObject, StreamObject and FileStreamObject. ZipObject is a wrapper around Zip::ZipArchive and reading the docs for ZipArchive will probably help you more then reading the docs for ZipObject. StreamObject is a script wrapper around the Stream class and can wrap any instance of a Stream derived class. FileStreamObject is a script wrapper around the FileStream class and is intended to replace FileObject. FileStreamObject should be significantly more reliable then script file io with FileObject, and can also be passed to ZipObject methods that take streams.
Note that StreamObject can only be instantiated in C++, so FileStreamObject is the only way you can create a stream in script. However, it would be incredibly simple to make a wrapper for MemoryStream or any other stream in Torque if it was needed to create them in script.
If you have any questions, feel free to ask. Just don't be offended if I don't reply for a while, I am horrendously busy on the next project and thus may be slow to answer.
T.
#6
The point of me wanting zip encryption is mainly for being able to have minimal art assets protection. Can we open image and sound through this system ?
04/02/2007 (3:54 pm)
I don't need insane security, so I would be glad to being able to decide the .zip password with scripts (anyway it'll be shipped as .dso, it might not be hack proof, but it's enough for most users, they won't find it just by opening .dso with text editor).The point of me wanting zip encryption is mainly for being able to have minimal art assets protection. Can we open image and sound through this system ?
#7
If all you want is minimal protection and you don't want to get a Pro license, then just use the default password. Sure, it's not very secure, but it'll still probably stop most casual copying. If you want more security then that, then you'll currently need a Pro license.
T.
04/02/2007 (4:12 pm)
The new code supports mod zips in exactly the same way the previous code did. That is a feature provided by the resource manager. Thus, you can create any zip supported by the new zip code and use it as a mod zip. So, if images and sound in mod zips used to work, they should still do. At one point there was some TGB specific file formats that didn't work from zips. I don't know if that's still the case, but it's something to look out for.If all you want is minimal protection and you don't want to get a Pro license, then just use the default password. Sure, it's not very secure, but it'll still probably stop most casual copying. If you want more security then that, then you'll currently need a Pro license.
T.
#8
The mention of "mod zips" was made, does this mean I can make a "mod" (ie; common or even my 'MyGame' folder, as games are mods, not sure if thats really changed in 1.5 -- didn't pay attention to that part ;p) ...
Back to the question ... as I'm understanding, I can do a "loadMod" type functionality, and load all the mod resources out of a zip ... which includes levels, images, sounds, etc ... there by, creating a VFS ... just not storing it inside the Game.exe ?
I'll look into this a bit more ... I've got a use for it, I believe ... sounds great either way ...
04/02/2007 (7:41 pm)
@Tom, thanks for the reply -- cleared up a few questions, and created a few new ones ... ;)The mention of "mod zips" was made, does this mean I can make a "mod" (ie; common or even my 'MyGame' folder, as games are mods, not sure if thats really changed in 1.5 -- didn't pay attention to that part ;p) ...
Back to the question ... as I'm understanding, I can do a "loadMod" type functionality, and load all the mod resources out of a zip ... which includes levels, images, sounds, etc ... there by, creating a VFS ... just not storing it inside the Game.exe ?
I'll look into this a bit more ... I've got a use for it, I believe ... sounds great either way ...
#9
The term mod is basically a carry over from TGE where they actually are more mod like, but TGB uses them slightly differently. Whether they actually are "mods" in the usual sense of the word is purely determined by the scripts. That functionality hasn't changed, it just uses the new zip code instead of the old zip code if the mod happens to be zipped.
I think the other thing that is confusing is the use of the term VFS. I think what you are actually talking about is the mod zip thing which will work in stock Torque without having to worry about any of the issues that go with the VFS code (of which there are a number).
When I use the term VFS, I am specifically referring to the modified version of Torque that can use a single zip that contains multiple mods and the main.cs. That code is very much work in progress thing and has only been used once for an "isnt this cool, please ignore the caveats" demo since I merged it into TGB trunk, and once before that for a similar demo. It also probably does not have the ability to load the zip from anywhere other then a resource in the exe.
To put it into perspective, I have a very high degree of confidence in the new Zip code because it has been extremely well tested on Windows and OS X and I have stepped through all of the code in the debugger many times so I know for a fact it is doing what it is supposed to do. The only issues that I expect to turn up in that code now are the obscure issues we couldn't catch in testing, but always show up when other people start hammering on the code.
The VFS code is #ifdef'd out because it has been used precisely twice and is mostly a hack I did simply because I could, thus I do not have anywhere near as much confidence in the VFS build. All I can tell you relating to the VFS is it's worked twice as it is, and it might work again, but there are probably a good number of show stopping issues if you were to use it for production use.
Hope that clears it up a bit :)
T.
04/02/2007 (8:03 pm)
Technically, in Torque every directory in the same dir as the exe is a mod. When I say "mod zips" I mean zipping any of those directories. Refer to any resource/forums post/TDN article on zipping of mods for more info on that.The term mod is basically a carry over from TGE where they actually are more mod like, but TGB uses them slightly differently. Whether they actually are "mods" in the usual sense of the word is purely determined by the scripts. That functionality hasn't changed, it just uses the new zip code instead of the old zip code if the mod happens to be zipped.
I think the other thing that is confusing is the use of the term VFS. I think what you are actually talking about is the mod zip thing which will work in stock Torque without having to worry about any of the issues that go with the VFS code (of which there are a number).
When I use the term VFS, I am specifically referring to the modified version of Torque that can use a single zip that contains multiple mods and the main.cs. That code is very much work in progress thing and has only been used once for an "isnt this cool, please ignore the caveats" demo since I merged it into TGB trunk, and once before that for a similar demo. It also probably does not have the ability to load the zip from anywhere other then a resource in the exe.
To put it into perspective, I have a very high degree of confidence in the new Zip code because it has been extremely well tested on Windows and OS X and I have stepped through all of the code in the debugger many times so I know for a fact it is doing what it is supposed to do. The only issues that I expect to turn up in that code now are the obscure issues we couldn't catch in testing, but always show up when other people start hammering on the code.
The VFS code is #ifdef'd out because it has been used precisely twice and is mostly a hack I did simply because I could, thus I do not have anywhere near as much confidence in the VFS build. All I can tell you relating to the VFS is it's worked twice as it is, and it might work again, but there are probably a good number of show stopping issues if you were to use it for production use.
Hope that clears it up a bit :)
T.
#10
I'd much rather use the mod zip concept over the single exe concept, as most of the projects I'll be working on in the near future will be artwork heavy ... and in a lot of cases, the artwork is level specific ... so it'd be nice to be able to just load up a 'mod zip' and grab the resources I need for that level ... then unload it ...
04/02/2007 (8:27 pm)
Crystal ... thanks for the clarification ... and yeah, I use the term "VFS" fairly loosely ... just referring to a "virtual file system" of any means where multiple files are stored inside of a single file and accessed as if that file were a virtual drive of some sort ... such as referring to modName/data/levels/file.t2d in code, but it's actually stored in modName.zip ... I'd much rather use the mod zip concept over the single exe concept, as most of the projects I'll be working on in the near future will be artwork heavy ... and in a lot of cases, the artwork is level specific ... so it'd be nice to be able to just load up a 'mod zip' and grab the resources I need for that level ... then unload it ...
#11
Extremely nice information. I haven't had a chance to look at the new version, but now that my film fest is out of the way, I'm definitely digging in!
04/02/2007 (8:42 pm)
Holy hell Tom!Extremely nice information. I haven't had a chance to look at the new version, but now that my film fest is out of the way, I'm definitely digging in!
#12
T.
04/02/2007 (8:46 pm)
Since we are being crystal clear, I feel that I should point out that I am deliberately tempting fate in my previous post. I find that fate is an extremely useful thing to abuse when you want to find bugs in things ;)T.
#13
About passwords, I tried to put the password "password" (using winrar in zip mode) but it didn't work with it (worked like a charm without password). Maybe it's because I'm using winrar, is there a better tool for compressing encrypted zip ?
04/02/2007 (8:50 pm)
Thanks a lot Tom, it's really a neat feature !About passwords, I tried to put the password "password" (using winrar in zip mode) but it didn't work with it (worked like a charm without password). Maybe it's because I'm using winrar, is there a better tool for compressing encrypted zip ?
#14
T.
04/02/2007 (8:53 pm)
Ah, sorry, I just checked, the default password is "changeme" (minus quotes) ... at least that is slightly more secure then "password" since it's less likely to be the first guess someone tries :)T.
#15
I tried to zip the game folder and/or common folder but it does not work. TGB complains that it is not able to initialize the project. What am I doing wrong?
Thank you
06/07/2007 (1:20 am)
When I build a project with TGB 1.5 Beta 3 I get the following structure:Project - common - game .project.exe .glu2d3d.dll .main.cs .OpenAl32.dll .opengl2d3dl.dll .unicows.dll
I tried to zip the game folder and/or common folder but it does not work. TGB complains that it is not able to initialize the project. What am I doing wrong?
Thank you
#16
www.garagegames.com/mg/forums/result.thread.php?qt=64924
...but no one has responded so I'm going to try my luck in this thread.
I've searched my drive for the core.zip file Mr. Brampton mentioned, but it isn't there (I assume it only comes with the Pro License?). I've also searched this site, the documentation and the internet in general and have been unable to find any code samples or more detailed information regarding ZipObjects. I'm coming up completely empty.
I really need to get ZipObjects to work, as they're how I intend to package and distribute scenarios for my game. I'm stuck until I solve this. I realize everybody's probably still buzzing over the 1.5 release and has other issues on their minds, but I would be very grateful if someone who knows what they're doing would lend a hand.
07/29/2007 (8:46 pm)
I started a topic about not being able to get ZipObjects to work properly here:www.garagegames.com/mg/forums/result.thread.php?qt=64924
...but no one has responded so I'm going to try my luck in this thread.
I've searched my drive for the core.zip file Mr. Brampton mentioned, but it isn't there (I assume it only comes with the Pro License?). I've also searched this site, the documentation and the internet in general and have been unable to find any code samples or more detailed information regarding ZipObjects. I'm coming up completely empty.
I really need to get ZipObjects to work, as they're how I intend to package and distribute scenarios for my game. I'm stuck until I solve this. I realize everybody's probably still buzzing over the 1.5 release and has other issues on their minds, but I would be very grateful if someone who knows what they're doing would lend a hand.
#17
As for working with this stuff, I am almost certain you require a Pro license as it requires some form of modifications in the C++ and a clean rebuild of the engine (if for anything, at least to change the password from 'password' to something more secure -- if your intent is to store your files securely)
But again, I haven't looked into the support in 1.5, just remember briefly reading that it was "sort of there". I also don't recall any form of support for Zip's being available in TGB prior to 1.5, without using a C++ resource which requires a Pro license.
07/29/2007 (8:56 pm)
I haven't looked into Zip support in 1.5 yet, but I do know that someone was talking about how it was available ... but was not considered to be stable or something ... I don't even believe it was listed as a new feature in the changelog.As for working with this stuff, I am almost certain you require a Pro license as it requires some form of modifications in the C++ and a clean rebuild of the engine (if for anything, at least to change the password from 'password' to something more secure -- if your intent is to store your files securely)
But again, I haven't looked into the support in 1.5, just remember briefly reading that it was "sort of there". I also don't recall any form of support for Zip's being available in TGB prior to 1.5, without using a C++ resource which requires a Pro license.
#18
Still, there is some documentation (do a search for "ZipObject"). So it does seem like it was intended to go in.
07/29/2007 (9:07 pm)
I'm not worried so much about security. I'm worried about my game's editor being able to pack a bunch of different files into one place and distribute it. If what you say is true, it screws everything up for me. I was told that this feature was in 1.5 and was counting on it being usable. I can't afford to get the Pro license just to be able to use archives.Still, there is some documentation (do a search for "ZipObject"). So it does seem like it was intended to go in.
#19
I just read the thread you mentioned, skipped over it originally -- my apologies.
The "E:/water.png" that you are trying to put into the Zip is most likely inaccessible due to the fact that all the code in TBG prevents you from accessing files outside of the game directory itself -- to my knowledge. You can overcome this limitation by tweaking the C++ Source (Pro License), but by default it restricts file access to the game directory, which is a generic safe-guard.
Try adding a file from E:\MyGames\GameName\data\images\ ... using "/images/water.png" or "~/water.png" or "./images/water.png" ... etc, of course, choosing the appropriate 'expando'.
As for the core.zip, not sure if Tom was referring to something that really exists ... but this thread was also originally intended for private beta discussion and I believe was opened to the public when 1.5 was released (or the beta was made public).
Oh and, the discussion I said I remembered reading ... was apparently this thread ;)
07/29/2007 (11:11 pm)
Daniel,I just read the thread you mentioned, skipped over it originally -- my apologies.
The "E:/water.png" that you are trying to put into the Zip is most likely inaccessible due to the fact that all the code in TBG prevents you from accessing files outside of the game directory itself -- to my knowledge. You can overcome this limitation by tweaking the C++ Source (Pro License), but by default it restricts file access to the game directory, which is a generic safe-guard.
Try adding a file from E:\MyGames\GameName\data\images\ ... using "/images/water.png" or "~/water.png" or "./images/water.png" ... etc, of course, choosing the appropriate 'expando'.
As for the core.zip, not sure if Tom was referring to something that really exists ... but this thread was also originally intended for private beta discussion and I believe was opened to the public when 1.5 was released (or the beta was made public).
Oh and, the discussion I said I remembered reading ... was apparently this thread ;)
#20
...to no avail. I've also tried using "addFile("~/water.png", "~/")" as well. The result is always the same: a zip file is created, but nothing is inside it.
The water.png file is in the main folder for my game, in the same place that the zip file appears. So I think that's the right place.
07/30/2007 (2:01 pm)
Actually, the E: drive was a flash drive which didn't even have TGB installed on it. But I've also tried the following:function createTestArchive()
{
%archive = new ZipObject();
%archive.openArchive("testArchive.zip", Write);
%archive.addFile("./water.png", "./");
%archive.closeArchive();
}...to no avail. I've also tried using "addFile("~/water.png", "~/")" as well. The result is always the same: a zip file is created, but nothing is inside it.
The water.png file is in the main folder for my game, in the same place that the zip file appears. So I think that's the right place.
Associate David Montgomery-Blake
David MontgomeryBlake