One Executable
by Chris Childs · in Torque Game Builder · 07/18/2005 (9:58 am) · 14 replies
Is there a utility to include all resources, compiled script files, and libraries into one executable? I did a search for executable to see if this topic is covered, but all I found was questions about people wanting to compile the Torque Engine, not make a single game executable.
I only have access to T2D, so I am limited on what threads I can view if anything exists in the TGE forums.
I only have access to T2D, so I am limited on what threads I can view if anything exists in the TGE forums.
About the author
#2
07/18/2005 (11:47 am)
I'm trying to remember a company that offered a packaging solution, but I believe it had issues running on-the-fly pieces of code.
#3
However, it is an interesting idea. And I've been toying around with the concept/design of a virtual file system myself.
07/18/2005 (12:19 pm)
Such a thing would have to be built into the T2D engine, not just done as a post-process utility. Mostly because the engine expects to load files from disc, not load them from the executable. A virtual file system would need to be built into the executable, thus allowing it to load files from within the executable while still allowing for development with disc-based files. Obviously, a mere utility packaging program cannot accomplish such a thing.However, it is an interesting idea. And I've been toying around with the concept/design of a virtual file system myself.
#4
If you wanted to do it 'easy' (not using compiled scripts) then you could easily embed the script as a resource, then at runtime load it up and pass it to exec, (If you look at the exec code, it's very easy to make a version of it that takes in script instead of a filename)
i'm not a C++ dev, so i dont know how exactly you do it, but i'm pretty sure it can be done (it is done easily in C#)
An improvement on the process i mentioned would be to compile the script first.
an improvement on that would be to also compress the dso resource, and/or encrypt it.
07/18/2005 (12:30 pm)
I dont think it's teribly difficult.If you wanted to do it 'easy' (not using compiled scripts) then you could easily embed the script as a resource, then at runtime load it up and pass it to exec, (If you look at the exec code, it's very easy to make a version of it that takes in script instead of a filename)
i'm not a C++ dev, so i dont know how exactly you do it, but i'm pretty sure it can be done (it is done easily in C#)
An improvement on the process i mentioned would be to compile the script first.
an improvement on that would be to also compress the dso resource, and/or encrypt it.
#5
07/18/2005 (12:31 pm)
That seems to be basically what it is (I believe). Much like dmg's under os x which are virtual mounts, it creates a virtual sandbox to play in. It has been about five months or so since I saw it mentioned, but I believe a GG employee mentioned it because the company had a booth at GDC 2004, I believe. When I checked their site, they didn't seem to have problems with things like log-files and such being generated, but it was with running scripting languages that would then effect some of the other core files. I don't remember the details as I believe it was a windows only solution. It was targeting game portals, though, I believe at the GDC.
#7
07/18/2005 (2:55 pm)
I believe that was the one I was thinking of an making up broad generalizations about since it didn't exist in my memory as anything except a very, very obscure impression.
#8
I'm surprised such a thing exists. Or could exist at all on Windows. I wonder what kind of trickery they used to pull that off.
07/18/2005 (2:57 pm)
Quote:There's Molebox. Dunno if it will work with T2D or not.
I'm surprised such a thing exists. Or could exist at all on Windows. I wonder what kind of trickery they used to pull that off.
#9
It's pretty cool stuff. I have a similar technology i wrote for .NET stuff, but this looks to be more versitle.
07/18/2005 (3:08 pm)
It probably works by intercepting win32 calls to the filesystem by the 'packed' files, and if they reference another packed file, load up their file into memory and route the call accordingly.It's pretty cool stuff. I have a similar technology i wrote for .NET stuff, but this looks to be more versitle.
#10
Wouldn't that require, however, for your executable to be linking to Win32 dynamically rather than statically? Unless you're talking about being able to intercept any and all filesystem calls by an application and fielding them yourself.
The other thing about this approach is that you can't work around it. For example, if you have a game that you want to make moddable, but you don't want them to be able to directly touch the game's standard data. Once behind this shell thing, it seems that it is impossible to actually read from the drive anymore.
07/18/2005 (5:37 pm)
Quote:It probably works by intercepting win32 calls to the filesystem by the 'packed' files, and if they reference another packed file, load up their file into memory and route the call accordingly.
Wouldn't that require, however, for your executable to be linking to Win32 dynamically rather than statically? Unless you're talking about being able to intercept any and all filesystem calls by an application and fielding them yourself.
The other thing about this approach is that you can't work around it. For example, if you have a game that you want to make moddable, but you don't want them to be able to directly touch the game's standard data. Once behind this shell thing, it seems that it is impossible to actually read from the drive anymore.
#11
Example... WinZIP installers, One executable, or are they? WinZIP simply compresses all of the installer files then extracts them into C:\Windows\Temp (I beleive ;D). It then runs Setup.exe, which uses .cab and .msi files. Full proof of this is WinRar's ability to extract some installers.
I see two possible solutions:
#1: Use the winzip way... Or even winzip itself, to create an archive that selfextracts before running...
OR
#2: Create a complex RAM disk style interface, wheras all the files *are* stored, just in memory... ;) You could even have T2D come up as a removable drive.... Hehehe, I get so sidetracked.
Then for mods, you could re-program T2D to look in My Documents\GameName or /home/user/gamename (linux), as well. This could also be used for saving games, neat huh?
@Smaug:
As you see, I completely disagree... It could, and would be easiest if, it was done by another software.
Feel free to correct me ;).
07/19/2005 (12:51 pm)
OK, there is a very good answer to this one....Example... WinZIP installers, One executable, or are they? WinZIP simply compresses all of the installer files then extracts them into C:\Windows\Temp (I beleive ;D). It then runs Setup.exe, which uses .cab and .msi files. Full proof of this is WinRar's ability to extract some installers.
I see two possible solutions:
#1: Use the winzip way... Or even winzip itself, to create an archive that selfextracts before running...
OR
#2: Create a complex RAM disk style interface, wheras all the files *are* stored, just in memory... ;) You could even have T2D come up as a removable drive.... Hehehe, I get so sidetracked.
Then for mods, you could re-program T2D to look in My Documents\GameName or /home/user/gamename (linux), as well. This could also be used for saving games, neat huh?
@Smaug:
As you see, I completely disagree... It could, and would be easiest if, it was done by another software.
Feel free to correct me ;).
#12
So for example, I want to call "myStuff.dll" from myProgram.exe. Well when myProgram.exe tries to load myStuff.dll, the CLR tries to find and load myStuff.dll but cant, so then instead of failing, it calls my function that programatically decrypts and loads myStuff.dll.
I have also seen programs that intercept calls to platform API's, which may be how MoleBox does it. You could always download it and find out :)
@Ricky: Loading into a ramdrive would be nice if you can keep it inaccessable to users. In windows, maybe you can set permissions to a security context made specifically for the software, and disallow others. While this isnt fool-proof, it would make things harder for hackers.
Though, the future of computing (in the windows space) is LUA. No, not the scripting language, but "Least-privileged User Accounts". This breaks all sorts of software, including most tricky deployment and software protection schemes.
07/19/2005 (2:48 pm)
@Smaug: again, I am not a C++ dev, so I dont know how they do it, but with .NET you can add your code to run when an application tries to load an assembly and fails.So for example, I want to call "myStuff.dll" from myProgram.exe. Well when myProgram.exe tries to load myStuff.dll, the CLR tries to find and load myStuff.dll but cant, so then instead of failing, it calls my function that programatically decrypts and loads myStuff.dll.
I have also seen programs that intercept calls to platform API's, which may be how MoleBox does it. You could always download it and find out :)
@Ricky: Loading into a ramdrive would be nice if you can keep it inaccessable to users. In windows, maybe you can set permissions to a security context made specifically for the software, and disallow others. While this isnt fool-proof, it would make things harder for hackers.
Though, the future of computing (in the windows space) is LUA. No, not the scripting language, but "Least-privileged User Accounts". This breaks all sorts of software, including most tricky deployment and software protection schemes.
#13
Some things currently cannot be read from ZIP files in T2D, such as particle effects or maps, but one who has a firm grasp on file I/O should be able to modify the C++ code to work around this. (Just be aware this would break future T2D compatibility, so be prepared to merge in your changes over on top of future updated code.)
The only exception would be reading OGGs from ZIPs, which is currently impossible due to the vorbis file reader used by T2D requiring seek, and random access isn't supported on ZIP streams. (Found this out the hard way when streaming oggs would quit only after playing for a few seconds when played from an ZIP archive.)
T2D itself would still expect the bootstrap main.cs in the EXE's root directory, but this is easy to take care of. A simple but effective way is to just include the script code as a hard-coded string.
The whole ZIP+EXE deal is basically how I used to do single-EXE games back in the DOS days. First part is the EXE, rest of binary is some kind of packed/packaged format. You could get away with this because DOS only loaded into RAM the number of bytes that was listed in the EXE's header. Anything after that was ignored by the OS and could be freely seeked/manipulated by the program. I haven't even looked under the hood about how PE's work, but I'm pretty sure it's similiar if not the same.
Consider this, though: As Smaug points out, single-EXE programs are hard to modify or patch. So if version 1.1 comes out, the user has to download the entire program again... Either that, or you have to make and maintain some kind of binary diff and use a patcher.
07/21/2005 (3:49 am)
Since T2D is quite happy to use just standard file/memory streams as a source of the files -- It should be pretty easy to do. The fastest way to get this working would be to append a ZIP file to the executable, and then modify the ZIP stream reader to include the EXE offset before it did a physical file seek.Some things currently cannot be read from ZIP files in T2D, such as particle effects or maps, but one who has a firm grasp on file I/O should be able to modify the C++ code to work around this. (Just be aware this would break future T2D compatibility, so be prepared to merge in your changes over on top of future updated code.)
The only exception would be reading OGGs from ZIPs, which is currently impossible due to the vorbis file reader used by T2D requiring seek, and random access isn't supported on ZIP streams. (Found this out the hard way when streaming oggs would quit only after playing for a few seconds when played from an ZIP archive.)
T2D itself would still expect the bootstrap main.cs in the EXE's root directory, but this is easy to take care of. A simple but effective way is to just include the script code as a hard-coded string.
The whole ZIP+EXE deal is basically how I used to do single-EXE games back in the DOS days. First part is the EXE, rest of binary is some kind of packed/packaged format. You could get away with this because DOS only loaded into RAM the number of bytes that was listed in the EXE's header. Anything after that was ignored by the OS and could be freely seeked/manipulated by the program. I haven't even looked under the hood about how PE's work, but I'm pretty sure it's similiar if not the same.
Consider this, though: As Smaug points out, single-EXE programs are hard to modify or patch. So if version 1.1 comes out, the user has to download the entire program again... Either that, or you have to make and maintain some kind of binary diff and use a patcher.
#14
I have not heard of the SelfExe plugin and wanted to know if that is something that is a Torque plugin, a Brain Editor plugin, or just an installer packaging tool that does not really allow the game to run from a single executable.
I would not mind converting the game engine to not using scripts, but as it was already mentioned, I would then have to roll in any updates manually.
@FenrirWolf: I use to have some good specification documents for the Portable Executable file format. There is a web site that has a list of all the different file formats; unfortunately, the link for that site was on my work computer at a previous job. But I do remember that PE is a modified version of COFF.
07/21/2005 (7:59 am)
I am a software engineer, so converting the game engine to not use scripts and embedding all the resources into dynamically loaded libraries would not be a big deal for me to do. The only reason why I had asked if there was a way to pack all the files into one executable is because the people at Twinno (who made the Brain Editor) claim that "When you have compiled your game and have all the files, it's possible using the SelfExe plugin"I have not heard of the SelfExe plugin and wanted to know if that is something that is a Torque plugin, a Brain Editor plugin, or just an installer packaging tool that does not really allow the game to run from a single executable.
I would not mind converting the game engine to not using scripts, but as it was already mentioned, I would then have to roll in any updates manually.
@FenrirWolf: I use to have some good specification documents for the Portable Executable file format. There is a web site that has a list of all the different file formats; unfortunately, the link for that site was on my work computer at a previous job. But I do remember that PE is a modified version of COFF.
Torque Owner James Johnson