Downloading custom content files from the server
by Port · in Torque 3D Professional · 08/28/2014 (4:37 pm) · 7 replies
I've implemented a dynamic mod system into a project, wherein mods can, among other things, define datablocks using content files (models, textures, sounds, etc) that are included with the mod on the server only. What I'm trying to do is to have the server compute a list of required files before loading datablocks and send it to the client, which then figures out which are missing and requests to download them.
Looking around in the engine a little, there's bits and pieces of a downloading system scattered all over (netConnection.cpp, netDownload.cpp and gameConnection.cpp, for example). Everything seems to be missing exactly what it needs to work though, and it looks as if that system is designed to work during the datablock load phase, which is exactly too late.
Currently, joining a server with missing model files just spams the client console to death with messages like "Failed to create shape", etc. before crashing.
Could anybody assist with implementing and/or finishing the engine implantation of this? Preferably, I'd also like to try to refer to content files by CRC/MD5 rather than name, and keep them in a cache/ folder or something.
Looking around in the engine a little, there's bits and pieces of a downloading system scattered all over (netConnection.cpp, netDownload.cpp and gameConnection.cpp, for example). Everything seems to be missing exactly what it needs to work though, and it looks as if that system is designed to work during the datablock load phase, which is exactly too late.
Currently, joining a server with missing model files just spams the client console to death with messages like "Failed to create shape", etc. before crashing.
Could anybody assist with implementing and/or finishing the engine implantation of this? Preferably, I'd also like to try to refer to content files by CRC/MD5 rather than name, and keep them in a cache/ folder or something.
#2
However, .mis files will not be sent, as objects are just ghosted via the engine over object transmission. If you want clients to be able to host the mission file of that level or play it without joining the server, you will have to send the contents of the file by reading it in script and using commandToClient(). Warning, this is really slow and you can only send 255 characters at a time.
08/28/2014 (8:32 pm)
AFAIK, if the client does not possess the png/jpg files and the dts file, it will request to send it from the server to the client. That happened at least in older versions of the torque engine. I'm pretty confident it still exists in Torque3D.However, .mis files will not be sent, as objects are just ghosted via the engine over object transmission. If you want clients to be able to host the mission file of that level or play it without joining the server, you will have to send the contents of the file by reading it in script and using commandToClient(). Warning, this is really slow and you can only send 255 characters at a time.
#3
Eventually, due to not being able to figure out the crazy way terrain collision updating worked I decided to write a file transfer method instead. I made a class that handled the transferring by posting NetEvents but it was really crude and not that secure overall. In the end it was hilariously slow. For files under 100kb it might have been decent.
I ended up abandoning the whole idea in the end but I do still have all of the code if anyone is really interested in seeing it. I'd have to figure out which of my many backups has the proper stuff, though. I'm also not all that sure if it worked 100% or not since it was awhile ago.
08/28/2014 (9:48 pm)
Oh man, I spent so much time on this same damn thing. I was testing out procedurally generated terrains that clients downloaded in chunks which were comprised of TerrainCell height data. Clients generated a list of all nearby non-downloaded terrain cells they needed and issued a request to the server which grabbed the data and broke it up into chunks to transfer.Eventually, due to not being able to figure out the crazy way terrain collision updating worked I decided to write a file transfer method instead. I made a class that handled the transferring by posting NetEvents but it was really crude and not that secure overall. In the end it was hilariously slow. For files under 100kb it might have been decent.
I ended up abandoning the whole idea in the end but I do still have all of the code if anyone is really interested in seeing it. I'd have to figure out which of my many backups has the proper stuff, though. I'm also not all that sure if it worked 100% or not since it was awhile ago.
#4
I've seen TGE games do so previously (early versions of Blockland, for example), but all the most important pieces of downloading are missing from T3D. It seems like it'd be too late when loading datablocks, seeing as the game already starts complaining about missing files (a lot). You'd have to have some sort of two-stage preload system, where the first merely checks what's required.
Partially. For example:
Everything needed seems to be missing though - there's methods for requesting files from the mMissingFileList in order, but they never seem to be called. The list is supposedly meant to be populated in gameConnection.cpp:1490:
gResourceManager doesn't even exist, though, and nor is getMissingFileList defined for ResourceManager (which, by the way, seems like the right place to start implementing the cache directory).
08/29/2014 (12:22 am)
Quote:Daniel Buckmaster
During datablock loading is the right time to handle this, since no objects have been created yet, and the client will be able to scan datablocks for shape files that are missing and request them. I'm pretty sure Tribes and so on used to do this.
I've seen TGE games do so previously (early versions of Blockland, for example), but all the most important pieces of downloading are missing from T3D. It seems like it'd be too late when loading datablocks, seeing as the game already starts complaining about missing files (a lot). You'd have to have some sort of two-stage preload system, where the first merely checks what's required.
Quote:Jeff Hutchinson
I'm pretty confident it still exists in Torque3D.
Partially. For example:
- All the code in sim/netDownload.cpp
- sim/netConnection.cpp:934
//----------------------------------------------------------------
/// @name File transfer
/// @{
protected:
/// List of files missing for this connection.
///
/// The currently downloading file is always first in the list (ie, [0]).
Vector<char *> mMissingFileList;
/// Stream for currently uploading file (if any).
Stream *mCurrentDownloadingFile;
/// Storage for currently downloading file.
void *mCurrentFileBuffer;
/// Size of currently downloading file in bytes.
U32 mCurrentFileBufferSize;
/// Our position in the currently downloading file in bytes.
U32 mCurrentFileBufferOffset;
/// Number of files we have downloaded.
U32 mNumDownloadedFiles;
/// Error storage for file transfers.
String mLastFileErrorBuffer;Everything needed seems to be missing though - there's methods for requesting files from the mMissingFileList in order, but they never seem to be called. The list is supposedly meant to be populated in gameConnection.cpp:1490:
// object failed to load, let's see if it had any missing files if(isLocalConnection() /*|| !gResourceManager->getMissingFileList(mMissingFileList)*/)
gResourceManager doesn't even exist, though, and nor is getMissingFileList defined for ResourceManager (which, by the way, seems like the right place to start implementing the cache directory).
#5
Player with models sees:
Player without models sees:
At least it prevents crashes.
08/29/2014 (1:02 am)
I've implemented a temporary solution that'll allow me to continue working, but I'd still rather include some form of content downloading system.Player with models sees:
Player without models sees:
At least it prevents crashes.
#6
I thought about offering the file transfer system as as a separate product but there was no interest at the time.
If I remember right, the code actually supported bi-directional file transfers, i.e. from server->client and client->server.
If there was enough demand for the functionality I could rip it out and offer it as a cheap pack.
Vince Gee
winterleafentertainment.com
08/29/2014 (4:58 am)
OneWorld Source has this functionality where it will sync the game folder with the server (Everything but shaders) before you connect to the server.I thought about offering the file transfer system as as a separate product but there was no interest at the time.
If I remember right, the code actually supported bi-directional file transfers, i.e. from server->client and client->server.
If there was enough demand for the functionality I could rip it out and offer it as a cheap pack.
Vince Gee
winterleafentertainment.com
#7
09/08/2014 (3:18 am)
@Vince: Rip It ... Package It ... Sell It ... 8-} 
Torque Owner Daniel Buckmaster
T3D Steering Committee
At the moment, all I've done is made a note to investigate this in the future, but my queue of stuff to do is pretty big :(.