This just in! Automatic mission downloading...
by Mark Frohnmayer · in Torque Game Engine · 04/01/2002 (10:30 pm) · 23 replies
Hey all,
Just did a checkin that allows the server to download mission specific data (terrains, interiors, textures, etc) to the client on connect.
For a long time we thought getting this to work in Torque was going to be a real pain, but it turned out to be easier, if hackier, than we thought... today's version only supports resource downloading of ghost object resources (ie - not datablocks), but tomorrow I'm going to try to get downloading of datablock resources in.
The basics of how it works... when the server ghosts the "Ghost Always" objects to the client - which includes the terrain and interiors, the client generates a list of resources that failed to load. It then sends this list back to the server and the server downloads files it has that the client doesn't have. It then restarts the ghost always phase and tries again. It is possible because of the hierarchy of file dependencies that the ghosting process will be reset multiple times.
One very lame thing about this process is that the search procedure for textures on the client generates a huge number of missing files, since for any texture it will search for base name, .jpg, .png, etc, down the full path to the shape. For this reason, the client will actually send up to 32 file names per event to the server, which the bitstream then compresses way down (deltas off the previous string) - it's a hack, but it works out well because of the way the bitstream compresses sequentially sent strings.
This code hasn't been well tested (just me on my machine), so if anyone wants to really bang on it, give it a try.
Just did a checkin that allows the server to download mission specific data (terrains, interiors, textures, etc) to the client on connect.
For a long time we thought getting this to work in Torque was going to be a real pain, but it turned out to be easier, if hackier, than we thought... today's version only supports resource downloading of ghost object resources (ie - not datablocks), but tomorrow I'm going to try to get downloading of datablock resources in.
The basics of how it works... when the server ghosts the "Ghost Always" objects to the client - which includes the terrain and interiors, the client generates a list of resources that failed to load. It then sends this list back to the server and the server downloads files it has that the client doesn't have. It then restarts the ghost always phase and tries again. It is possible because of the hierarchy of file dependencies that the ghosting process will be reset multiple times.
One very lame thing about this process is that the search procedure for textures on the client generates a huge number of missing files, since for any texture it will search for base name, .jpg, .png, etc, down the full path to the shape. For this reason, the client will actually send up to 32 file names per event to the server, which the bitstream then compresses way down (deltas off the previous string) - it's a hack, but it works out well because of the way the bitstream compresses sequentially sent strings.
This code hasn't been well tested (just me on my machine), so if anyone wants to really bang on it, give it a try.
About the author
#2
If this is an April Fools joke we will be forced to hunt you down like a rabid warthog with a bowl infection.
04/01/2002 (11:45 pm)
Hmm... looks at the date still being April 1st....If this is an April Fools joke we will be forced to hunt you down like a rabid warthog with a bowl infection.
#4
Anyway, it doesn't sound too farfetched, so if it's a joke it's not a good one!
04/02/2002 (1:41 am)
Hehe, one thing I've learned is that companies wouldn't do an April fools joke that might annoy customers. Especially GarageGames.Anyway, it doesn't sound too farfetched, so if it's a joke it's not a good one!
#5
I posted an update on the forums that my fluid rendering system had been updated on April 1st and they all fell for it ....
... it had bugs and all sorts of problems in the code, they never say that one coming .... :)
groan.
04/02/2002 (2:16 am)
Harold,I posted an update on the forums that my fluid rendering system had been updated on April 1st and they all fell for it ....
... it had bugs and all sorts of problems in the code, they never say that one coming .... :)
groan.
#6
torque\engine\interior\interiorInstance.cc(593) : error C2593: 'operator &&' is ambiguous
Error executing cl.exe.
The line in question was this:
if(mInteriorRes && mLMHandle != 0xFFFFFFFF)
Change the line to:
if(bool(mInteriorRes) && mLMHandle != 0xFFFFFFFF)
and interiorInstance.cc will compile fine.
But I get the following error:
netGhost.obj : error LNK2001: unresolved external symbol "public: void __thiscall NetConnection::sendNextFileDownloadRequest(void)" (?sendNextFileDownloadRequest@NetConnection@@QAEXXZ)
../example/torqueDemo.exe : fatal error LNK1120: 1 unresolved externals
Error executing link.exe.
OK that error is fixed by adding the netDownload.cc file to Torque Demo Files->Source Files->Sim folder.
This file isn't included in the Linux Make files either. You need to edit the torque/engine/targets.torque.mk file and add the following:
netDownload.cc \
in your SOURCE.SIM=\ section.
04/02/2002 (2:16 am)
OK got an error when compiling this:torque\engine\interior\interiorInstance.cc(593) : error C2593: 'operator &&' is ambiguous
Error executing cl.exe.
The line in question was this:
if(mInteriorRes && mLMHandle != 0xFFFFFFFF)
Change the line to:
if(bool(mInteriorRes) && mLMHandle != 0xFFFFFFFF)
and interiorInstance.cc will compile fine.
But I get the following error:
netGhost.obj : error LNK2001: unresolved external symbol "public: void __thiscall NetConnection::sendNextFileDownloadRequest(void)" (?sendNextFileDownloadRequest@NetConnection@@QAEXXZ)
../example/torqueDemo.exe : fatal error LNK1120: 1 unresolved externals
Error executing link.exe.
OK that error is fixed by adding the netDownload.cc file to Torque Demo Files->Source Files->Sim folder.
This file isn't included in the Linux Make files either. You need to edit the torque/engine/targets.torque.mk file and add the following:
netDownload.cc \
in your SOURCE.SIM=\ section.
#7
It fixed it for me.
Would it be a problem if we could default the project to use '/subsystem:windows' rather than 'console'?
Aww go on, yer know you want to.
04/02/2002 (3:42 am)
Thanks Harold,It fixed it for me.
Would it be a problem if we could default the project to use '/subsystem:windows' rather than 'console'?
Aww go on, yer know you want to.
#8
Yes, the server files get written directly to the same file location. It is possible that it would be better to write them to cache, but then every file load would have to check the cache on fail, which could be automated through the ReasourceManager... I'll leave that one as an exercise for the reader for now :)
Currently all writes go through the ResourceManager's openFileForWrite function, which automatically prevents writing outside the mod paths.
Not sure what you mean about date/time stamping... the client only downloads the file if it doesn't already exist, but it doesn't compare the file contents with the server's... unfortunately that behaviour is beyond the scope of this change, but it is definitely something we have planned for the next engine.
Also, after thinking through things a bit (and looking at getting the downloading working with datablocks) I'm going to redo how some of the downloading stuff is handled. Stay tuned!
04/02/2002 (9:48 am)
Melv,Yes, the server files get written directly to the same file location. It is possible that it would be better to write them to cache, but then every file load would have to check the cache on fail, which could be automated through the ReasourceManager... I'll leave that one as an exercise for the reader for now :)
Currently all writes go through the ResourceManager's openFileForWrite function, which automatically prevents writing outside the mod paths.
Not sure what you mean about date/time stamping... the client only downloads the file if it doesn't already exist, but it doesn't compare the file contents with the server's... unfortunately that behaviour is beyond the scope of this change, but it is definitely something we have planned for the next engine.
Also, after thinking through things a bit (and looking at getting the downloading working with datablocks) I'm going to redo how some of the downloading stuff is handled. Stay tuned!
#9
The bit about date/time stamp checking was related to the fact that files are only downloaded if they are missing on the client. If you modify or update files on the server, you would want these modifications to propogate as people connected. I understand this is not part of your update but perhaps an idea for the future?
I suppose a better way would be to have a version level on the client which the server checks. If below the current, only then would it proceed to download the latest version files followed by updating the client version. Centralised control, that's all.
You've paved the way for this kind of functionality.
Again, good stuff.
04/02/2002 (9:53 am)
Nice one Mark.The bit about date/time stamp checking was related to the fact that files are only downloaded if they are missing on the client. If you modify or update files on the server, you would want these modifications to propogate as people connected. I understand this is not part of your update but perhaps an idea for the future?
I suppose a better way would be to have a version level on the client which the server checks. If below the current, only then would it proceed to download the latest version files followed by updating the client version. Centralised control, that's all.
You've paved the way for this kind of functionality.
Again, good stuff.
#10
04/02/2002 (10:21 am)
What is the estimate on what the server load will be if/when this is fully implemented?
#12
04/02/2002 (12:20 pm)
So this is for real, right? Wasn't an April Fool (I hate April)? 
#15
Next time the client runs, no uploads.
04/02/2002 (12:45 pm)
Essentially, the remote client wouldn't need the files to run the mission, the server would upload them to the client.Next time the client runs, no uploads.
#16
04/02/2002 (12:53 pm)
oh ok, like q2 did,wel so did q3 but q3 sucked at it, so this is a very good thing , thanks
#17
I looked at the above fix's and they are already in the
code.
Couple of these warnings looks like something I may
of screwed up with my compiler. Unreferenced local
variable. Sigh..
VC6++ sp5.
----------------------------------------------------
c:\programming\main\torque\lib\ljpeg\jdatadst.c(121) : warning C4018: '!=' : signed/unsigned mismatch
c:\programming\main\torque\lib\zlib\trees.c(1110) : warning C4018: '<' : signed/unsigned mismatch
c:\programming\main\torque\lib\lungif\dgif_lib.c(941) : warning C4101: 'Error' : unreferenced local variable
c:\programming\main\torque\lib\lungif\dgif_lib.c(941) : warning C4101: 'i' : unreferenced local variable
c:\programming\main\torque\lib\lungif\dgif_lib.c(941) : warning C4101: 'j' : unreferenced local variable
egif_lib.c
c:\programming\main\torque\lib\lungif\egif_lib.c(226) : warning C4018: '!=' : signed/unsigned mismatch
c:\programming\main\torque\lib\lungif\egif_lib.c(580) : warning C4018: '!=' : signed/unsigned mismatch
c:\programming\main\torque\lib\lungif\egif_lib.c(705) : warning C4101: 'NewKey' : unreferenced local variable
c:\programming\main\torque\lib\lungif\egif_lib.c(704) : warning C4101: 'NewCode' : unreferenced local variable
c:\programming\main\torque\lib\lungif\egif_lib.c(809) : warning C4018: '!=' : signed/unsigned mismatch
c:\programming\main\torque\lib\lungif\egif_lib.c(825) : warning C4018: '!=' : signed/unsigned mismatch
Linking...
netGhost.obj : error LNK2001: unresolved external symbol "public: void __thiscall NetConnection::sendNextFileDownloadRequest(void)" (?sendNextFileDownloadRequest@NetConnection@@QAEXXZ)
../example/torqueDemo_DEBUG.exe : fatal error LNK1120: 1 unresolved externals
Error executing link.exe.
torqueDemo_DEBUG.exe - 2 error(s), 11 warning(s)
04/03/2002 (12:31 am)
I am getting compile errors with this latest head.I looked at the above fix's and they are already in the
code.
Couple of these warnings looks like something I may
of screwed up with my compiler. Unreferenced local
variable. Sigh..
VC6++ sp5.
----------------------------------------------------
c:\programming\main\torque\lib\ljpeg\jdatadst.c(121) : warning C4018: '!=' : signed/unsigned mismatch
c:\programming\main\torque\lib\zlib\trees.c(1110) : warning C4018: '<' : signed/unsigned mismatch
c:\programming\main\torque\lib\lungif\dgif_lib.c(941) : warning C4101: 'Error' : unreferenced local variable
c:\programming\main\torque\lib\lungif\dgif_lib.c(941) : warning C4101: 'i' : unreferenced local variable
c:\programming\main\torque\lib\lungif\dgif_lib.c(941) : warning C4101: 'j' : unreferenced local variable
egif_lib.c
c:\programming\main\torque\lib\lungif\egif_lib.c(226) : warning C4018: '!=' : signed/unsigned mismatch
c:\programming\main\torque\lib\lungif\egif_lib.c(580) : warning C4018: '!=' : signed/unsigned mismatch
c:\programming\main\torque\lib\lungif\egif_lib.c(705) : warning C4101: 'NewKey' : unreferenced local variable
c:\programming\main\torque\lib\lungif\egif_lib.c(704) : warning C4101: 'NewCode' : unreferenced local variable
c:\programming\main\torque\lib\lungif\egif_lib.c(809) : warning C4018: '!=' : signed/unsigned mismatch
c:\programming\main\torque\lib\lungif\egif_lib.c(825) : warning C4018: '!=' : signed/unsigned mismatch
Linking...
netGhost.obj : error LNK2001: unresolved external symbol "public: void __thiscall NetConnection::sendNextFileDownloadRequest(void)" (?sendNextFileDownloadRequest@NetConnection@@QAEXXZ)
../example/torqueDemo_DEBUG.exe : fatal error LNK1120: 1 unresolved externals
Error executing link.exe.
torqueDemo_DEBUG.exe - 2 error(s), 11 warning(s)
#18
I don't mean to sound sarcastic or anything it's just easy to browse and not read.
Have you added the new 'netDownload.cc' to your project. Not just have it, actually add it into the project?
Oh yeah, I've not examined your exact warnings but those look normal. In a way these should be removed if they are obsolete.
04/03/2002 (12:47 am)
Have you read the above correctly?I don't mean to sound sarcastic or anything it's just easy to browse and not read.
Have you added the new 'netDownload.cc' to your project. Not just have it, actually add it into the project?
Oh yeah, I've not examined your exact warnings but those look normal. In a way these should be removed if they are obsolete.
#19
I'll double check it Melv.
Randy...
04/03/2002 (8:43 am)
You would not sound sarcastic if you read my post where I said I read the fix and it appears to be in.I'll double check it Melv.
Randy...
#20
04/03/2002 (8:52 am)
Clean build? 
Associate Melv May
A few things ...
Do the server-piped files get written directly to the same file location (under the MOD) from which they reside on the server?
If they are written to client disk then would it be better to write them to a cache area on the client rather than directly in place? This cache area could be a special place that is always searched if a file is not found. This would stop unwanted file writes on the client and provide a place that you don't mind getting trashed. Also, can we restrict it to MOD directory and below for security reasons ... don't want kernel32.dll being overwritten!!
Could the files have their data/time stamp checked so that you always get the latest files version and not just a version?
Geez ... sounds like I'm never satisfied, I'll try this tonight Mark and thanks.
All the best.