Silent loading in background
by Yuri Dobronravin · in Torque 3D Professional · 05/21/2013 (8:33 am) · 5 replies
Hi guys!
I'm doing some research currently. The ultimate goal will be very big level (tens of kilometers) that consist of smaller chunks.
While player moving through the world some chunks will be loaded and some unloaded.
Each chunk is stored in simple mission file (shapes and part of terrain).
Loading chunks itself is trial task, just call from the script: exec('new_chunk.mis')
The problem that I facing is a big lag that stops the whole game process for a few seconds while new objects are created. Most of the time is consumed for the texture loading and material creation.
My approach is straight forward. I've created additional thread and try to load all stuff into the memory and only then create objects. However it appears to be a lot more complex than I thought it will be.
Maybe anyone already have tried something similar or have better ideas how to deal with the situation?
I'm doing some research currently. The ultimate goal will be very big level (tens of kilometers) that consist of smaller chunks.
While player moving through the world some chunks will be loaded and some unloaded.
Each chunk is stored in simple mission file (shapes and part of terrain).
Loading chunks itself is trial task, just call from the script: exec('new_chunk.mis')
The problem that I facing is a big lag that stops the whole game process for a few seconds while new objects are created. Most of the time is consumed for the texture loading and material creation.
My approach is straight forward. I've created additional thread and try to load all stuff into the memory and only then create objects. However it appears to be a lot more complex than I thought it will be.
Maybe anyone already have tried something similar or have better ideas how to deal with the situation?
#2
The materials/resources system is too complex in T3D. Everything uses different kind of manager, that are not thread safe. Seems like I don't have other choice but to run some kind of thread safe resource loader.
Won't be very hard I hope.
05/22/2013 (4:20 am)
@Stefan, thnx for the reply!The materials/resources system is too complex in T3D. Everything uses different kind of manager, that are not thread safe. Seems like I don't have other choice but to run some kind of thread safe resource loader.
Won't be very hard I hope.
#3
05/22/2013 (5:48 am)
Have you considered that you might not need a separate thread - just some way to break the work into smaller time slices? For example, a ticking object might load a single asset each tick. I'm not sure if loading a single asset is enough to cause a noticeable pause. Or if Torque's asset loading is fine-grained enough to allow you to do this... 
#4
I tried thread-safing everything around textures, but it was a pain to spot all the places and I wanted to avoid too much locking.
So I created a new manager which was updated every frame or so. It kept a number of workers which would load textures from disk and upload them to the GPU.
When textures were loaded in T3D materials, they would get the warning texture. Meanwhile, a request is dispatched to the manager we talked about earlier. It would assign a worker to the texture requested, load it and upload it to a texture object.
Then on each loop, the manager would swap the texture handle in the material with the one being loaded. This requires no locking on the material being loaded, but I think I had to tell D3D to be thread-safe, and to guard some logic in GFXTexHandle or whatever it was called. I think they're added to a list somewhere which has to be locked.
Can't remember all the details. But that was specific to textures only.
05/22/2013 (8:37 am)
Yeah that's my experience too. I tried thread-safing everything around textures, but it was a pain to spot all the places and I wanted to avoid too much locking.
So I created a new manager which was updated every frame or so. It kept a number of workers which would load textures from disk and upload them to the GPU.
When textures were loaded in T3D materials, they would get the warning texture. Meanwhile, a request is dispatched to the manager we talked about earlier. It would assign a worker to the texture requested, load it and upload it to a texture object.
Then on each loop, the manager would swap the texture handle in the material with the one being loaded. This requires no locking on the material being loaded, but I think I had to tell D3D to be thread-safe, and to guard some logic in GFXTexHandle or whatever it was called. I think they're added to a list somewhere which has to be locked.
Can't remember all the details. But that was specific to textures only.
Torque Owner Stefan Lundmark
You spend the most time with reading the drive and that is easily threaded. But then you have to copy the memory somewhere and/or parse it or upload it to the GPU. That is a bit more difficult to thread.
What are you having problems with in particular?