TLK and big interiors

#1 It's relighting the scene, that's going to take a while with large interiors. To generate some solid load/relight stats try this:

-Make sure you're running a release build (this is absolutely critical)
-Before testing each machine delete all *.dso and *.ml files
-First run: test relight time (this first test requires a full mission relight, so you'll see how long lighting takes on this machine)
-Second run: test straight load time (this second test doesn't require a relight, so you'll see how long loading the mission takes)

Also make sure to keep accurate accounts of the times, memory usage, and machine specs (especially the CPU and memory info).

-John

#2 John, I'm making .ml files by myself on server, and then sending it with game updates. I'm distributing precompiled .DSO files only. All computers use the same build and all files are the same.
And yeah.. I'm running release build.
Actualy I have already working alpha version of game, with proper update system, etc.
The thing is, that on one computer load (lightScene()) takes just a few seconds, while on another it takes minute or more.
It's NOT relighting the scene as the .ML file is already there and prepered (tested) on my computer.
some stats:
P4, 3GHz, 1GB RAM, GeForce FX 6800GT 256 - total loading time appx 25 seconds
P4, 3GHz, 1GB RAM, GeForce FX 6600 256 - total loading time appx 38 seconds
P4, 3GHz, 1GB RAM, Radion 9600 256 - total loading time appx 1 minute 40 seconds
P4, 3GHz, 1GB RAM, Radion 9800PRO 256 - total loading time appx 29 seconds
P3, 1.3GHz, 512 RAM, Radeon 8500 64 - total loading time apprx 55 seconds
P4, 2.6GHz, 768 RAM, GeForce FX 6600 128 - total loading time appx 1 minute 35 seconds
P4, 2.6GHz, 512 RAM, NVIDIA GeForce FX 5200 128 - load takes too long, so the connection times out
P2, 450MHz, 768 RAM, GeForce3 Ti 200 64 - total loading time appx 1 minute 55 seconds
and more.. I have timings and dxdiag.txt files from many computers and still can't see what can be a problem.

#3 Whats going on in the console log during the slow runs, and are the same machines always slow?

#4 There is nothing in console log except normal lines:

...
*** Phase 3: Mission Lighting
 Successfully loaded mission lighting file: 'world/data/missions/online_3270e905.ml'
Mission lighting done
Mapping string: MissionStart to index: 14
...

and yes, same machines always slow.

#5 I'd suggest that you turn on profiling, which should give you at least a feel for where the specific issues lie. For what it's worth, that is a huge interior, and Torque itself isn't optimized for missions that large. My first glance instinct is that it is probably something to do with purging the textures to the video card during the light:

InteriorLMManager::purgeGLTextures()

which is called from

gInteriorLMManager.purgeGLTextures();

in

SceneLighting::lightScene

Note: the comment above the purgeGLTextures() line seems out of place, in that is says:

// dont want to delete the current textures because bitmaps may already be loaded

but we do it anyway?

#6 Definitely compile in the profiler, but makes sure to keep running the release version, so your results are accurate. You might want to add the script code to enable the profiler right before the call to lightScene and the code to dump the profiler info right after. That way you can directly compare the results between the fast and slow machines.

#7 Yeah, I'll compile now a build with enabled profiler and do some test.. i'll keep informing on progress.

#8 Blah... no luck yet... looks like something wrong in the code with PROFILE_START/END combination somewhere.. when I run on these part I get:
Ordered by stack trace total time -
% Time % NSTime Invoke # Name
1.#IO 1.#IO 0 ROOT
Ending profile session...
... I've reverted all my START/END's back (removed) and still get this.. huh..
will try to track all of the PROFILE_* sets in the code and find what's wrong

#9 So far, I've tracked down the following:
if computer have less or equal 512MB of FREE memory - then the TGE, upon performing "SceneLighting::lightScene", loads HUGE amount of virtual memory.. and when the game finished loading (you came into 3d), the TGE freeing the memory it took.. so, if you have fast and defragmented HDD - windows page file works fast, the game loads normal. If you have enough RAM - the game loads extremely fast ... if not - takes 1-2-3 minutes..

is there any easy way of determing where START/END combinating can be broken in sources of engine?
I'm doing manual scan of all instances of PROFILE_* and checking ... huh... good luck to me ;)

#10 Strange... I've tracked all instances of PROFILE_* ... all looks fine, all opened profiles are closed.. but I have this as a dump:

*** Phase 3: Mission Lighting
+++ PROFILER ENABLED +++
 Successfully loaded mission lighting file: 'world/data/missions/prepare_df5f16a0.ml'
Mission lighting done
Mapping string: MissionStartPhase3Ack to index: 2
Profiler Data Dump:
Ordered by non-sub total time -
%NSTime  % Time  Invoke #  Name
 -1.#IO  -1.#IO        0 UpdateActionThread
 -1.#IO  -1.#IO        0 Player_UpdatePos
 -1.#IO  -1.#IO        0 Player_PhysicsSection
 -1.#IO  -1.#IO        0 Player_ProcessTick
 -1.#IO  -1.#IO        0 ContainerCastRay
 -1.#IO  -1.#IO        0 SwapBuffers
 -1.#IO  -1.#IO        0 DrawText
 -1.#IO  -1.#IO        0 CanvasRenderControls
 -1.#IO  -1.#IO        0 glFinish
 -1.#IO  -1.#IO        0 CanvasPreRender
 -1.#IO  -1.#IO        0 RenderFrame
 -1.#IO  -1.#IO        0 ClientNetProcess
 -1.#IO  -1.#IO        0 AdvanceClientTime
 -1.#IO  -1.#IO        0 ClientProcess
 -1.#IO  -1.#IO        0 SimAdvanceTime
 -1.#IO  -1.#IO        0 ServerNetProcess
 -1.#IO  -1.#IO        0 AdvanceObjects
 -1.#IO  -1.#IO        0 AdvanceServerTime
 -1.#IO  -1.#IO        0 ServerProcess
 -1.#IO  -1.#IO        0 ProcessTimeEvent
 -1.#IO  -1.#IO        0 TimeManagerProcessMain
 -1.#IO  -1.#IO        0 TelDebuggerProcessMain
 -1.#IO  -1.#IO        0 TelconsoleProcessMain
 -1.#IO  -1.#IO        0 ProcessInputEvent
 -1.#IO  -1.#IO        0 PlatformProcessMain
 -1.#IO  -1.#IO        0 NetProcessMain
 -1.#IO  -1.#IO        0 JournalMain
 -1.#IO  -1.#IO        0 MainLoop
 -1.#IO  -1.#IO        0 NewFontGetCharInfo
 -1.#IO  -1.#IO        0 oneUTF32toUTF8
 -1.#IO  -1.#IO        0 oneUTF16toUTF32
 -1.#IO  -1.#IO        0 convertUTF16toUTF8
 -1.#IO  -1.#IO        0 oneUTF8toUTF32
 -1.#IO  -1.#IO        0 convertUTF8toUTF16
 -1.#IO  -1.#IO        0 SceneGraph_rezoneObject
 -1.#IO  -1.#IO        0 MemoryRealloc
 -1.#IO  -1.#IO        0 MemoryFree
 -1.#IO  -1.#IO        0 MemoryAlloc

Ordered by stack trace total time -
% Time  % NSTime  Invoke #  Name
  1.#IO   1.#IO        0 ROOT
+++ PROFILER DISABLED +++
Mapping string: MissionStart to index: 11

any ideas? or should I post this case to SDK Bugs?

#11 I just tested the profiler here and it works fines, so it shouldn't be TLK or TGE (btw I tested TLK 1.4 - is that what you're using?).

Try installing a new copy of TLK and running the code right out of the box. If the problem is not in the fresh TLK build then diff the two to see what may be different (thats where the problem should be).

Regarding loading the lighting info, yes that uses a lot of memory. Torque loads all of the light maps into memory, and then uploads them to the card. This is not the optimal way to handle loading the light maps, but it hasn't been addressed yet. Generally it's only noticeable in very large worlds with many interiors or detailed interiors with very high lighting detail.

-John

#12 Yeah, I'm on TLK 1.4.
I'll try to run now some tests on clean TGE 1.4 and TLK 1.4 and then do some diffs..

btw, this "big" interior object located under the ground, and there is a tunnel made to get there..
Additionaly, in my DIF file, there is no any details maps.
In total I have only 6 introrior objects in mission.
first is a "gates" to underground, second is a tunnel, and third is this "big" interior. Both, tunnel and "dungeon" is not visible for "sun".
The other three interoir objects are not big.

Do I need to set anything (worldspawn in .map?) special to prevent calculating shadows/lightmaps for my interior?

#13 Whoops, sorry if that was a little ambiguous - I meant it's noticeable on interiors with a lot of geometry detail (lots of surfaces), not detail maps.

You can reduce the lighting scale by increasing the worldspawn property 'light_geometry_scale' (it needs to be a power of two for map2dif). This reduces the amount of memory used. You can also reduce the light map border size in map2dif (found in lmapPacker.h) to a smaller size - this helps a lot too.

#14 Okay.. thanks John and Stephen for hints..

i've tried to profile_* stock TLK and TGE - no luck.. not sure what I do wrong.. I've set
#define TORQUE_ENABLE_PROFILER
in torqueConfig.h and recompiled.
If I set profiler for some other steps - it works fine, but when I try to use it with lightScene() - i get no results..

But so far we found a temporary way of handling that..
Our artist using torque pipeline, so setting "light_geometry_scale" in worldspawn property works for our case.
He is playing around now with different values of both "light_geometry_scale" and "light_border_width". So far with our old huge interior we get 35% loading time less then it was before.
So with this - I think we can live.

But anyway, I'm still looking forward to make PROFILERs working with the lightScene part... any ideas?

#15 Ok that makes sense, it seems the profiler only starts collecting data the next time Torque makes it out to its main processing loop, so when dump is called the profiler is enabled, but empty. To get an accurate reading you'll probably need to enable the profiler when Torque starts up.

#16 Huh.. If I set profiler at start-up, then I'll lost in the amount of data in report ;)
but yeah, that's a way to get at least some stats.. I'll try to move profilerEnable(1/0) to other places in scripts to get data..

thanks again.

#17 Based on some of your hints:

Quote:We have made a quite big interior (appx 50 zones and 3.5k surfaces).

Quote:
if computer have less or equal 512MB of FREE memory - then the TGE, upon performing "SceneLighting::lightScene", loads HUGE amount of virtual memory.. and when the game finished loading (you came into 3d), the TGE freeing the memory it took.. so, if you have fast and defragmented HDD - windows page file works fast, the game loads normal. If you have enough RAM - the game loads extremely fast ... if not - takes 1-2-3 minutes..

And on John's post:

Quote:Regarding loading the lighting info, yes that uses a lot of memory. Torque loads all of the light maps into memory, and then uploads them to the card. This is not the optimal way to handle loading the light maps, but it hasn't been addressed yet. Generally it's only noticeable in very large worlds with many interiors or detailed interiors with very high lighting detail.

My 2 cents: no need to profile anything, the problem is obvious. Ever opened the task managet and saw how much RAM your game is using after loading the mission? I bet it's a massive amount. The game takes longer to load because the data cannot fit the physical RAM and is being paged in the virtual memory. Probably Windows is also expanding the virtual memory too.

In my experience with Torque, it became obvious that the lightmaps use way too much system RAM. A mere 2MBs+ .ml file can easily eat 200MBs+ of system memory. A dozen interiors exported with default lightmap settings is enough to eat 100+ megs. So far, all single-mission games we shipped had lightmaps turned off for distribution, since the RAM required to use the lightmaps was way beyond our targeted audience.

I haven't studied the code behind the lightmap storage/loading yet to suggest any way to improve this behavior (the .ml file is too small compared to the RAM it eats after loaded... there might be room for optimization). The possible solution/workaround I can think of is using many smaller missions and working on the mission loading code to make the loading faster (this is very possible) or even transparent.

I think the lightmap packing code can be modified to allow the lightmaps to be sav/loaded per interior, in a folder (using the CRC of each object), allowing interiors to be loaded/unloaded from "sub mission files" without having to destroy and re-create the server. But I'm not sure if I'll ever come to do such thing, because we are moving away from DIFs and implementing a mesh-based "interior" format for our TSE build. We'll still make TGE games, but only the budget ones, and those can live with vertex lighting or a tad of loading between areas.

#18 The 2M ml lighting files are compressed, these expand out to 200+M. The fastest ways to reduce the light map size is to shrink the border size or increase the lighting scale (details mentioned above).

Right now Synapse Gaming is working on a few undisclosed games that make heavy use of lighting (using TLK of course! :), these all use very minimal memory for lighting. Just like all art assets, lighting needs to be tuned for your game - done right it can be very tight.

-John

#19 Agree with John. fine-tuning needed.. it's the "must".
After our artists played with the parameters mentioned above, we reduced start-up of our "base" mission from 1 minute and 50 second to 45 seconds (on PC 1.3ghz with 512 ram and 64mb video).. and it works fine.. FPS not less then 45-50 in areas where we have lots of shape/fxgrass and foliage replicators together with some interriors.
but having separate light-map files for each interior and load/unload it per need sounds interesting.. but i'm not that good in c++ and "lighting" stuff to try to do it myself ;)

#20 I would love to get the lightmaps exporter to human readable image files someday, so I could look at the bastards and have a real idea of how much RAM they're eating. Right now they are mysterious magically compressed files that will eat more or less RAM or leak more or less light when you tweak border size and lighting scale values through pure trial and error. There's no indication of how big the lightmaps are, until you see Windows trying to use the virtual memory.

That's why I'm working into importing and rendering Gile[s] files into TSE. With that I can have a better control of how many lightmaps the scene will use, and how big each one of them is. Not an easy task, but 100% possible.