Streaming large worlds

#1
The terrain is not streamed from disk. Atlas did that but is not a part of Torque anymore.

Likewise, there is no background-streaming of assets ATM.

PS: There are *far more* implications and requirements to building huge terrains and worlds than having an engine that dynamically pages level data. In fact, I think the issues coming up on the content creation side are far more difficult to solve (effort, tools, etc).

#2 Yes you are right. My idea was to split up the world into "chunks" but load/unload them on demand based on visiblity/distance/available memory.

This would allow to work on chunks pretty much in isolation, and stitch them together later.

Of course the coordinate system issues are problematic, but can largly be avoided if you do not go overboard with world size ( 10x10 km was probably already an exaggeration on my part ) and doing transformations relative to the viewer position then doing them in full world space...

Of course the ZBuffer precision issues are also to be thought off, i already know that i have to dig into the Torque Render Pipeline to support ZBuffer "Slices" but it should be doable ( i did it in my own engine, but switched to Torque for the tooling and other benefits, i can't do in my own time )

#3 This is a sensitive topic for me at the moment, as we need it aswell for our game project (thought isometric/orthographic game) so it can be done a bit more simply then for a 3D perspective game.

Most game engines these days are starting to add seemless infinity world scalability (most does it by adding streaming terrain), and torque is the only one removing this function :P

While yes, its more simple not to have it, being more simple isnt always better.

As the dudes at CryEngine 3 says : If they say it cant be done, do it anyways and prove them wrong.

Simply put, this engine (in my humble oppinion) really needs streaming terrain, and put together with something alike Umbra's occlusion culling techology (UE3 partner, see the reel) and some other middleware technology (self made at GG ofcourse, something similar), multicore rendering, and this engine would rox the world.

Dont get me wrong, T3D is a good engine, but it can be pushed alot harder to be a even good'er (is that even a word :P) engine.

EDIT : Looking in wireframe mode inside T3D, there isnt even occlusion culling inside T3D ?

It (to me) seems to render the intire map, visible or not.

#4 I agree with Bo, wholeheartedly. People always want big terrains. The debate over whether they really need them is endless....they want them, regardless.

I was very surprised that they got rid of the huge terrain capability of Atlas. I have heard people bitching and moaning about Terrain size since 1.4. It seemed like GG finally responded to those cries with Atlas. Now, I know Atlas had problems, but the current terrain solution, (regardless of the ease of adding a terrain) seems like a step back, at least for what I want to do. I don't know enough about the technical problems to understand why terrain was dealt with the way it was, but it is not good for me.

Will we see some added terrain functionality in the RPG content pack? I hope so.

In the meanwhile, if we can't get one big terrain, I would be happy with a well documented, step by step explanation of how to create zones, or any other solution whereby I can create large terrains with what we have now.

#5 The LoD system works well to some degree, the problem i personally have is the fact that seemingly the terrain is loaded and rendered regardless of whether you can see it or not, even the cell based LoD is distance based not visual.

I know we are still in early beta, which is why i comment rather than complain, the improvements from beta1 to beta2 have been significant as far as terrain performance is concerned.

The other problem is you can have massive terrains or detailed terrains, dependong on your system the 4096x4096 terrain provides a reasonably sized and detailed terrain 1m squares (but see above for issues with this) or a massive terrain 8m squares (over 1000km sq).

a point to note is that the LoD system is much more effective with larger square sizes, so if you dont need a highly detailed terrain the huge terrains work quite well, or at least this is what my experimentations have shown.

I dont know how difficult paging is.. didnt TGE/TGEA use paging even for legacy terrain? or how difficult it would be to add, this is not remotely close to anything i personally have experience with, bu, while i can probably manage most projects with this kind of terrain either paging or some kind of seamless zoning or preloading zones would be very useful

#6 On a side note, on the Umbra's occlusion tech, it seems to give each object its own occlusion box (probably on loading), and make terrain into occlusion boxes aswell (automatic) into same sized boxes.

It then cuts off everything behind (near clipping), and infront of camera (far clipping), and then calculates (realtime) if a occlusion object/terrain zones in camera sight is blocked by other objects/zone data.

Well, its of course just a guess based on this image :)

#7 i'm guessing since there no price tag on the site anywahere that umbra is prohibitively overpriced technology for indie developers. of course all the $100k+ engine have it built in...

*scribbles numbers on his lottery ticket and calls 1-800-dialadeity for luck*

as a slight side note, have you noticed how torque terrain actually culls the back side of mountains you are close too so that you can see better the insane number of polys in the terrain cells you shouldnt be able to see at all? wireframe ftw :)

#8 @Bloodknight

Im pretty sure you wont even get access to the technology unless you have ALOT of cash + a good engine behind you, and even then it will take you ages probably and tons of paperwork (beside the 20-100k$ it most likely costs).

No engine has it "build in" as far as i understand (ofcourse some have there own tech), Umbra is a addon for UT3 ect engines.

Also Umbra uses GPU acceleration (shader system i guess ?) and has multicore support.

And yeah, the terrain in wireframe mode in T3D is concerning, and alot of FPS in still to be had from what i can gather in terrain system alone, but in beta, im just happy to play around with T3D :)

EDIT:
Some more on Umbra (GDC 2009) quite awesome, really wish torque had something even remotely similar :).

#9 Before terrains could be paged, you'd need asynchronous resource loading. That's not trivial to add, and would require changes in countless areas of the engine, since everything assumes the resources are always there.

#10 Atlas was unmaintainable and had caused issues since it was included. The better solution was to support multiple terrain blocks. These blocks could probably be modified to load and unload. Realistically you will probably only need 4 blocks in memory at any one time, and you could create impostors for further blocks (if your view-distance was that high).

Another reason streaming terrain was cut was because, as terrain only, it was not useful. It needed to be possible to build huge mesh structures, of any kind, into an auto-lod format. 'Atlas' had a clipmap solution for managing the terrain textures, but mesh shapes need that functionality as well.

This tech comes in the form of the Ubermesh (still in research), and a more generalized virtual texture solution called Sparse Virtual Textures (this is similar to iD's Megatexture). These solutions have the advantage of working for all cases, instead of just terrain. If an artist wants to make a huge, subterranian cave system in Max/Maya...they need the same advantages that Atlas provided to terrain, without the limitations of Atlas.

@Manoel: In Rokkitball we actually had threaded texture re-coloring (it used the same 'Promise' system that I describe in this blog entry). It is possible, but it was a hacky solution. It did, however, make obvious what needed to be refactored in order to support this behavior on a larger scale.

#11 Megatextureing really isn't SVT its closer to clipmaps. SVT is a software version of what the old Permedia 3 did in hardware.
Tis good stuff, but has itself some issues (just too late texturing, anistropy and cache overflow issues).

For 2D world streaming, I'd go with a block loader, which should be fairly easy and more easy to control. Real on demand streamers are nightmares when it comes to memory issues, and given we have to support 32bit OS's memory fragmentation is a real issue we have to deal with in the next few years...