Torque3D x32 up to 4GB

#1

Quote:
Can anybody explain me why is disabled the Linker option which let's handle more that 2GB by the application (/LARGEADDRESSAWARE:NO). I'm talking about a x32 build of course. There is a technical reason for doing that?

Because a lot of legacy code assumes negative pointers are invalid. Some play around with the MSB and this can cause issues.

1. If you're manipulating pointers as numbers, you need to accomodate for differences that are larger than a signed 32-bit type or you'll get overflow.
2. GetCursorPos fails, so you'll have to use something else, like GetCursorInfo.

Quote:
If you set this enabled the application can use up to 4GB, which is really good. Since e.g, It let me use a 8192x8192 base texture for the terrain.

LARGEADDRESSAWARE lets you allocate more than 2GB, not up to 4GB as some is reserved by Windows. It doesn't automatically let you use 8192x8192 as your GPU has to support the size too, but it does let you allocate more memory.

#2 Thank you very much Stefan. Now I'm aware of possible problems if I do this. Very helpful!

I have read that typically old 3rd party code has bugs with 32 bit pointers >2GB because using 31 bit pointers, leaving the most significant for any other use. But, I'm running my program under 64 bit Windows and I have read that forcing the system to use top-down memory allocation via a registry setting that seams to deal with that problem. I must say that I haven't modify the registry and my application has not experienced any problem... yet. All library used in my project seems to work fine with LARGEADDRESSAWARE.

The problem with texture is that having the GPU support for that texture size, the application crash when allocating memory (malloc return 0) for it.

Although, I have to say that, having this on count, I would'n recommend for any body using that "trick". We are in the area of serious games (simulation), and we have the option to choose the hardware and software platform where it runs our application.

stackoverflow.com/questions/3052028/detect-pointer-arithmetics-because-of-largea...
msdn.microsoft.com/en-us/library/windows/desktop/bb613473(v=vs.85).aspx

#3 Any time!

Yeah, it seems some people manipulate that last bit and rely on it for their code.

If I really had to worry about this, I would compile for 64-bit or use another process which allocates more memory - like some browsers do.

If malloc fails then there's no current page available that fits 268MB. This could be caused by memory fragmentation, though.

How much memory does your computer have? 268MB is a lot.

#4 My computer has 16GB and windows 7 x64.

The crash happen in gBitmap.cpp [Ln334] when loading the 8192x8192 terrain base texture.

mBits    = new U8[mByteSize];

mByteSize is 201326592

Sure it should be a fragmentation issue, because in the moment of crash the application is using 750 MB and we only beg for 192 MB more...

Compile for 64-bit is not an option now because there is a lot of work changing a many .dll to x64. Using another process which allocates more memory is something we should study. Thanks!

#5 IPC (shared memory) is a good way to do that, but it won't help if the issue you're running into is fragmentation.

It could be fragmented by other processes, too.

What happens if you lower the byte size by half? Does malloc return non-NULL?

#6 Thinking about it, if the problem is only a fragmentation issue, shouldn't be happening even if LARGEADDRESSAWARE flag is enabled?

With LARGEADDRESSAWARE enabled, I could load up to 6 different textures of 8192px on GuiBitmapCtrl until crash, using up to 2.5 GB. You are right, 1.5 GB must be reserved by Windows and other stuffs ...

With LARGEADDRESSAWARE disable, I can load just one 8192px texture. The second load crashes, using up to 1GB.

I know that using a 8192px texture make the engine crash for everyone... Is not my particular case.

#7 I *think* it should happen regardless, yes. It's a good point.

I wish I could help you, this is an interesting issue.

#8 8192x8192x4 (RGBA) bytes is 256Mbytes. There may be other data on top of that getting loaded for each image.

What is the end goal of loading such a large terrain? Are you just trying to push boundaries here?

#9 It seems that when loading terrain texture base it uses GFXFormatR8G8B8, which is right because makes no sense loading the alpha channel in here...
Base Texture 4096px
Base Texture 8192px

#10 I salute you for trying to push to 8192x textures, when i put up the 1 liner to allow 4096x textures, some people didnt seem to get why it would be wanted... but the clarity on distance and midrange detail when treating the diffuse that way is so much better than when using standard diffuse textures tiled, gives a much more realistic and flowing terrain.

#11 What I am trying to understand is this:
Does this only affect the development of the game? Or is this going to increase the memory requirements at game runtime? I assume the terrain file that is saved does not have to load a huge amount of data?

I am more interested in the end user than the niceties of development. If it does not affect the end user requirements then this is great. Otherwise this may be high end feature?

I am not disparaging the effort (I think this is awesome). Just trying to understand the end results.

#12 Volumetric clouds? Nice :)

#13 @Andy
I think this is the way to go, at least in simulation. We even have 16384px textures ready, but not until Torque3D x64 gets more proven. (including by ourself).

@Demo
The end result is to give the end user (Spanish Army... don't worry we are in NATO ;)) the possibility of creating their own terrains for their simulators, but with best possible quality. This not only relay on base texture of course, we are creating a software that uses several input layers: DTED, orthophotos, Shapefiles. Shapefiles give us the information of where are vegetation, rivers, etc. Then it loads those files and programmatically outputs a Toque3D level. The end user requirements are taking on count by us, because we supply the hardware with our simulators....

@Daniel
Yeah, there are SunDog SilverLining clouds!. They are great for hiding aircrafts from ground-air missiles ;). We have plans to release a resource with a class that interfaces Torque3D with SilverLining that replaces the ScatterSky functionality.

#14

Quote:
It seems that when loading terrain texture base it uses GFXFormatR8G8B8

It will be padded on your GPU anyway, to 32-bit per pixel.

Now that I think of it, you might be using 2x the memory. I think a copy of the texture is saved for device resets. That might explain it.