Exceeded maximum number of datablocks with Forest Editor
by Kevin Mitchell · in Torque 3D Professional · 02/06/2011 (2:16 am) · 23 replies
So I just converted 3 packs into brushes. And now I get this error. Should these items even be considered datablocks since they are not used in the scene yet? And not every board will have every element that's in the brushes. I have trees flowers, rocks, etc in there though they are needed for various boards shouldn't this been datablock by usage instead of datablock by define?
About the author
Riding Solo since 2005. Current Project: Fated World 2005-Present RPG Engine Tool Kit - Now available.
#5
Go into console/sim.h and modify DataBlockObjectIdBitSize.
By default, IDs (which are 32bit) are set up to use 10 bits for datablock IDs, 6 bits for message IDs, and 16 bits for objects IDs resulting in a maximum of 1024 datablocks, 64 messages, and 65536 objects.
02/06/2011 (1:58 pm)
Go into console/sim.h and modify DataBlockObjectIdBitSize.
By default, IDs (which are 32bit) are set up to use 10 bits for datablock IDs, 6 bits for message IDs, and 16 bits for objects IDs resulting in a maximum of 1024 datablocks, 64 messages, and 65536 objects.
#6
In all case that's a interesting information.
Does the ids are reusable? I talk more about object than datablock. Exemple I create a array, delete it
Does this id might be reused?
1024 datablock, it can be reached quickly! does that mean If we want more, we should setup id on 64 bits ? May that be a big issue?
Thanks!
02/06/2011 (2:21 pm)
What are the message that you count in this?In all case that's a interesting information.
Does the ids are reusable? I talk more about object than datablock. Exemple I create a array, delete it
Does this id might be reused?
1024 datablock, it can be reached quickly! does that mean If we want more, we should setup id on 64 bits ? May that be a big issue?
Thanks!
#7
They are used by the Message system (util/messaging), a somewhat obscure system that does have its uses, though.
That's the problem. Datablock IDs always get reused since the namespace is flushed each time the server goes down. However, there's no reuse and not even error checking on the ID allocation. If you let it run for long enough, it will at one point have exhausted the object ID namespace and start producing errors as the ID allocation routine stupidly dishes out invalid or datablock-reserved IDs to normal objects.
The way the whole thing was architected initially, there were never meant to be that many datablocks because these are set up to transfer from scratch for each established connection. However, by now, the entire setup here has grown into one big legacy issue and is next to collapsing under the strains of what has been heaped on top of it.
As for increasing the ID namespace size, it makes sense to first try to stay within its limits. One way to do so is to more selectively load datablocks for a game.
02/06/2011 (2:39 pm)
Quote:What are the message that you count in this?
They are used by the Message system (util/messaging), a somewhat obscure system that does have its uses, though.
Quote:Does the ids are reusable? I talk more about object than datablock. Exemple I create a array, delete it
Does this id might be reused?
That's the problem. Datablock IDs always get reused since the namespace is flushed each time the server goes down. However, there's no reuse and not even error checking on the ID allocation. If you let it run for long enough, it will at one point have exhausted the object ID namespace and start producing errors as the ID allocation routine stupidly dishes out invalid or datablock-reserved IDs to normal objects.
Quote:1024 datablock, it can be reached quickly! does that mean If we want more, we should setup id on 64 bits ? May that be a big issue?
The way the whole thing was architected initially, there were never meant to be that many datablocks because these are set up to transfer from scratch for each established connection. However, by now, the entire setup here has grown into one big legacy issue and is next to collapsing under the strains of what has been heaped on top of it.
As for increasing the ID namespace size, it makes sense to first try to stay within its limits. One way to do so is to more selectively load datablocks for a game.
#8
02/06/2011 (3:21 pm)
My brush count is 1389 object using Sticks & Twigs and Flower Power, FPS Environment Pack, Gold Rush Mine Pack , Ancient Empires:Persia Content Pack.
#9
02/06/2011 (3:23 pm)
And some of the Midevil Packs from dexsoft
#10
But when your just assigning datablocks for assets for the forest kit object?
NPCS / Animated objects / other kinds of smart objects. ow can u stay under 1024 for an entire game?
02/06/2011 (3:26 pm)
Quote:As for increasing the ID namespace size, it makes sense to first try to stay within its limits. One way to do so is to more selectively load datablocks for a game.
But when your just assigning datablocks for assets for the forest kit object?
NPCS / Animated objects / other kinds of smart objects. ow can u stay under 1024 for an entire game?
#11
I'd recommend turning only those assets into forest items that you actually use in the game rather than batch importing all 1389 objects.
This is just a limit for datablocks, not for objects in general. Usually, one doesn't need a whole lot of datablocks.
02/06/2011 (3:34 pm)
Quote:But when your just assigning datablocks for assets for the forest kit object?
I'd recommend turning only those assets into forest items that you actually use in the game rather than batch importing all 1389 objects.
Quote:NPCS / Animated objects / other kinds of smart objects. ow can u stay under 1024 for an entire game?
This is just a limit for datablocks, not for objects in general. Usually, one doesn't need a whole lot of datablocks.
#12
find
change it to 20
I had this same issue a while back, did this, no more errors, havent come up with any issues, yet......
02/06/2011 (3:36 pm)
hacky hacky hackedy hack:find
DataBlockObjectIdBitSize = 10,in sim.h
change it to 20
I had this same issue a while back, did this, no more errors, havent come up with any issues, yet......
#13
That's not a good change. You're limiting regular object IDs to 6 bits which is just 64 IDs. Even a regular engine startup with exhaust that space and let the object IDs run into the datablock range.
One thing you can do is to change the SimObjectId type to U64. However, there's a lot of places that use U32 instead of SimObjectId so these have to manually sync'd.
02/06/2011 (3:42 pm)
Quote:change it to 20
That's not a good change. You're limiting regular object IDs to 6 bits which is just 64 IDs. Even a regular engine startup with exhaust that space and let the object IDs run into the datablock range.
One thing you can do is to change the SimObjectId type to U64. However, there's a lot of places that use U32 instead of SimObjectId so these have to manually sync'd.
#14
Out of curiosity (I'm away from my dev computer for the next two weeks) - what do the forest object datablocks contain? Could some of it be moved into a multi-shape datablock, or read from the shape files themselves?
02/06/2011 (9:45 pm)
Seems like a case of too finely-grained datablock usage? Might it be beneficial to refactor the forest system to include several shapes in a single datablock? I've not even looked at the code, so I don't know whether this is feasible, but I'm certainly looking at something like this for a procedural building system I've got cooking - each building has a single datablock that specifies entire lists of different files for walls, floors, etc., which can be varied randomly or according to the floorplan file.Out of curiosity (I'm away from my dev computer for the next two weeks) - what do the forest object datablocks contain? Could some of it be moved into a multi-shape datablock, or read from the shape files themselves?
#15
So we should change the code so we book a range of this for datablock, messages and object id will take the remaining.
I mean 1024 in 4294967295 is nothing ^^ and before you reach the maximum number of object, you might think about rebooting your server like on a weekly maintenance.
Honestly, I thought this was working like that until Rene post :(
@Rene, do think we can change something in the it's currently managed without breaking out all the code?
02/06/2011 (11:47 pm)
What's confusing is that: 32 bits = 4294967295So we should change the code so we book a range of this for datablock, messages and object id will take the remaining.
I mean 1024 in 4294967295 is nothing ^^ and before you reach the maximum number of object, you might think about rebooting your server like on a weekly maintenance.
Honestly, I thought this was working like that until Rene post :(
@Rene, do think we can change something in the it's currently managed without breaking out all the code?
#16
One could probably collate stuff here but it would probably make things rather unsightly.
However, what's definitely a problem is that all forest item datablocks are loaded en bloc for every game.
<Cut the cr*p...>
//Edit: BTW, consoleReflection in b3 has an issue that causes object IDs to be eaten up way quicker than in 1.0. That's fixed in 1.1 final in that consoleReflection is gone (it was a misdirected effort of mine).
02/07/2011 (1:09 am)
Quote:Seems like a case of too finely-grained datablock usage?
One could probably collate stuff here but it would probably make things rather unsightly.
However, what's definitely a problem is that all forest item datablocks are loaded en bloc for every game.
<Cut the cr*p...>
//Edit: BTW, consoleReflection in b3 has an issue that causes object IDs to be eaten up way quicker than in 1.0. That's fixed in 1.1 final in that consoleReflection is gone (it was a misdirected effort of mine).
#17
Sorry for the confusion I have caused here guys. After waking up now and actually checking things out, I realize I completely read this backwards. So much for posting when you're tired and confused...
Anyways, the ID allocation works the way that elvince describes here and the bit allocation does not eat out of the complete bit space.
That's extreme but still, as deepscratch has said, it'll work.
We should probably raise the default to at least 11.
02/07/2011 (1:56 am)
Sometimes.... sigh...Sorry for the confusion I have caused here guys. After waking up now and actually checking things out, I realize I completely read this backwards. So much for posting when you're tired and confused...
Anyways, the ID allocation works the way that elvince describes here and the bit allocation does not eat out of the complete bit space.
Quote:change it to 20
That's extreme but still, as deepscratch has said, it'll work.
We should probably raise the default to at least 11.
#18
PC Specs:
Core 2 duo system
8 GB of ram,
radeon HD graphics card.
My system is not slow what so ever. Is there a way to just have it load the forest info only when entering the brush mode? Or like Daniel stated earlier a different define type for these?
I really like the forest editor but when its loading every element every mission even though none are being used is killer to mission load time.
02/07/2011 (2:14 am)
It works but the problem still remains that every mission load all datablocks are reloaded. Even though those forest elements are not being used. The load time of a mission for me is now from 15 seconds to 1:53 seconds. PC Specs:
Core 2 duo system
8 GB of ram,
radeon HD graphics card.
My system is not slow what so ever. Is there a way to just have it load the forest info only when entering the brush mode? Or like Daniel stated earlier a different define type for these?
I really like the forest editor but when its loading every element every mission even though none are being used is killer to mission load time.
#19
02/07/2011 (2:20 am)
If i used the forest kit I'd have to have a mini game every time someone zoned into a new map area. That would make me either create large zones or not use the forest editor at all. Even though the brush increase productivity of creating a map so much. Is there a way to just have it use the raw files, dea,dts,dif,etc to just paint the objects in a selected list to the screen with out defining them as datablocks? I love this tool but the implementation is killing me as of current...
#20
there is a way,
I dont know how,
but,
you can set it up to load the dataBlocks on demand,
not at game load.
the point of loading everything at game start, I guess, is to remove that bit of lag when a datablock is loaded on the fly.
loading everything just in case you might need it doesnt seem too practicle.
so ideally, things like trees and boulders, large props, basicaly anything that you would generaly place in editor, you could set up to load on demand,
and things like weapons, AI, characters, anything that is in the world for game start, should be set to preload.
this will keep the overheads waaaay down,
but you may, very very occasionaly, get a loading hic, like a millisecond, when something unplanned for needs to get loaded.
but you plan the game, you are the designer, plan ahead for that rare occasion.
again, I dont know how its done, I have seen threads about it, if anyone knows what I am talking about, could you link us to those threads?
its an issue that belongs with older gen engines....1024 dataBlocks in a next-gen game engine in 2011??
just my particles library is more than that!!!
02/07/2011 (2:42 am)
ok, long post....there is a way,
I dont know how,
but,
you can set it up to load the dataBlocks on demand,
not at game load.
the point of loading everything at game start, I guess, is to remove that bit of lag when a datablock is loaded on the fly.
loading everything just in case you might need it doesnt seem too practicle.
so ideally, things like trees and boulders, large props, basicaly anything that you would generaly place in editor, you could set up to load on demand,
and things like weapons, AI, characters, anything that is in the world for game start, should be set to preload.
this will keep the overheads waaaay down,
but you may, very very occasionaly, get a loading hic, like a millisecond, when something unplanned for needs to get loaded.
but you plan the game, you are the designer, plan ahead for that rare occasion.
again, I dont know how its done, I have seen threads about it, if anyone knows what I am talking about, could you link us to those threads?
its an issue that belongs with older gen engines....1024 dataBlocks in a next-gen game engine in 2011??
just my particles library is more than that!!!
Torque Owner Richard Ranft
Roostertail Games