Large RPG loading and rendering issues
by Howard Dortch · in Torque 3D Professional · 02/08/2012 (9:05 am) · 62 replies
As the title implies. I load a mission file with buildings, doors, switches etc. As each item loads the engine creates a unique ID for reference. Now I have dynamic items that get loaded as POI items. I have them listed in a database (sqlite) so at the end of the mission load I load the dynamic POI stuff.
Here is the problem: Before all the items from the mission file actually get rendered in the game, the dynamic item loader gets called and one or more items from the mission file will disappear.
So my assumption is that one of the dynamic items is overwriting the ID number from the mission file and it causes the disappering item issue.
Couple of questions:
When creating the mission file would it help to bundle things in Simgroups ?
When creating the dynamic items should I delay creation until I am sure all mission items have been created?
Any help on the proper way to do this?
Here is the problem: Before all the items from the mission file actually get rendered in the game, the dynamic item loader gets called and one or more items from the mission file will disappear.
So my assumption is that one of the dynamic items is overwriting the ID number from the mission file and it causes the disappering item issue.
Couple of questions:
When creating the mission file would it help to bundle things in Simgroups ?
When creating the dynamic items should I delay creation until I am sure all mission items have been created?
Any help on the proper way to do this?
#42
Also, I had a time when stuff randomly vanished because I was deleting rather than removing decals with script. As it couldn't find the actual decal it just deleted random objects - took me ages to find that typo ...
04/15/2012 (6:24 pm)
Silly question but they have updated Dx and GFX drivers to the latest? I've seen all sorts weirdness when they're out of date.Also, I had a time when stuff randomly vanished because I was deleting rather than removing decals with script. As it couldn't find the actual decal it just deleted random objects - took me ages to find that typo ...
#43
No decals in this, footprints is all but the objects go missing before any steps.
04/17/2012 (7:13 pm)
Yes updated drivers etc...No decals in this, footprints is all but the objects go missing before any steps.
#44
I think I have about 3300 objects placed in the mission file, as well as about 150 dynamically instantiated objects created via "new object" in script after the engine is launched. I am using a dedicated server/remote client setup, and I have an issue that in short form, is thus:
Picture a large map, with a road running around the square perimeter inside some walls. Along the road are torches with a particle effect, and buildings with doors built with datablocks, just like Howard's.
Loading the character in any arbitrary corner, and beginning to run about the map, the torches flare into life as one approaches, as do the doors pop into view*. After completing 2/3 of a circuit, the particle effects stop being generated (no flame appears), and the doors stop popping into view. However, on approach, the doors still have an operating collision block--the keystroke/raycast which opens them still operates, they can be opened and walked through, and then close again.
In my case, it isn't a particular door object, which doors and torches stop working depends on where you start on the map, so it definitely appears to be due to an "amount" of some object being exceeded.
Animations will stop playing on the character at this point, if an AFX effect is triggered, and NPCs on the map will become invisible, though they will still attack and do damage.
My build is 1.1 with AFX, so the datablock bits are set to 13, with message bits the standard 6, and thus I would expect to see 13 bits left for 8192 objects. In reading through this thread, I'm getting the impression that 4096 is the more practical limit, so now I'm concluding I've long since hit the object ceiling and will be ripping great chunks of objects out soon to see.
Last night, I was working through this thread here: http://www.garagegames.com/community/forums/viewthread/124215/1
and someone suggested that SimObjectID be upgraded to a U64. I went digging around and found tantalizing "//TODO: 64bits" comments around line 208 in "typetraits.h", so I don't think it's going to be that easy as the datatype isn't present (as well as mostly likely creating a host of other issues).
Now, like Howard, I've enjoyed building a "rich" level. It does seem a bit unkind that the engine can render all these details so beautifully, only to hit a hard limit based on a data type. As operating systems are now commonly 64 bit, I'd purely love to see this roadblock removed, as "deeper and richer levels" is practically my mantra.
My video drivers are the Nvidia GeForce 296.10 release, running twinned 550 Tis.
*I've implemented Vince Gee's dynamic distance resource. I have taken it back out, and still seen the same issue, which only makes sense if it's an object count issue. However, so I put it back in. I'll probably have it back out soon for another test after I take out half the objects in the level arbitrarily in another test.
04/18/2012 (7:29 am)
You know, I came here this morning to ask about a problem I'm having which sounds very similar to yours, Howard. I think I have about 3300 objects placed in the mission file, as well as about 150 dynamically instantiated objects created via "new object" in script after the engine is launched. I am using a dedicated server/remote client setup, and I have an issue that in short form, is thus:
Picture a large map, with a road running around the square perimeter inside some walls. Along the road are torches with a particle effect, and buildings with doors built with datablocks, just like Howard's.
Loading the character in any arbitrary corner, and beginning to run about the map, the torches flare into life as one approaches, as do the doors pop into view*. After completing 2/3 of a circuit, the particle effects stop being generated (no flame appears), and the doors stop popping into view. However, on approach, the doors still have an operating collision block--the keystroke/raycast which opens them still operates, they can be opened and walked through, and then close again.
In my case, it isn't a particular door object, which doors and torches stop working depends on where you start on the map, so it definitely appears to be due to an "amount" of some object being exceeded.
Animations will stop playing on the character at this point, if an AFX effect is triggered, and NPCs on the map will become invisible, though they will still attack and do damage.
My build is 1.1 with AFX, so the datablock bits are set to 13, with message bits the standard 6, and thus I would expect to see 13 bits left for 8192 objects. In reading through this thread, I'm getting the impression that 4096 is the more practical limit, so now I'm concluding I've long since hit the object ceiling and will be ripping great chunks of objects out soon to see.
Last night, I was working through this thread here: http://www.garagegames.com/community/forums/viewthread/124215/1
and someone suggested that SimObjectID be upgraded to a U64. I went digging around and found tantalizing "//TODO: 64bits" comments around line 208 in "typetraits.h", so I don't think it's going to be that easy as the datatype isn't present (as well as mostly likely creating a host of other issues).
Now, like Howard, I've enjoyed building a "rich" level. It does seem a bit unkind that the engine can render all these details so beautifully, only to hit a hard limit based on a data type. As operating systems are now commonly 64 bit, I'd purely love to see this roadblock removed, as "deeper and richer levels" is practically my mantra.
My video drivers are the Nvidia GeForce 296.10 release, running twinned 550 Tis.
*I've implemented Vince Gee's dynamic distance resource. I have taken it back out, and still seen the same issue, which only makes sense if it's an object count issue. However, so I put it back in. I'll probably have it back out soon for another test after I take out half the objects in the level arbitrarily in another test.
#45
In my case, at least, it does seem to be related to the number of objects. I recalled that while looking at SimObjectID last night, that the bit order starts at bit #3 when counting, which means after datablocks seize 13, and messages take 6, I don't have 13, but 10 bits left for objects--which means assuming "8192" was a vast overestimate... I probably have room for about 2048 objects, and I would say I have definitely hit that ceiling.
I went back a couple of generations on my backup, and found an older mission file, which when checked would only turn up one missing object on a circuit of the map. I concatenated a bunch of small objects, largely walls, exporting them from the engine via "Export to Collada..." and reimporting them as a single Collada object, and on further circuits, no objects have turned up missing.
So I'm satisfied I have a cause for this one, at least--it's not a thrilling thing, but now I only have to find workarounds for some set limit of objects instead of plunging through the unknown.
@Howard: I hope you find your error with similar alacrity, Howard, and I apologize if I have threadjacked.
04/18/2012 (9:48 am)
Yep, as per usual, I post at great length right before I find the bug...In my case, at least, it does seem to be related to the number of objects. I recalled that while looking at SimObjectID last night, that the bit order starts at bit #3 when counting, which means after datablocks seize 13, and messages take 6, I don't have 13, but 10 bits left for objects--which means assuming "8192" was a vast overestimate... I probably have room for about 2048 objects, and I would say I have definitely hit that ceiling.
I went back a couple of generations on my backup, and found an older mission file, which when checked would only turn up one missing object on a circuit of the map. I concatenated a bunch of small objects, largely walls, exporting them from the engine via "Export to Collada..." and reimporting them as a single Collada object, and on further circuits, no objects have turned up missing.
So I'm satisfied I have a cause for this one, at least--it's not a thrilling thing, but now I only have to find workarounds for some set limit of objects instead of plunging through the unknown.
@Howard: I hope you find your error with similar alacrity, Howard, and I apologize if I have threadjacked.
#46
04/20/2012 (6:52 am)
@Netwyrm not a problem. You gave me more places to look, I will probably make those same changes, thanks.
#47
as a side question I would like to know which of the following is preferred. I have to reference these objects in various places on the server (global reference) but it seems any will work but what is the "proper" way?
A. $someobj = new Staticshape()
B. new StaticShape(someobj)
C. $someobj = new StaticShape(someobj)
05/04/2012 (6:59 am)
Still having the issue, reorganized the code some and now different things are missing. Object 6223 seems to always be the one missing no matter what it is or how the code is arranged. Nothing in the engine with that value, 184f hex means nothing either.(strange) as a side question I would like to know which of the following is preferred. I have to reference these objects in various places on the server (global reference) but it seems any will work but what is the "proper" way?
A. $someobj = new Staticshape()
B. new StaticShape(someobj)
C. $someobj = new StaticShape(someobj)
#48
example %name as Astald
function Player::showGraveStone(%this, %obj)
{
%position = %this.getPosition();
%name = %this.getShapeName();
%name = %name @ "_Grave";
new StaticShape(%name) {
isAIControlled = "0";
dataBlock = "graveStone_01";
category = "Effects";
class = "Grave";
position = %position;
rotation = "1 0 0 0";
scale = "1 1 1";
canSave = "1";
canSaveDynamicFields = "1";
};
%name.grave.shapeName = %name;
}
anywhere you can call the object with this name
like Astald.position etc
05/06/2012 (5:22 pm)
i use that new StaticShape(someobj)example %name as Astald
function Player::showGraveStone(%this, %obj)
{
%position = %this.getPosition();
%name = %this.getShapeName();
%name = %name @ "_Grave";
new StaticShape(%name) {
isAIControlled = "0";
dataBlock = "graveStone_01";
category = "Effects";
class = "Grave";
position = %position;
rotation = "1 0 0 0";
scale = "1 1 1";
canSave = "1";
canSaveDynamicFields = "1";
};
%name.grave.shapeName = %name;
}
anywhere you can call the object with this name
like Astald.position etc
#49
06/14/2012 (5:40 am)
Still having this problem. Can anyone suggest a way to fix this? Game is finished but I can't sell it when parts randomly disappear. 
#50
The other question is, are the objects not showing up statics and shapebase derived?
Do the objects exist on the server but not the client?
If it's only a client-side issue....
If they are shapebase derived, then they are going through the ghosting routines. My guess is that your static shapes aren't having issues.
The reason I ask this is kinda silly.
Lets assume you have a small mission file and you are ghosting 1000 objects in the ghost range. (Assuming you are using my resource)
I don't remember for sure, but I do know Players and vehicles get the highest priority. Then, everything else is ranked.
It then sends the ghosts down to the client.
Now... let say you have 500 objects in ghost range, players and vehicles go first, then it tries to send as many as possible of the remaining objects. IF it doesn't finish sending all 500 objects, on the next ghost pack, it starts at the beginning and does it all over again. And if I remember correctly, the last 300 ghosts won't be sent down to the client.
Couple things I would do, if your running my resource, drop the ghost range down to 50. Especially if you have alot of object in close proximity.
Let me know,
Vince
06/14/2012 (9:56 am)
Stupid question, what happens when you run the client and server together?The other question is, are the objects not showing up statics and shapebase derived?
Do the objects exist on the server but not the client?
If it's only a client-side issue....
If they are shapebase derived, then they are going through the ghosting routines. My guess is that your static shapes aren't having issues.
The reason I ask this is kinda silly.
Lets assume you have a small mission file and you are ghosting 1000 objects in the ghost range. (Assuming you are using my resource)
I don't remember for sure, but I do know Players and vehicles get the highest priority. Then, everything else is ranked.
It then sends the ghosts down to the client.
Now... let say you have 500 objects in ghost range, players and vehicles go first, then it tries to send as many as possible of the remaining objects. IF it doesn't finish sending all 500 objects, on the next ghost pack, it starts at the beginning and does it all over again. And if I remember correctly, the last 300 ghosts won't be sent down to the client.
Couple things I would do, if your running my resource, drop the ghost range down to 50. Especially if you have alot of object in close proximity.
Let me know,
Vince
#51
The object(s) that goes missing is defined in the mission file as a staticshape i.e. new StaticShape(HatchOverride) After the mission loads one of the shapes usually ID 6232 is not there. All objects have a onAdd function (server side I assume ) and I print the object, name and ID value. So when I get in the game and an object is missing I do a echo(objname.getPosition()) and that object does not exist even tho the onAdd function says it was created and an ID number assigned. I look in the console find the object in question from the onAdd and type 6232.dump() and it says the object does not exist. onAdd says it got created or it would not be called.
I am not aware of your resource so I assume I am not using it.
06/14/2012 (10:42 am)
I run the game for now as a player server so no dedicated server. The object(s) that goes missing is defined in the mission file as a staticshape i.e. new StaticShape(HatchOverride) After the mission loads one of the shapes usually ID 6232 is not there. All objects have a onAdd function (server side I assume ) and I print the object, name and ID value. So when I get in the game and an object is missing I do a echo(objname.getPosition()) and that object does not exist even tho the onAdd function says it was created and an ID number assigned. I look in the console find the object in question from the onAdd and type 6232.dump() and it says the object does not exist. onAdd says it got created or it would not be called.
I am not aware of your resource so I assume I am not using it.
#52
06/14/2012 (11:33 am)
Have you tried echoing something in the onRemove function too? That might give you a hint about when the object is disappearing.
#53
06/15/2012 (7:14 am)
@Guy good idea, hadn't thought of that. I implemented onRemove on todays object to randomly disappear and sure enough it is being removed so now the question would be why. I dont call remove on anything till the game session ends so this verifies what I been saying. One part will randomly disappear and I see it's ID 6263 yet again. Thanks ....
#54
06/15/2012 (7:28 am)
Can you set things up to step through your object creation code with a debugger, then you might be able to see exactly when the onRemove gets called and trace things back from there. 
#55
Then look for any of those messages in the console log. If I remember correctly, if you create a new object with the same name it will first destroy/remove the existing object. This will just confirm that your names are unique when you create the objects. Just make sure that %name is changed to the actual variable that you use.
06/15/2012 (7:31 am)
@Howard, put this code before you create your objects:if(isobject(%name)){
warn("!!!! Object '" @ %name @ "' already exists. !!!!"
}Then look for any of those messages in the console log. If I remember correctly, if you create a new object with the same name it will first destroy/remove the existing object. This will just confirm that your names are unique when you create the objects. Just make sure that %name is changed to the actual variable that you use.
#56
06/15/2012 (9:58 am)
@Ryan not sure I can put that code in the mission file. I use Torsion and did a global find in files for things that get removed and I am certain there are no duplicates. Remember it's not always the same object. If I add or change anything then a different object gets deleted. I can solve the problem by adding in a dummy object in the mission file just before the one that disappears and that fixes it since the dummy gets deleted but thats a fix not a cure...Thanks...
#57
06/15/2012 (10:02 am)
@Guy I did that a few months back when an object consistently disappeared. Running the debuger it never happened. Which made me think it was a timing issue in a thread. One thread (mission file loader creator) was getting a ID and another thread hit a dynamic creation and was trying to obtain the same number. Repeat it just "seems" like that may be the issue. Obtain Semaphore is a beautiful thing...
#58
I added setScopeAlways() to the onAdd() but still no render on some random part.
I would really like to sell this game someday..
07/08/2012 (9:00 am)
And here we go again. I got tired of the object loading failure so I decided to put all the object in the mission file (yet many more hours wasted) Now I have objects I can select with the world editor. And yet again some objects are missing but this time they exist in the fact they get assigned a valid id number and my redundancy object checker says the object is still valid after a game load. Soooo I open the editor, click on the object and sure enough I get the object highlited with the x,y,z pull bars. Yaaay ... I go click on the object to move it to see if it will render and the game just exits to desktop. Really? I click on the object in the object selection window and change it's position in the world manually (or anything else for that matter) and the game just exits, nothing in the console., just straight to desktop. I added setScopeAlways() to the onAdd() but still no render on some random part.
I would really like to sell this game someday..
#59
07/08/2012 (1:08 pm)
Apologies if this is something you're already aware of - run a debug build from within visual studio, then instead of the game exiting to desktop with no useful information, it will exit to visual studio and give you the callstack and relevant variables to examine. That way you will be able to see exactly which engine function caused the crash and the execution path that lead to that function being called. You will also be able to examine any variables in scope to see if there are any bad pointers or similar causing the crash. 
#60
This may be related to a problem the torque engines have had since the move to socket based code many years ago. I haven't investigated this fully but it's clear the current code is broken.
This should allow the loading of as many objects as you like from the mission but also affects high traffic moments as well.
It's worth noting I've pushed back the packet size limits and made fixes related to packet update rate as well so if this doesn't help I may need to post the other changes I've made. Let me know.
10/28/2012 (11:43 pm)
Hi All,This may be related to a problem the torque engines have had since the move to socket based code many years ago. I haven't investigated this fully but it's clear the current code is broken.
// start jc
// this little sucker limits LAN utilization to 1% of
// our modern 1Gb large window hardware.
// hard coded values ruin lives!
// error = setBufferSize(udpSocket, 32768);
error = setBufferSize(udpSocket, 32768*8);
// end jcThis should allow the loading of as many objects as you like from the mission but also affects high traffic moments as well.
It's worth noting I've pushed back the packet size limits and made fixes related to packet update rate as well so if this doesn't help I may need to post the other changes I've made. Let me know.
Torque 3D Owner Howard Dortch
Default Studio Name