Lord of the Files
by Netwyrm · 04/30/2016 (4:56 pm) · 2 comments

Well, I feel good this morning, in some whooping Jerry Lee Lewis great balls of fire way. Unusual state for me, and won't last liable another five minutes, but it looks like a Great Bug has been put to rest, at last. It took me a while, always does, but I finally had the idea to grab the total object count from missionGroup, and track it. When I saw the results from that, I knew I was onto something.
And there was the root cause of the "deafness" bug... mobs are spawned and placed without spawnspheres, but underneath it all a spawn marker still gets inserted to pop from. I'd written that basic mob code four years ago, and plumb forgot about that.
And so every time an NPC was inserted, an invisible spawn marker was going in, too. And not getting culled when the mob was removed. Leading, over time, to thousands and thousands of invisible spawn markers, piling up to densities which collapsed into a networking black hole on the map.
Engine is amazingly stable, even with that much constant stupid but eventually it 1) first stops listening for new connections, and 2) chokes. What was causing the collapse--in new, empty zones--was the ramping up of mob insertion as the maps finally start to get missions and ambushers.
Rewards were almost immediate--one tester has been soldiering through endless crashes of late, because his computer seems a lot more sensitive to maps filled with useless objects than the rest of the testers. Last night, he played for hours--with a friend onboard for the first time--and didn't crash.
So they fought the Ssarg at Trevarium... and the game got out of the way and let them play.
Great balls of fire!
Here's the code:
echo (missionGroup.getCount());and
function missionGroup::onObjectAdded(%this,%obj)
{
echo (%obj.getName());
echo (%obj.getClassName());
}Pretty basic stuff, but it took me forever to think to look there... invisible objects are invisible, and I generally look at code just written as proximate cause of any failures. But if a dedicated server is hanging up after a couple of days, though continuing to run for anyone logged in for another day or two, or you get mysterious disconnects for no apparent reason in a level "with nothing in it", look at the objects in the map at run time.
WHAT IS BEST IN LIFE? TO CRUSH THE BUGS, TO SEE THEM DRIVEN BEFORE YOU, TO HEAR THE LAMENTATIONS OF THEIR CLOSED TICKETS!
About the author
My adventures in T3D are chronicled at http://www.worldofantra.com. Please be aware the subject is sword-and-sorcery, and the occasional bloody or bare body part may be in scope.
#2
That bug has been in place for years, and has likely caused all sorts of issues meanwhile attributed to other causes. It didn't really start biting until recently because of a ramp up in mobs as the zones get populated to production levels.
The zone servers--visible on status page at the website--and trying to keep them available as a playable game emerges--crumbling for no apparent reason has been a little stressful.
One weird side effect--so many bugs have been found and crushed seeking the source of this one there might well have been a net positive impact on the stability of the game code.
And now the total mission object count has been permanently added to the feedback each servlet sends to the monitoring console every 60 seconds, so it will always, always be visible from now on!
04/30/2016 (7:29 pm)
Hi Simon!That bug has been in place for years, and has likely caused all sorts of issues meanwhile attributed to other causes. It didn't really start biting until recently because of a ramp up in mobs as the zones get populated to production levels.
The zone servers--visible on status page at the website--and trying to keep them available as a playable game emerges--crumbling for no apparent reason has been a little stressful.
One weird side effect--so many bugs have been found and crushed seeking the source of this one there might well have been a net positive impact on the stability of the game code.
And now the total mission object count has been permanently added to the feedback each servlet sends to the monitoring console every 60 seconds, so it will always, always be visible from now on!

Associate Simon Love
This bug, once you know about it, might seem super obvious and simple to fix but such bugs have a tendency to hide in places you'd never think to look.
Every time I've attempted to tackle larger T3D projects I've panicked due to the number of things one has to keep track of!
Keep it up!