Multiplayer Broken?

#1 Usually this happens when a datablock fails its preload or your client/server assets are out of synch.

#2 Aww man, not again! I had just tested multiplayer two weeks ago...

#3 Yep, something is getting fouled up when ghosting to the connecting client...

Time to practice some polish curse words!

#4 NUUU, Say it isn't so!

#5 Uh.. any ideas how to fix it? I seem to recall this happening in TGEA 1.8.1 as well.. I swear no one tests multiplayer before the pack it up for a release :P

#6 @Helk
Multi what?


apologies for crap joke

#7 Anyone have any ideas how to fix this? It's pretty serious...

#8 I'm sure GG is on the case, it's still a beta product :)

#9 If you absolutely need to have multiplayer for testing purposes, use Beta3 (or 1, but not 2) for the time being... after all it's only been a couple of days and it is the weekend.

#10 Really? Is multiplayer broken in beta4?
But is it a specif genre kits, so perhaps is just a wrong datablock, or every project hangs up when going multiplayer?
Perhaps I'll wait to port my code to beta4.

#11 Beta3 was the first beta I tested separate client and server, and I had this issue there already. So I've already seen this in Beta 3 - not sure if it's the same issue, but I traced it to ghosting. (This was in my own game based on Beta 3, not the vanilla codebase.)

Good to know this is something already taken care of in other flavors of Torque. Once I port everything to Beta 4 (it's nice to see the changes, and I always learn a lot from them) I'll try to find this bugger.

@Steve: Don't stop with those jokes! :) (In return, I will shut up about my ideas for a Multiplayer Bokken)

#12 /sarcasm on
1) Woot, i thought the torque network was uber invincible and coudnt be broken ?
2) Beta, yes, but was it not supposed to be released in end of may :P ?
/sarcasm off

GG has however seem to be quick out with patches now, so you probably wont have to wait "along" time for a fix, and community seem pretty fast in helping to fix stuff aswell so :)

#13 Hey guys. Realize that this bug was reported at about 5pm on a Friday. It is not surprising that it hasn't been fixed or addressed yet ;)

I have tracked down the bug, but I would like to take this opportunity to first pimp one of the least known features in Torque, as well as give you guys who are working on networked games an idea about why Torque networking is really all that and a bag of chips.

TORQUE_DEBUG_NET

If you add TORQUE_DEBUG_NET to the pre-processor, it will enable a bunch of code (you can search for it yourself if you'd like) that performs tons of error checking on the net traffic. (There is also TORQUE_DEBUG_NET_MOVES)

In the case of this bug, it found the error instantly, and as soon as a client joined the server, an assert window popped up and said:

Quote:
unpack did not match pack for event of class GhostAlwaysObjectEvent.

This means that the 'pack()' and 'unpack()' methods on the 'GhostAlwaysObjectEvent' class are different. Popping over to netGhost.cpp, I see that the pack/unpack methods are the same, but that they call the pack/unpack methods for the object which is being ghosted in the GhostAlwaysObjectEvent.

So I set a breakpoint in GhostAlwaysObjectEvent::pack() on the line:

U32 retMask = obj->packUpdate(ps, 0xFFFFFFFF, bstream);

And then I run, connect, and the breakpoint is triggered. Stepping in to the methods, I hit the GroundPlane. Hmm, that looks kosher to me. Next...the SpawnSphere, nope, Sun, nope, SkyBox, nope, LevelInfo...AHA! The assert triggers! A quick look at the code, and what do I see?

// Pack
stream->write( mAdvancedLightmapSupport );

// Unpack
mAdvancedLightmapSupport = stream->readFlag();

What is the problem with this? Well a quick look into the implementation of write( bool ) shows it uses a U8 write, where as write/read flag does something different. A subtile bug? Yeah it sure was...and it would have been almost impossible to find without Torque networking.

If you are skimming my post for the bug fix, here it is...
Change line 107 in levelInfo.cpp from:

stream->write( mAdvancedLightmapSupport );

to

stream->writeFlag( mAdvancedLightmapSupport );

You guys who are doing multi-player games will be eating and breathing adding game-specific functionality which needs to be networked. Adding things to pack/unpack methods is an everyday occurrence. It is for reasons like this that I would not chose any other networking model to implement any multiplayer game. I found and fixed this bug in less than 5 minutes, with TORQUE_DEBUG_NET (It took me several times longer to describe how I found and fixed it). If I had not had the tools provided to me by Torque, it would have been a tedious task of back-stepping several thousand SVN revisions...and I wouldn't have even known which file was broken.

I apologize for this bug, since it was my bug, but I hope that my description of debugging a network error will help you guys out as you implement your game functionality with the help of Torque networking.

#14 Thanks Pat! And here I was doing things the "hard way", I had never actually paid any attention to those Debug flags before -- now I know :)

#15 This is why I am frequently seen saying something like, "...it is misunderstood..." I realize that this kind of functionality is poorly documented, and I do my best to explain it when threads (like this one, thank you very much Heik) give me the opportunity to do so...but it is kind of difficult to discuss something like debugging a problem like this, without a problem like this. I guess one could write a document from the perspective of, "First create an error, and then I'll show you how to fix it," but it seems like nobody would read that.

The main problem is that things like this aren't tutorials, or showing you how to make something cool...and so they get ignored, or skipped over by people because, lets face it, you usually search for, "How do I do XYZ," and when something like this breaks, you don't even know what to search for to find the answer. From the developer perspective, "it just crashes now," and that isn't a great repro case. The developer posts on the forums with something like, "Hey I changed something and it crashes now." The answer they get is, "Revert," but really the answer they need is, "Hey you changed a pack/unpack, this is why it crashes (net streams reading/writing different lengths of data), and here is how to find/fix it." That is why functionality like this is difficult to document and expose.

#16 Would this be a good time to mention

Error	1	error C2039: 'writeflag' : is not a member of 'BitStream'	c:\Torque\Torque 3D 2009 Beta 4\Engine\source\T3D\levelInfo.cpp	107

then?

#17 I just leveled up. Many thanks for explaining this, Pat.

#18 @Steve: case sensitive, try writeFlag ;)

#19 Steve, 'writeflag' is not a member of BitStream, but 'writeFlag' is :)

#20 Ah, wasn't aware I had to look out for case sensitivity (not a coder).

edit: lol, this explains some of my previous cockups when hacking around with C++ then. Doh!