How Quality is your Quality Assurance team?

#1 Odd, I've not had any issues with SSBB.

I HAVE had issues with the wireless Gamecube controllers interfering with my Wifi connection! My friend from Detroit (I live in Austin TX) and I happened acrossed the issue while gaming the other night. Once we switched our controller channels down to 1, we had <200ms lag the whole match, which felt just about normal. Of course over wifi a fast paced game like SSBB will never feel perfect. It just isn't possible using the fastest technology available on the market today. That is the limitation we must live with.

Nintendo actually released SSBB 6 months early in Japan so they could stress test the internet play extensively before bringing it to America. Granted, the US is severely lacking in networking infrastructure. Several nights a week internet play on Dragonrealms (a MUD), City of Heroes, Two Worlds and even Sins of a Solar empire suffer not because of poor netcode, but because of the crappy routing service throughout the country.

I've been impressed by game netcode lately. Even when I was having 20% packet loss I never dropped from Sins or Two Worlds. Even SSBB just freezes until everyone can re-synch. Gone are the days where a bit of lag would drop you - which is a huge advancement if you ask me.

It is funny you should mention Shadowbane. After having our city teleported INTO THE OCEAN half a dozen times we all gave up and moved on.... I played on the War server IIRC and was a part of the DeathHeadLegion's almighty conquest over the entire map Rawr! Fun times. The game had so much potential but was marred by bugs and a feedback team that seemed nonexistant :(

Just my thoughts. Been having a blast with SSBB and I can't believe it's being taken to task for lacking Quality Assurance issues just 3 days after launch when my experience with it has been phenominal.

#2 Well, if you worked on ShadowBane, you shouldn't be surprised if something seems to work in a testing environment, and when the game is released it doesn't work. Although you knew exactly what could go wrong, things went wrong anyway, with lots of network problems.

In my opinion QA has got much better during the past few years. Especially for console titles, there usually is a zero-crash policy.

Network problems can, obviously, still happen. You can do all the testing you want with 100 testers or so, but when the game is released and you get 1 million players from day one, network problems are always extremely likely, and I don't think you can blame QA too much for that.
And Nintendo in particular... I believe they're just learning how to handle such a massive online user base.

#3 You in Austin huh? So am I .... well Hutto close enough. I wont say Shadowbane didnt have its share of bugs did you play the first release? Or was it the expansion packs? I worked on the first release. I agree with a lot of what you say on the advances in infrastructure in the US, but wonder if that is the case with there being a difference between the US and Japan why wouldnt they test for that situation. I know you cant catch everything and its impossible to think you can. I wonder if they did testing from the US side of the network.

#4 I am working on a network game right now. I think some of this may be my nerves speaking here. Kind of scared to release and have something like that happening to me.

#5 Yeah I was playing SB in the european beta, and immediately jumped on the US release. I had a 50th leather wearing, mind blast chucking, dual psi-blade slinging warlock (one of the few) within 2 weeks of launch. It was the first and last MMO I'd ever hit the level cap on!

Those early months were great...until we discovered how rough around the edges everything was. Aside from the poor graphics compatability (had to swap vid cards from nvidia, to ati, then back again as new bugs popped up), lag (teleporting into a new area took up to a minute on cable, during which time you could be attacked mercilessly), and messed up building support... it was a great game and had many "firsts" for the genre.

But yeah... logging in to find the city out in the ocean, only to chain-die-rezz-die-rezz-die-rezz-oh crap all my inventory is now destroyed ... log out..wait a week..log in... city is missing because some other guild seeded where we used to be :/ That cut our guild down to 10 people from 30.

I don't believe Nintendo's involvement with SSBB is very heavy on the online side. I'm also sure they did US testing, but Japan was the best case for LOTS of people trying it at once. Many companies don't get such an opportuntiy. You use their matching service, but any lag during the game is almost entirely the responsability of the user. The decision to use Wifi and not offer a wired option also complicates matters... not many people know how to setup a solid wifi link, wheras everyone can plug in a cable.

In any case its less than a week after launch, and this is Nintendo's first online title. Some slack is warranted :P I can't wait to see how future titles like Mario Kart end up.

As for your own game, all I can recommend is test, test, test... What kind of game is it?

#6 Don't want to give out to many details on the game itself before making a bigger announcment. Its a basically a 2d shooter using elements from standard 3d FPS games such as Team Fortress 2 and Battlefeild 2. It has a strong emphasys on team work like those games but meant for casual game play. Looking to make my way on InstantAction or XBLA

#7 What do you do in Ausitn?

#8 I'm pretty happy with the SSBB connections when dealing with friends. I'll post my code later. The matchups though don't work as well as they could for which Nintendo has wholly accepted the blame. Hopefully, the match-making will be in better shape over the next few days.

#9 I think I need to spend some more time with that game. So many people like it and the 20 minutes I put into it may not be enough.

#10 At vSide, we're lucky enough to have the budget to hire some outside folks to do QA. Depending on the match between the product and the QA team, the signal-to-noise can vary widely. Our current testers are from iBeta and have been invaluable in testing, retesting, confirming fixes, etc.

#11 SSBBM online play is horrid right now and hge amounts of people are complaining about it. In my opinion, Nintendo hs always marginalized online play as being an "add-on bonus feature" of some sort, when clearly the western world wants it. I expect they didn't put much thought into stress testing the service out in the US.

They have enough money to make an incredible online service - I just don't think they care to expend the time or energy.

#12 What I've played of Smash Bros online was pretty decent so far. I only played with my brother (him in OH, me in FL) for about an hour Sunday evening. We only experienced some minor lag during the last match of that hour. The biggest complaint I have about the online component is the friend code crap, but that's another issue all unto itself.

#13 I don't see the friend code as being any more difficult then setting up a steam account or registering on xfire though.... what about it is annoying you?

My gripe has to do with a lack of Voice chat support. I think it would have been a home run had they added it.. maybe it is still a possibility with WiiConnect in the future.

I've not has much experience with the PS3 or Xbox360's online matchmaking. In what ways is it better?

#14 @Jason

Yeah the huge lack of voice support is killing me with the Wii. Maybe when they released this game the West was an afterthought or secondary market.


@Orion

Do they make any promise on bug reporting? Such as we promise to catch 90% of the bugs or something like that. Like what metric do you use to measure a company like that.

#15 @ Jason

It's the game specific friend code part. If they just used the codes for the system and imported your friends list into the friend roster I wouldn't have found it so annoying. Just the process of having write down another long string of numbers and relay it to a friend, and add each other to the roster.

I do agree with voice chat complaint. My brother and I were actually using our cellphones to handle the smack talk for that hour we played.

#16 I got to thinking that what I will do when playing with my friends who own PS3's is just have them sign on with voice chat and have them all use that for chatting while we play on the Wii.

It's embrassing that you have to use another console to provide the functionality missing from the Wii, but most of my friends have either a 360 or PS3, so this solution seems to be the only one we have right now.

I agree, you gotta have voice when you play a game like SSBBM

#17 Personally I've played SSBB online and the networking seemed borderline perfect. No issues at all, playing with people in other states.

The lack of voice chat is one thing, I'm more annoyed at the complete user-hostility that is the friends list. You'd think that nintendo could come up with something better than a two-way sixteen-digit handshake done outside the system for your friend list [that's borderline impossible to find thanks to a hideously nonintuitive place to keep it]. Then you'd think they could come up with something better than *another*, *different*, two-way twelve-digit handshake also done outside the system just to play one game with my friends. That's a whole lot of stuff susceptible to simple tpyos, and a whole lot of pain in my ass just to chase people down.

They can figure out if your friends have metroid or not [friend tokens], but they can't figure out if your friends have SSBB or not?

I guess what they're shooting for is *not* having people use their email address as their system ID. Obviously this might lead to kid-unfriendly-situations like little bobby trying to add "donkeypuncher@hawtmale.com" to his friends list. I still figure that nintendo in all their glory could come up with something better than this total trash, and if that means sucking it up and accepting that email exists, then that's what they should do.


But then, I have an XBox also and I have hours of complaints I can make about that. The UI is awful, you have to pay a subscription if you want to play online [which I refuse to, so I can't]. The "wireless adapter" is half the cost of a wii [Cheaper, more reliable, and more secure to set up for XBox connectivity is a wireless bridge using a 50 dollar linksys wrt54g]. It is so loud when it's running that I have to keep it inside a separate cabinet just so I can hear the game. In a cabinet or not, it still runs at the temperature of the surface of the goddam sun.

Blah. Overall I'm woefully unimpressed with the XBox in general and the wii's buddy setup specifically.


*breathe*

Gary (-;

#18 @Gary
I agree with you all the way. And snorted pretty loudly at DonkeyPuncher sad thing is I don't why that is funny..

It's so funny how many different things that you think you can test for and correct before you release. I think the key take away here is that you can't please everyone.

#19 @Donald - good point on the metrics. Honestly i'm not sure how we chose them, but i think it was more through qualitative stuff like recommendations than quantitative measurement. We've had other outsourced QA which had a way lower signal/noise.

if we *were* using metrics, some ones i'd like to see:

* percentage of issues reported which are: blocker / critical / minor / not an issue
* ratio of real issues reported before release to total number of real issues a month or two after release
* average time between an issue being introduced and being reported
* number of exchanges per issue between the dev team and the QA team. (trying to quantify the quality of the bug reports. ie, do they give clear explanations, clear repro's, etc)

#20 I thought about one time using my QA background to try and put together QA for outsourced projects. But didn't think it would get me into the gaming career any quicker.