GG Member Ratings
by Shane McLeod · in Site Feedback · 01/14/2003 (4:11 pm) · 12 replies
I posted this in another thread and rather than derail the other thread, I decided to start a new one.
I would really like to see a way to rate members that accept jobs, for pay or not. Twice now I've had people join the team just to disappear within a very short time. There should be a way to rate a members performance so that others don't end up relying on people who just bail without saying a word. Perhaps something along the lines of EBay.
In EBays system, the buyer rates the seller based on product quality and expeditious delivery. The seller may then respond to the rating comment with his/her own comment, which helps to prevent hostile rating manipulation. Either way, it's all public and future buyers can look at the ratings and comments and make a semi-informed decision.
What I propose is a system where teammates (new applicants or old) can be rated on whether they finished the job, produced timely work, etc. Teammembers could also rate companies for things such as ethical and honest conduct, organization and implementation. The rating itself could be a single number (say from 1-100) and the comments would spell out what influenced the rater to determine the rating they gave.
I would really like to see a way to rate members that accept jobs, for pay or not. Twice now I've had people join the team just to disappear within a very short time. There should be a way to rate a members performance so that others don't end up relying on people who just bail without saying a word. Perhaps something along the lines of EBay.
In EBays system, the buyer rates the seller based on product quality and expeditious delivery. The seller may then respond to the rating comment with his/her own comment, which helps to prevent hostile rating manipulation. Either way, it's all public and future buyers can look at the ratings and comments and make a semi-informed decision.
What I propose is a system where teammates (new applicants or old) can be rated on whether they finished the job, produced timely work, etc. Teammembers could also rate companies for things such as ethical and honest conduct, organization and implementation. The rating itself could be a single number (say from 1-100) and the comments would spell out what influenced the rater to determine the rating they gave.
About the author
#2
One thing about slashdot, and don't quote me on this becuase I'm not positive this is true, but I think in their system higher rated people have more impact on the ratings.
-Spencer
01/14/2003 (5:47 pm)
This seems like an idea worth exploring, though I think it'd have to be designed in such a way as to prevent abuse. In general, however, the more I dig through GG, the more I wish there was some sort of democratic rating and filtering system for all content, even posts, like on Ebay or slashdot.org. One thing about slashdot, and don't quote me on this becuase I'm not positive this is true, but I think in their system higher rated people have more impact on the ratings.
-Spencer
#3
There are eight levels of posts: +5 through -2 (+1 is normal. The normal minimum is -1, but if you've ben REALLY bad, you get -2). A select group of random users every so often get 5 "moderation points," each of which they can use to either bump up a post they think is worth the attention by a level or knock down obvious trolls, flames, etc. by a level. In addition to all of this, you can set yourself up to only view posts of a certain rank or higher.
Every user starts out with 0 "karma points" out of a possible 50 maximum and no minimum AFAIK. Every time your post recieves a moderation up, you gain a karma point; every time it goes down, you lose one. If you've been rated highly enough (25 karma points), you can elect to make your posts at a +2 rank instead of the normal +1. You also get automatic bonuses for other attributes (i.e. a very long post will get an automatic +1 in the ranking).
However, this is not a system I'd personally like to see implemented here on GG. At Slashdot (which, BTW, is full of OSS zealots), it's quite easy to post an opinion that differs from the norm (i.e. saying you prefer Windows to Linux for some tasks) and actually end up being punished for it. Since the moderators are random and anonymous, there's no accountability for moderators' actions and it (IMO) creates a very hivethink site. Some of the discussions that have occurred on this board are pretty valuable BECAUSE of the differing opinions presented, and a system like Slashdot's isn't the best way to foster a community based on those sorts of principles, IMO.
01/14/2003 (6:50 pm)
Slashdot Rating System in a Nutshell(tm):There are eight levels of posts: +5 through -2 (+1 is normal. The normal minimum is -1, but if you've ben REALLY bad, you get -2). A select group of random users every so often get 5 "moderation points," each of which they can use to either bump up a post they think is worth the attention by a level or knock down obvious trolls, flames, etc. by a level. In addition to all of this, you can set yourself up to only view posts of a certain rank or higher.
Every user starts out with 0 "karma points" out of a possible 50 maximum and no minimum AFAIK. Every time your post recieves a moderation up, you gain a karma point; every time it goes down, you lose one. If you've been rated highly enough (25 karma points), you can elect to make your posts at a +2 rank instead of the normal +1. You also get automatic bonuses for other attributes (i.e. a very long post will get an automatic +1 in the ranking).
However, this is not a system I'd personally like to see implemented here on GG. At Slashdot (which, BTW, is full of OSS zealots), it's quite easy to post an opinion that differs from the norm (i.e. saying you prefer Windows to Linux for some tasks) and actually end up being punished for it. Since the moderators are random and anonymous, there's no accountability for moderators' actions and it (IMO) creates a very hivethink site. Some of the discussions that have occurred on this board are pretty valuable BECAUSE of the differing opinions presented, and a system like Slashdot's isn't the best way to foster a community based on those sorts of principles, IMO.
#4
Edit : did not mean to pass judgement either on Slashdot(well, maybe a bit ;)), or your idea, Mad. Certainly not your idea. I just thought Corvidae has underlined a possible pitfall in an elegant, yet quite to the point way. That is all :)
01/14/2003 (7:04 pm)
Well said Corvidae, very well said. :)Edit : did not mean to pass judgement either on Slashdot(well, maybe a bit ;)), or your idea, Mad. Certainly not your idea. I just thought Corvidae has underlined a possible pitfall in an elegant, yet quite to the point way. That is all :)
#5
01/14/2003 (7:11 pm)
Well, this is getting a little off track from my suggestion. I'm suggesting that if you are on a team, your GG reputation should reflect your work.
#6
01/14/2003 (7:22 pm)
I agree entirely. It was brought up, though, and I figured I might as well throw it out there.
#7
Use a trust metric.
edit: Bad link, see post below for correct information.
01/14/2003 (7:33 pm)
www.advogadro.com.Use a trust metric.
edit: Bad link, see post below for correct information.
#8
Mac: Sorry, didn't mean to derail your post. I like your suggestion, especially the open nature of comments between parties.
Have you seen the rating system at www.elance.com? Another site I've had limited experience with, but they seem to have implemented a system like the one you describe.
01/14/2003 (7:46 pm)
James: Thanks for the clarification! I've only had limited experience with slashdot, but a friend of mine describe its system in such glowing terms, I thought it might be a good rating system. Of course he is a Linux zealot, hmmm...Mac: Sorry, didn't mean to derail your post. I like your suggestion, especially the open nature of comments between parties.
Have you seen the rating system at www.elance.com? Another site I've had limited experience with, but they seem to have implemented a system like the one you describe.
#9
James - Thanks for the info. I wasn't exactly sure how it worked but after your explanation, I can see how the system would get skewed.
Ben - That link isn't working for me. I keep getting DNS errors with it. Strange, I'll try again later. Care to briefly explain the trust metric?
Spencer - No problem at all. I was just concerned that if we slapped too much into it, it may be too monolithic for GG to consider. That is assuming that they haven't already considered this. I checked out the link you provided and if you're talking about the feedback rating, that's exactly what I mean. Great site by the way. I didn't even know they existed. Thanks.
01/14/2003 (7:58 pm)
Ted - I agree completely. Comments should be mandatory. If someone doesn't have the ba..s err fortitude to back up their rating with a comment, they shouldn't be rating at all.James - Thanks for the info. I wasn't exactly sure how it worked but after your explanation, I can see how the system would get skewed.
Ben - That link isn't working for me. I keep getting DNS errors with it. Strange, I'll try again later. Care to briefly explain the trust metric?
Spencer - No problem at all. I was just concerned that if we slapped too much into it, it may be too monolithic for GG to consider. That is assuming that they haven't already considered this. I checked out the link you provided and if you're talking about the feedback rating, that's exactly what I mean. Great site by the way. I didn't even know they existed. Thanks.
#10
Maybe somebody that could spend more time on it is interested in implementing a system on a different site. I really have no idea how to implement this, or how it would work, so you are on your own. If it works well, we would put up a link and promote the site.
Jeff Tunnell GG
01/15/2003 (8:42 am)
When we originally envisioned GarageGames, we thought we would have a developer rating system. However, the more we thought about it, the less we thought it was a good idea. There is no good way to rate people that would not turn into a huge political mess. So, we decided to put our efforts elsewhere.Maybe somebody that could spend more time on it is interested in implementing a system on a different site. I really have no idea how to implement this, or how it would work, so you are on your own. If it works well, we would put up a link and promote the site.
Jeff Tunnell GG
#11
01/15/2003 (9:00 am)
You could make the ratings more specific. You guys already store the info about what projects people are involved in and such. So you could use that to determine whether or not you can even rate someone. So if I go along on this site for years, I get involved in a few projects, that gets stored. Then someone wants to rate me and say what a wonderfully sexy, funny, and awesome coder I am, they would select for which project they are rating me, then they would fill in 10 for sexy, 10 for funny, and a 10 for coding skillz. (C: (or something). That way, when you click on someone's profile, it will have a section that shows their ratings. They would see that for Project A I was sexy, funny, and an awesome coder, and for Project B, I was very sexy, extremely dillusional, and an incredible coder. This type of approach would allow for other types of ratings down the line. Perhaps people can rate my intelligent forum posts, or how extremely useful my resource submssions are, etc.
#12
www.advogato.org/trust-metric.html is a paper on an implementation of a trust metric used to rate OSS developers on Advogato. It seems to be a pretty slick system.
01/15/2003 (9:23 am)
Yes... Sorry, my bad. I always get that link wrong :)www.advogato.org/trust-metric.html is a paper on an implementation of a trust metric used to rate OSS developers on Advogato. It seems to be a pretty slick system.
Torque 3D Owner Ted Southard