GNOME Bugzilla – Bug 325562
exact points should probably be hidden behind more general descriptions
Last modified: 2009-09-17 11:49:24 UTC
Slashdot used to show the exact number of karma points you had, and it was just too gameable- people would end up trying to get points instead of actually doing whatever they were supposed to do. They have basically made it so that if you have over some threshold (used to be 50 points) your karma shows up as 'Excellent', and at lower levels there is apparently 'good' or 'positive'.
Besides reducing gaming, a text label based on threshholds is also more informative- I have no idea if someone with 10 'points' is an expert, a virtual newbie, or somewhere in between.
Might also have a switch in there where we replace the label with 'maintainer' if they are a maintainer, regardless of their # of points.
Suggested labels might be 'expert', 'regular contributor' (sort of long), and 'newbie' (though that has negative connotations.)
First, note that developers are already pointed out on show_bug.cgi; if anyone complains about not being listed as such you can tell them it's their own fault for not paying attention to my d-d-l and devel-announce-list emails.
Anyway, the suggestion sounds reasonable. bug 325330 comment 1 could help us decide where to place the thresholds. Even if we don't do this (see below), it may make sense to put such descriptions on the points explanation page as well as linking to the report requested in bug 325330 so that people have a better feel for what the points mean.
One alternate thing to consider is that points do serve as a way to encourage people to work in bugzilla. I think it is worth note that the weekly bug summary doesn't divide people into categories such as beginner/expert/clinically-insane-bugsquadder (I still love that description of yours) but uses actual numbers and thus is also subject to gaming. I believe that there have been problems with this, but not nearly enough to counterbalance the encourages-new-people-to-do more advantage it has. Further, one of the contributing factors to why it's a logarithmic scale is because that is more likely to encourage newbies while also discouraging gaming for anyone that has done a larger amount of work. Since the amount of work required to gain a point goes up exponentially with each level, the incentive to try to gain points diminishes quite rapidly; it'll take you or Kjartan probably years to gain an extra point, whereas newbies can advance probably every day if not sooner.
I'm not sure if I was clear above so let me just state that I'm not arguing against the suggestion so much as pointing out that there's another factor to weigh and stating that I have no clue which is more important at this time. Personally, I'd like to try the system as is for a while and then make the change suggested by Luis if we run into problems.
"First, note that developers are already pointed out on show_bug.cgi; if anyone
complains about not being listed as such you can tell them it's their own fault
for not paying attention to my d-d-l and devel-announce-list emails."
Oh, yeah, I know; I should have been more clear that I was thinking that maybe this would be a good way to consolidate these two data points, which currently are intended to show the same basic thing ('is this person competent/expert?') but are in different locations and which use different information.
Good point about the logarithmic nature discouraging gaming; I hadn't considered that- it probably resolves the serious concerns on that count.
So let me re-explain/re-emphasize the other count :) It seems like the primary goal here is to provide here a tool for newbies to see 'oh, this person is an expert, let me not mess with their judgment.' However, currently bugzilla is going about that by providing a number which is completely opaque to a newbie (really, to just about anyone.) If I'm a newbie, how am I supposed to know that a '5' is someone in the top 10-20% of bugzilla users? My first assumption was that 5 was pretty low, and I'd guess most people will assume it is on a ten point scale, so that it is average. So... I still feel fairly strongly that if this is to enhance the user experience, the information (which is great) needs to be communicated in a more user-understandable way.
Can't argue with that; makes sense.
gaming on the one hand motivates people to do some work;
on the other hand it may lead to people quickly closing bugs or marking duplicates just to gain extra points for the weekly bug summary and make progress on their way for more points.
I have the idea of tweaking the magic formula a bit that calculates the points.
I think of a bonus system where people get bonuses for doing helpful and valuable triaging work. Developers should give points where they see that triagers have gone good work that helped a lot on resolving issues.
Then again it should only comsume little time for developers.
I did observe the unfortunate side effect described in comment #4: some people close bugs or mark them as duplicate too quickly, resulting in more points for them but more effort for other developers who have to re-open the bugs later and sometimes deal with complaints from the bug reporter.
I am not sure that using labels such as "expert" or "regular contributor" would really discourage this bad behavior. On the other hand, adding this to the points formula would be a nice deterrent:
... - 1.5 * log2 (bugs reopened by others)
Unfortunately, "bugs reopened by others" is not easy to compute (according to my limited knowledge of bugzilla).
There is one more problem with descriptions. One time I've been on forum about subject I know well (I can't remember which). I've been first time and... next to my login was 'Newbie'. Points are more objective.
I think it could not be so easy with reopened bugs (it could seems to be closed for example and sbd find bug in patch).
I think the points formula should be adapted to also include the number of patches provided. This is probably quite easy to implement. The number of patches is in my opinion a good indicator of the quality of one's contributions.
log_10(1 + #comments) + log_2(1 + #bugs_closed) + log_2(1 + #bugs_reported)
+ log_2(1 + #patches_attached)
What do you think?
Daniel: That's an entirely separate issue, namely bug 350824. My comments on the matter can be found there.
I totally agreed with Daniel on adding #patches to the score. Think about how much time the submitter went through to create a patch. Definitely a lot more than adding comment and most likely more than reporting a bug. Not taking this into account is probably not acknowledging the efforts they put in.
This doesn't seem to be a significant enough problem to address, as it hasn't been enough of an issue in the *years* since this bug has been filed for anybody to do anything about it.