Well, it has been an interesting experience to say the least.
If you haven’t read my two most recent entries about changes to the scoring system on the site, you should read them.
The gist of this blog entry is to tell you that ALL changes to the scoring system have been canceled and reverted.
A recap of what happened.
For the last couple of years, story scores on the site have been edging up higher and higher due to various psychological reasons. The median for all scores of last year is about 8.87. Which means that half of the stories are scoring more than 8.87 and the other half below 8.87. The results of this median is that story scores are in general, extremely high.
Over the last year, I kept hearing from many authors how ridiculous the scores are becoming, and from readers how they can’t trust the system to guide them to the better stories. The phrase ‘I don’t read anything that is not scoring over 9.25’ came up very frequently.
So, because of these things, I set out to correct what’s causing these issues. I planned to do two things:
The first is to change the wording of the voting form to be more consistent and somewhat realistic of what those votes reflect, which is the reader’s feeling about the stories they read. The current wording is somewhat confusing, some grades are related to feelings, some others are related to the quality of the work. I seem to have made the grave mistake of wording the top score (10) entry in a way that few readers really could give a 10 but to the really best of stories (which logically is the way it should be).
The second was to create a different score evaluation system that took into consideration what date the story was posted and the average of all the stories posted in the same period of time. The new system would have taken care of the over-time creep up of scores and made the playing field between old and new, more level. It would have also leveled the playing field between older stories that were scored using the old wording and the newer stories scored using the new wording. There wouldn’t have been a large discrepancy between the meaning of older stories and newer ones.
This new valuation system would have shifted the median of scores down to a more logical median. My goal was to have a median of 6.
I guess I aimed too far down.
On December 1st, 2006, I announced the upcoming changes, and implemented the vote form wording change.
The wording change had its intended effect and stories posted after the change had a more moderate, more reasonable score average. Many seemed pleased with the change, although, I only received two approvals, one from an author and the other from a reader.
However, the opposition was far more vocal and came from authors. I received many worried notes and nearly ten authors asked for their stories to be removed if the score weighing were to take place.
Since I’m not in the business of pissing everybody off – and to me, ten vocal opposing people represent a 100 opposing silent ones – I canceled the score weighing system implementation.
I left the new wording in the forms and the newer stories continued to be scored lower on average. I received few notes from authors angry about unfairness of the lower scores and one demanded that I remove his stories from the site.
I agree that having lower average scores is not fair, considering that readers won’t really know or care why the older stories have higher score average than the newer ones. They’ll understand the drop as a drop in story quality. I even received one note from a reader asking why all the newer stories were crap (based on scores).
So, last night, I reversed the change in the voting form, and in an ironic twist, had to use the same code that I created to calculate a weighed scoring system to raise all the score of the stories posted after the wording change to a level more in line with the scores of stories posted before the change.
And now a week after the start of the changes, we’re back to the same old system that many mocked and distrusted and complained about. We’re back to a kindergarten-like system where everybody is a winner and everybody gets an A for their effort.
I don’t know what I’m going to do next, but I’ll think of something to keep the scores under control. I really don’t want to see the scores on the site compress any further and lowest scoring stories reach 9+ and high scoring stories reach 9.9+.
If anybody has any suggestions to make the system work better, without confusing the hell out of authors, I’m all ears. And please, no suggestions of a complicated score weighing system. I had the perfect one and it was rejected. If there were supporters for the new system that I planned and announced, they mostly kept it to themselves.
Update: Please see my following entry about the subject.