I canceled the planned changes. Too many authors asked for the removal of their stories from the site. Sorry for all the trouble.
This is a follow up to my previous blog entry about the changes to the scoring system. If you haven’t read that one, please check it to see what this whole thing is about.
This follow up is to address as many of the comments that have been received so far as possible.
A simple clarification: the voting system itself is not really changing. It still works the same way. It’s the results representation that’s changing to allow for a clearer distinction between tiny variations in the scores. I’m just shifting the median for all scores from whatever it is now, to an artificial one of 6. For example, the current top scoring serials on the site contains eight stories with a score of 9.77. The new representation would simply magnify smaller variations within the .77 bit.
As for suggestions offered, there were plenty, and that’s good.
Few things to clarify with regards to the nature of the site, to shed light on why some things are the way they are and why I can’t/won’t change some things related.
The site gets accessed from all over the world. In most places, internet access is not unlimited. Many, many readers pay for every minute that they spend on the site. So a large chunk of the story accesses are for quick downloads to read offline. Can’t force those readers to vote. Voting works only when reading a story while on line.
Many of those world wide readers don’t have English as their first language, hell, I don’t have English as a first language. More than half of the readers don’t feel and definitely aren’t qualified to judge the grammar and sentence composition of the text their reading. Can’t force them to cast a grammar vote. However, they can tell whether they like a story or not.
Things not doable:
* Forcing readers to vote: Not good.
Readers should never feel that they must vote. This action would cause a lot of junk voting. It would be worse than not voting.
* Forcing readers to comment: Not good.
5% of all readers vote and less than 1% actually comment; trying to force those numbers up will drive people away from the site. Not good.
* Dropping a certain percentage of votes from the top end and the bottom end: That would require keeping individual votes indefinitely. Not doable; requires too much resources.
Unless everybody is willing to chip in for a larger disk array for the site ($15,000 +) and for the cost of hosting it ($600 per month), then it is not possible to keep votes indefinitely.
* Allowing readers to change their vote later and allowing voting for stories previously downloaded: Not doable.
The site has a system in place for blocking score manipulation. Those changes would break it completely and make scores open to easy manipulation. That’s a bad thing.
* Changing voting method for an additional criteria like grammar: Not exactly fair.
Older stories that had their votes cast already would be at a severe disadvantage. Plus it would require readers to vote for multiple criteria.
* Disallowing votes for serials until their completed: Not fair.
Many authors rely on votes to give them motivation to write. No votes means way less feedback.
Plus, doing that would create even more bias towards serials. If scoring is only allowed after a story is completed, the only those who stuck with the story till then end would vote, which by default means they liked it and their votes would automatically be very high.
* Automatic vote casting for non voters: Not Good.
Since there is such a large difference between the number of downloads and the number of votes, casting a 6 for each non-voter means that the scores will never go above 6.5 or below 5.5, that’s even worse than it is now.
Things Doable:
Adding an additional voting panel for individual chapters. The results of this panel would be simply sent to the author, but not displayed on the site.
Adding an additional voting panel for grammar and stuff. The results of this panel would be simply sent to the author, but not displayed on the site.
—-
Thanks to Aleph Null’s suggestion. He provided the solution that I needed for the new system to be more fair for older stories. It’s so simple, I can’t believe that I didn’t think of it first. The new system will calculate the median for each year and then calculate each story’s weighed score depending on when it was posted or last updated.
—-
Everybody seems to think that I’m doing this as a spur of the moment thing. I’m not. I created the initial code more than a year ago. I knew it would piss a lot of authors off. After all, having your scores go down from 9+ to 7+ is a bummer.
I’ve been thinking about the issue and monitoring the median for the last year and a half. And now, the median has reached a ridiculous level. The effect of the extremely high average of the scores was evident in the comments posted. Many said that they don’t read anything that has a score below 9. Why is that?
From the authors comments, it was clear again that the authors’ expectations of the system are misplaced. Every author wants the system to be the equivalent of the film critics. Unfortunately, it’s not and it can never be. It’s like a poll at the door exit from the theatre.
Just look at the reviews. There are 30 people on the site able to post a review for a story. I would invite everybody to count how many reviews are submitted per month.
I tried the multi-criteria voting system in 1999 where it asked readers to rate three things: story line, quality and appeal. In its first week 8 votes were cast. Just Eight in a whole WEEK!
It was an abject failure.
People don’t want to think about it. It has to be a single easy choice. Anything other than that would be used by a slim minority of those already voting.
The new display method would be closer to showing what people are thinking instead of showing what they’re doing.
As for the ‘Impossible to improve’ option. I know it sounds ridiculous, but it’s on purpose.
The reason for it is best illustrated in Stormy Weather’s response:
Under the old system I rated stories 9s or 10s … and sometimes 8s. With the wording of the new system, the stories I read will be getting 9s and 8s and 7s. With the way 10 reads now, I can’t see myself giving it anymore… unless there’s something out there that really knocks me out of my chair.
The new wording is meant to keep the 10 for those who knock your socks off with their work. How do you really reward those authors that put so much work into their stories and have a great creativity that results in truly great story? Do you give them the same as you’re giving everybody else?
Is a 9.5 really meaningful when almost everybody is getting over 9.2?
I want people to stop and think for a bit before casting that vote.
And it seems that the new wording is being fairly effective. From a sampling of the most recently posted stories, the scores seem to be a bit more realistic.
The first two weeks after the new score display is implemented will be very rough, especially on me. I know I will hear about some extreme displeasure with what’s happening, and I’m definitely NOT looking forward to it.
We’ll all just need to get used to the new numbers. Lower our mental line in the sand for the new scores from 9 to 7 and everything will be fine soon.
Update:
I’ve been refining the system before full deployment and got some interesting numbers.
I’ve defined a set periods that the system will use to define which median value to apply to a story and got the following:
+------------+------------+--------+
| From Date | To Date | Median |
+------------+------------+--------+
| 1998-01-01 | 2001-07-01 | 8.25 |
| 2001-06-30 | 2003-01-01 | 8.44 |
| 2002-12-31 | 2004-01-01 | 8.60 |
| 2003-12-31 | 2005-01-01 | 8.65 |
| 2004-12-31 | 2006-01-01 | 8.93 |
| 2005-12-31 | 2006-12-01 | 8.87 |
| 2006-12-01 | 2008-01-01 | 8.33 |
+------------+------------+--------+
The 2001-07 date corresponds with when the system went from no-login anybody can vote as many times as they wished for any story, to the log in system where nobody could normally vote more than once per story.
The 2006-12-01 date corresponds with the wording change in the forms.
As you can see, there is a definite rise in the general voting over the years. the only anomaly is the difference between 2005 and 2006, the median dropped from 8.93 to 8.87. The explanation may not be obvious, but the drop corresponds with moving the form from below ‘the end’/to be continued’ line to above it and hiding the score of the story in the form so readers couldn’t readily compare the story’s existing score and didn’t have as much of an incentive to go higher.
The change is needed to future proof scores. Say an author posts a story this year that scores 9.7 which is pretty good now. If nothing is done, and the scores keep creeping up, this same story score, which now is pretty good, in two years would look lame compared to stories posted in two years.
At the current rate of score creep up, in a couple of years, stories that score below 9.25 would be crap and the top end would reach 9.9+.
If going unchecked, the scores will reach a level that everybody would be forced to vote a 10 for anything otherwise it would be below everything else at the time.
And for those worrying about the old scores, not to worry, internally, the system will still work as it is now. The same scores are kept internally, but they’ll be displayed in an interpretted way. I’m not sure yet, but I may show both scores in the authors stats pages along with the median that each story is compared against.