Although I swore I would never discuss scores again because I’ve done it literally hundreds of times since the end of 2006, I will attempt to clarify things here again since there is a lot of misinformation floating around.
So the purpose of this post is to keep this info available for the future.
First point of many: All scores are adjusted. All of them. And it’s scores, not votes. Individual votes are never touched. A story’s general score gets weighed by the system.
What makes it look like the above is not true is the fact that the older stories’ scores, the ones posted before December 19, 2006 (that’s when I brought the new system online), had ridiculously high scores, so high that even with the insane amount of weighing that these stories’ scores are subjected to, some of those scores are still unbelievably high compared to new scores.
The one reason that mandated the need for the new system is Human Nature!
One would think that a rating system for stories would be fairly simple; Get votes, average them out and be done with it. Sadly, that turns out to be very naïve. So how does human nature make this naïve?
Consider the following:
– Not all readers vote. So voters are self selective and not necessarily a real representation of the general readership.
– People are nice in general (unless they have an agenda). They don’t like to punish or discourage. The general tendency is to only vote when they like a story.
– People are lazy. They don’t vote on stories they don’t finish. To finish a story, they have to like it. So the general tendency is to get votes from those who liked the story in general.
– People are easily influenced. If most scores are higher than 8, then voting less than 8 becomes psychologically impossible for most people. How can you give a story a vote lower than the average when it’s fairly readable?
All the above would drive any system’s scores ever higher. It’s an unescapable trend.
Voting on SOL started way back in 1998 and went through few changes. By the end of 2006, the above described trend had pushed the scores on the site to a ridiculous level.
Consider this: the ‘average’ story, which on a scale of 1 – 10 should get theoretically a 6, was getting no less than 9. Most readers said that they never read anything below 9.25. Most readers started to think of the voting system as a silly joke. When I first checked the median of scores for the year 2006, I was shocked to find it around 9.15 (I don’t remember the exact number).
To have a median of 9.15 means that half of the stories are on a point spread of less than.6 points and the other half is on a spread of more than 8 points. That can’t be right, not logically anyway.
Something needed to be done. Fix the system or scrap it. No other choices.
So, after brainstorming for a while, I came up with the new system and after having it on the site for more than five years now, I’m convinced that it was the right choice, and considering all factors, the best system that could be used. Yes, it has flaws, I know them all. Some can be fixed (and will be), and some can’t (and must be lived with). But even with these few flaws, it’s still the best.
There is one negative to the system, and yes, only one negative. It pisses authors off.
Nobody likes anything that lowers their impression of their own work. So to have a system that lowers their scores is a major turn off. So, I allow authors to opt out of the scoring system selectively. They can opt out completely or partially or not at all.
The system’s calculations are actually quite simple. All it does is it shifts the median of the stories’ scores to 6. That sounds complicated, but it is actually quite simple. If you sort stories by their weighed score and by their average score you get the exact same lineup of stories.
So it gives no advantage to any story.
Why do I mention ‘stories posted in the same period’ often? it’s because such a system must consider the factors that affect how and why people vote.
Changing the voting form’s wording affects votes. Even the placement of the voting form affects votes. Before 2007, the voting form had nothing more than number choices from 1 – 10. After the end of 2006, the choices changed. So it wouldn’t be fair to have stories posted before and stories posted after the change be treated the same. Something needs to compensate for the change.
So, now whenever I change something, I start a new period. Later this year, when I scrap the TPA and start keeping individual votes, I’ll need to start a new period.
The system takes all the stories posted in each period and finds the median of their scores (the basic scores calculated by averaging all their votes). That median becomes the focal point of the calculations and the system recalculates the average scores to create weighed scores whose median is 6.
The brilliant thing about the current system is that it kills the psychological effect higher and higher scores and compensates for shifts in voting trends regardless of their source.
If votes trend higher for some reason, the system keeps the scores in check and doesn’t allow the psychological effect of bigger numbers to influence future votes.
And if votes trends lower, the same thing happens the other way around.
Again, the system works on the whole set of stories being posted. So high quality stories will still get high scores and low quality stories still get low scores and the readers’ collective opinion is still represented exactly the same. Which story is more liked by readers is preserved by keeping the same story lineup whether the listing is sorted by average score or by weighed score. Just the representation of numbers is affected.
Love them or hate them, weighed scores are here to stay.
A hint of things to come:
The TPA is going away. I may, I MAY, keep it for authors to use to give other authors feedback, but it will be separated from the score and will have no influence on it.
I will be keeping individual scores, so removing outlier votes from the results will become possible.
Few things to keep in mind:
SOL has competitors that send minions to do some sabotage. It’s often in the form of bombs of 1s.
Some people are irrational about certain subjects and will either give a lot of 10s or a lot of 1s to those stories that touch on those subjects.
If you need clarification about some points, please don’t hesitate to post your questions and comments here. I’ll do my best to respond and I’ll update the article with anything that may be missing.