Scoring and Voting System’s Upcoming Changes

As I announced back in 2012, changes are coming to the site’s voting system. Since work on the updates has progressed enough to make things clear, I guess it’s time to update you all on the coming changes.

1 – The TPA system is going away. The system never lived up to its potential. The members that could use it effectively and correctly are very few and those who used it tended to ruin the results. So it became a waste of resources that delivered very little value for everyone. The existing votes will be converted to normal votes (what used to be basic).

2 – The updated system now keeps individual votes and that means new useful stuff and less drawbacks.

3 – Due to the way the updated system works, there will be no display of the old bScore. Only the weighed score (the bScore was already hidden from new authors).

So what useful stuff will result from #2? Well, now we can calculate a story’s score while dropping the top 5% and the bottom 5% of votes; thus eliminating outliers and minimizing the effects of vindictive/manipulative/destructive votes. This top and bottom drop will kick in once the story reaches 20 votes. So, I don’t want to hear anything anymore about votes of ‘1’. Those are taken care of with this change.

I’ve been keeping individual basic votes since 2009, so those, with the individual votes from the TPA, will make up the source for the scores in the updated system (millions of votes already). I’ll be dropping the old vote results completely. Theoretically, all the scores displayed for all the stories on the site will now reflect the same voting conditions. What that means is that the old stories’ scores most likely will change drastically. I haven’t completed enough work to make a comparison between the old scores and the new scores for the top performing classic stories, so I don’t really know what will happen yet.

The updated system will allow readers to change their votes. So no more “where is the voting form” except when authors turn off voting. And readers will be able to update their votes on serials as chapters get posted.

Update 2013-05-21: All the changes have been implemented.

Voting system. Again!

Although I swore I would never discuss scores again because I’ve done it literally hundreds of times since the end of 2006, I will attempt to clarify things here again since there is a lot of misinformation floating around.

So the purpose of this post is to keep this info available for the future.

First point of many: All scores are adjusted. All of them. And it’s scores, not votes. Individual votes are never touched. A story’s general score gets weighed by the system.

What makes it look like the above is not true is the fact that the older stories’ scores, the ones posted before December 19, 2006 (that’s when I brought the new system online), had ridiculously high scores, so high that even with the insane amount of weighing that these stories’ scores are subjected to, some of those scores are still unbelievably high compared to new scores.

The one reason that mandated the need for the new system is Human Nature!

One would think that a rating system for stories would be fairly simple; Get votes, average them out and be done with it. Sadly, that turns out to be very naïve. So how does human nature make this naïve?

Consider the following:

– Not all readers vote. So voters are self selective and not necessarily a real representation of the general readership.

– People are nice in general (unless they have an agenda). They don’t like to punish or discourage. The general tendency is to only vote when they like a story.

– People are lazy. They don’t vote on stories they don’t finish. To finish a story, they have to like it. So the general tendency is to get votes from those who liked the story in general.

– People are easily influenced. If most scores are higher than 8, then voting less than 8 becomes psychologically impossible for most people. How can you give a story a vote lower than the average when it’s fairly readable?

All the above would drive any system’s scores ever higher. It’s an unescapable trend.

Voting on SOL started way back in 1998 and went through few changes. By the end of 2006, the above described trend had pushed the scores on the site to a ridiculous level.

Consider this: the ‘average’ story, which on a scale of 1 – 10 should get theoretically a 6, was getting no less than 9. Most readers said that they never read anything below 9.25. Most readers started to think of the voting system as a silly joke. When I first checked the median of scores for the year 2006, I was shocked to find it around 9.15 (I don’t remember the exact number).

To have a median of 9.15 means that half of the stories are on a point spread of less than.6 points and the other half is on a spread of more than 8 points. That can’t be right, not logically anyway.

Something needed to be done. Fix the system or scrap it. No other choices.

So, after brainstorming for a while, I came up with the new system and after having it on the site for more than five years now, I’m convinced that it was the right choice, and considering all factors, the best system that could be used. Yes, it has flaws, I know them all. Some can be fixed (and will be), and some can’t (and must be lived with). But even with these few flaws, it’s still the best.

There is one negative to the system, and yes, only one negative. It pisses authors off.

Nobody likes anything that lowers their impression of their own work. So to have a system that lowers their scores is a major turn off. So, I allow authors to opt out of the scoring system selectively. They can opt out completely or partially or not at all.

The system’s calculations are actually quite simple. All it does is it shifts the median of the stories’ scores to 6. That sounds complicated, but it is actually quite simple. If you sort stories by their weighed score and by their average score you get the exact same lineup of stories.

So it gives no advantage to any story.

Why do I mention ‘stories posted in the same period’ often? it’s because such a system must consider the factors that affect how and why people vote.

Changing the voting form’s wording affects votes. Even the placement of the voting form affects votes. Before 2007, the voting form had nothing more than number choices from 1 – 10. After the end of 2006, the choices changed. So it wouldn’t be fair to have stories posted before and stories posted after the change be treated the same. Something needs to compensate for the change.

So, now whenever I change something, I start a new period. Later this year, when I scrap the TPA and start keeping individual votes, I’ll need to start a new period.

The system takes all the stories posted in each period and finds the median of their scores (the basic scores calculated by averaging all their votes). That median becomes the focal point of the calculations and the system recalculates the average scores to create weighed scores whose median is 6.

The brilliant thing about the current system is that it kills the psychological effect higher and higher scores and compensates for shifts in voting trends regardless of their source.

If votes trend higher for some reason, the system keeps the scores in check and doesn’t allow the psychological effect of bigger numbers to influence future votes.

And if votes trends lower, the same thing happens the other way around.

Again, the system works on the whole set of stories being posted. So high quality stories will still get high scores and low quality stories still get low scores and the readers’ collective opinion is still represented exactly the same. Which story is more liked by readers is preserved by keeping the same story lineup whether the listing is sorted by average score or by weighed score. Just the representation of numbers is affected.

Love them or hate them, weighed scores are here to stay.

A hint of things to come:

The TPA is going away. I may, I MAY, keep it for authors to use to give other authors feedback, but it will be separated from the score and will have no influence on it.

I will be keeping individual scores, so removing outlier votes from the results will become possible.

Few things to keep in mind:

SOL has competitors that send minions to do some sabotage. It’s often in the form of bombs of 1s.

Some people are irrational about certain subjects and will either give a lot of 10s or a lot of 1s to those stories that touch on those subjects.

If you need clarification about some points, please don’t hesitate to post your questions and comments here. I’ll do my best to respond and I’ll update the article with anything that may be missing.