Voting system. Again!


Although I swore I would never discuss scores again because I’ve done it literally hundreds of times since the end of 2006, I will attempt to clarify things here again since there is a lot of misinformation floating around.

So the purpose of this post is to keep this info available for the future.

First point of many: All scores are adjusted. All of them. And it’s scores, not votes. Individual votes are never touched. A story’s general score gets weighed by the system.

What makes it look like the above is not true is the fact that the older stories’ scores, the ones posted before December 19, 2006 (that’s when I brought the new system online), had ridiculously high scores, so high that even with the insane amount of weighing that these stories’ scores are subjected to, some of those scores are still unbelievably high compared to new scores.

The one reason that mandated the need for the new system is Human Nature!

One would think that a rating system for stories would be fairly simple; Get votes, average them out and be done with it. Sadly, that turns out to be very naïve. So how does human nature make this naïve?

Consider the following:

– Not all readers vote. So voters are self selective and not necessarily a real representation of the general readership.

– People are nice in general (unless they have an agenda). They don’t like to punish or discourage. The general tendency is to only vote when they like a story.

– People are lazy. They don’t vote on stories they don’t finish. To finish a story, they have to like it. So the general tendency is to get votes from those who liked the story in general.

– People are easily influenced. If most scores are higher than 8, then voting less than 8 becomes psychologically impossible for most people. How can you give a story a vote lower than the average when it’s fairly readable?

All the above would drive any system’s scores ever higher. It’s an unescapable trend.

Voting on SOL started way back in 1998 and went through few changes. By the end of 2006, the above described trend had pushed the scores on the site to a ridiculous level.

Consider this: the ‘average’ story, which on a scale of 1 – 10 should get theoretically a 6, was getting no less than 9. Most readers said that they never read anything below 9.25. Most readers started to think of the voting system as a silly joke. When I first checked the median of scores for the year 2006, I was shocked to find it around 9.15 (I don’t remember the exact number).

To have a median of 9.15 means that half of the stories are on a point spread of less than.6 points and the other half is on a spread of more than 8 points. That can’t be right, not logically anyway.

Something needed to be done. Fix the system or scrap it. No other choices.

So, after brainstorming for a while, I came up with the new system and after having it on the site for more than five years now, I’m convinced that it was the right choice, and considering all factors, the best system that could be used. Yes, it has flaws, I know them all. Some can be fixed (and will be), and some can’t (and must be lived with). But even with these few flaws, it’s still the best.

There is one negative to the system, and yes, only one negative. It pisses authors off.

Nobody likes anything that lowers their impression of their own work. So to have a system that lowers their scores is a major turn off. So, I allow authors to opt out of the scoring system selectively. They can opt out completely or partially or not at all.

The system’s calculations are actually quite simple. All it does is it shifts the median of the stories’ scores to 6. That sounds complicated, but it is actually quite simple. If you sort stories by their weighed score and by their average score you get the exact same lineup of stories.

So it gives no advantage to any story.

Why do I mention ‘stories posted in the same period’ often? it’s because such a system must consider the factors that affect how and why people vote.

Changing the voting form’s wording affects votes. Even the placement of the voting form affects votes. Before 2007, the voting form had nothing more than number choices from 1 – 10. After the end of 2006, the choices changed. So it wouldn’t be fair to have stories posted before and stories posted after the change be treated the same. Something needs to compensate for the change.

So, now whenever I change something, I start a new period. Later this year, when I scrap the TPA and start keeping individual votes, I’ll need to start a new period.

The system takes all the stories posted in each period and finds the median of their scores (the basic scores calculated by averaging all their votes). That median becomes the focal point of the calculations and the system recalculates the average scores to create weighed scores whose median is 6.

The brilliant thing about the current system is that it kills the psychological effect higher and higher scores and compensates for shifts in voting trends regardless of their source.

If votes trend higher for some reason, the system keeps the scores in check and doesn’t allow the psychological effect of bigger numbers to influence future votes.

And if votes trends lower, the same thing happens the other way around.

Again, the system works on the whole set of stories being posted. So high quality stories will still get high scores and low quality stories still get low scores and the readers’ collective opinion is still represented exactly the same. Which story is more liked by readers is preserved by keeping the same story lineup whether the listing is sorted by average score or by weighed score. Just the representation of numbers is affected.

Love them or hate them, weighed scores are here to stay.

A hint of things to come:

The TPA is going away. I may, I MAY, keep it for authors to use to give other authors feedback, but it will be separated from the score and will have no influence on it.

I will be keeping individual scores, so removing outlier votes from the results will become possible.

Few things to keep in mind:

SOL has competitors that send minions to do some sabotage. It’s often in the form of bombs of 1s.

Some people are irrational about certain subjects and will either give a lot of 10s or a lot of 1s to those stories that touch on those subjects.

If you need clarification about some points, please don’t hesitate to post your questions and comments here. I’ll do my best to respond and I’ll update the article with anything that may be missing.

 


9 responses to “Voting system. Again!”

  1. Lazeez, the “focal point” is the median score of stories *posted in each period*.
    I have two questions regarding the period:
    1. What is the period? Is it one year? Is it the entire time from 2007 to now?
    2. For a multi-chapter story, is the “period” calculated off the first chapter? The last chapter? Or the last update of any chapter?

    • @Sam
      I try to define a period as a stretch of time when voting conditions were similar.

      The current set of periods is as follows:

      1998-01-01 -> 2001-07-01: 8.18 (no log in requirement)
      2001-06-30 -> 2003-01-01: 8.27
      2002-12-31 -> 2004-01-01: 8.40
      2003-12-31 -> 2005-01-01: 8.46
      2004-12-31 -> 2006-01-01: 8.75
      2005-12-31 -> 2006-12-19: 8.56
      2006-12-18 -> 2009-01-01: 8.00 (started with the new system)
      2008-12-31 -> 2015-01-01: 7.93

      The periods between the first one and the new system period are divided by years as the whole period was marked by an ever increasing score median.

      For all stories, the system uses the posting date as where the story falls. Usually, most votes are received at the beginning. But even later votes are subject to the same conditions as later stories. And since the voting condition changes affect older stories too, the medians are updated once per day.

      • I feel like a broken record, but here it is again:

        2003-12-31 -> 2005-01-01: 8.46
        2004-12-31 -> 2006-01-01: 8.75
        2005-12-31 -> 2006-12-19: 8.56
        2006-12-18 -> 2009-01-01: 8.00 (started with the new system)
        2008-12-31 -> 2015-01-01: 7.93

        That’s all I’ve ever said. There is a dramatic drop in score from the time the system changed until today. Why is it so hard to just admit that? The whys, what for’s and why not’s don’t matter. It just is what it is.

        What people are doing is arguing the reality of the figures you posted. It’s there, right there to see. Why are we arguing about it?

        Ezzy

  2. When you remove tpa are you going to convert the a scores into votes?. I have a tpa only story and I would hate to lose all the ratings ive had.

    John

  3. G’day Lazeez,

    Thanks for the clarification of the scoring system. I suspect the dropping of the TPA will make some unhappy, but make more happier.

    I must admit I find it as a good indicator of having a story that people like but may have a technical issue that I could fix in a revision. I have noticed there is a correlation between a revision fixing errors and an improvement in the technical score on the votes after the revision is up. I could be wrong, but I suspect the ones who switch to use the technical score are those who have a better understanding of the technical aspects of the language and writing, even if some of them have a few misconceptions and get peeved when I use a style of writing different to what they’re used to – ie. present tense first person as against past tense third person.

    Anyway, I’m not wedded to any system and use the score simply as a basic guide to see if people like the story or not. I find the downloads and the feedback to be a stronger indication of how the story is received.

    Regards,

    Ernest

  4. I’m OK with the changes in general. I think they’ll be an improvement.

    But I have a suggestion for improvement that is along a different line from what I’ve heard discussed. I think a source of confusion is that readers rate on a scale of 1-10, and the scores that are reported are also on a scale of 1-10, but *they are really quite different*. The authors are of course annoyed to see an average story by reverse lookup evaluated as “not bad” (what 6 stands for). Readers would also be confused, saying, “Hey, I thought that story was really pretty good, why does everyone else say “not bad”? I would suggest that rather than a median of 6.00, you should switch to an entirely different numbering system that can’t be confused with the first. I say, make your new center 1000. Roughly:
    6.00 -> 1000
    5.00 -> 900
    4.00 -> 800
    7.00 -> 1100
    8.00 -> 1200
    etc.

    Now you have the natural 1000 rather than the mysterious 6.0. And you’re reflecting the fact that there is no relationship between “not bad” and 6.0 — because there hasn’t been, since 2006! You can provide that conversion table to people who are trying to compare old and new scores.
    (Another option would be scores centered around zero, but that probably doesn’t meet human psychology very well, since a negative score shouldn’t actually be taken to mean it’s of negative value and not worth reading).

    Another request: Since you’re going to be keeping individual scores, it would be great if the authors could see a count of how many votes they got of each score — a vector of 10 counts, (plus a summary of old information saying something like “129 votes averaging 6.15”). At the very least I hope the authors will still be able to see the bScore as we can today, to allow conclusions such as “most people rated my story between ‘very good’ and ‘great’”.

    • @Sterling:
      The author statistics page already provides this vector. You see it if you click on the TPA score on the statistics page.

      And, I would prefer to leave the scores on the existing 1-10 range. It’s not that any other scale is inherently wrong, but that each one requires a mental adjustment — I am already used to the 1..10 one.

  5. Uh, Purvv: dotB did give his pen name…Now, as to the points he raised: I actually agree with some of them. I mean, I know that my votes mean less, under the new system… but then, they meant next to nothing under the old system, too. Because people voting wouldn’t bother to think about any actually criteria, and hand out 10s to everything, even piles of unmitigated crap. Hopefully under the new system, that won’t happen as much… but we shall see. It’s disturbing that I’ve seen so many 10s in the new scores, honestly. The QScores are doing, however, what they were intended to do: ensure that only rare stories get 9 scores. Which is as it should be.