New Review System and Other Notes

I read all the comments on the previous post. One important issue that most of those expressing their opinions keep forgetting: the voting system should serve reader goals as well as author goal. While authors view story scores as a kind of feedback – which it is, an author should not forget that the score should also help readers decide what to read and what not to read. I know it’s hard to think that there should exist a mechanism to help readers eliminate stories from their potential reading choices, but it is important.

As a reader, wouldn’t you want something to help you wade through the thousands of stories on the site? There is no perfect system, but for a site like storiesonline where readers are limited in how many stories they can access per 24 hours, some kind of help must be provided for them to find something to read without wasting their download allocations. Codes and scores serve that purpose. No reader can judge each and every story on the site for themselves.

Specific issues:

Individual chapter scores: Don’t work because the reader/voter has no real idea how a particular chapter fits within the grand scheme of a story, so while one chapter can be great fun to read, another can be hard to stomach, but necessary to the overall plot of the story. Readers should not be allowed to steer the direction of the story by a simple mechanism. An author should feel comfortable taking the story in any direction that they want without having to worry about the instant votes of readers that may not approve of the stuff in one or more chapters.

Author votes: Why shouldn’t an author vote for their own story? The president of the US votes for himself. Plus, those who worry about skewing the score, come on, the author can vote only once; after 20 more votes, the author’s vote becomes nearly irrelevant.

Thoughts on the Current Voting system

The current voting system, while not perfect, works well enough for most readers. But, authors looking for valid feedback from the score will be disappointed. The current score is merely an approval of the story in all facets combined by the readers’ collective.

I’m contemplating dropping the voting system to a lower scale, from 1 to 5. I’ve been analyzing votes for the last couple of days and it seems that some values are almost never used. It seems that voters don’t have enough to go on to judge in such small increments. I’m not sure about this step, but I’m contemplating it. And since the current voting system seems to be more of an approval system by readers, it would be better if the change was to be made, that the new scale would be of the ‘1 – Hated it’, ‘2 – Didn’t like it much’, ‘3 – It was OK’, ‘4 – Great fun’ and ‘5 – Absolutely Loved it’ variety. Of course choice of words would be thought out properly. Such a lower scale would be easier to understand by the voters, and authors won’t mistake it for the extensive feedback system that they wish it to be. But, since the current system works well enough, I’m not really sure that I will be making such a change.

The New Reviewing System

The decision to re-implement the reviewing system has been finalized and I’ve started the feature draft.

Every reader will be able to submit a review for stories. The review will have the user’s Login ID in it, so no anonymous reviews. That will cut down on those willing to submit a review, but since it’s all anonymous to begin with, it’s not really a step back in privacy, plus, anybody wanting to submit a review that affects the story and its author should be able to stand behind what they say. Also, since readers can change their Login ID, the old id and the new id will show if they do make such a change.

Reviews will have to be a comment (may have a size limit), and a three-criteria score. Plot and Character, Writing Quality and Appeal to Reviewer. There will be no stroke score; many stories have no sex in them so it shouldn’t be a factor in judging a story.

I’m not sure how I’m going to combine all three scores to come up with one review score to be displayed in story listings. Of course, all individual scores will be visible in expanded-review view. Should grammar be as important as plot and character? after all, an author can have multiple editors going over their work, so grammar score would not necessarily mean that the author has a great grasp of the english language. I’m thinking 25% for grammar (can be greatly enhance by an editor), 25% for appeal to reviewer (reflection on personal taste of the reviewer) and 50% for Plot and Character (that’s where the author’s talent shows)

There will be no reviewer profiles, it’s enough work to submit reviews, reviewers shouldn’t worry about identifying their tastes and fetishes (which can usually be deduced from their reviews anyway.)

Every author will be able to choose whether to allow reviews for their stories or not (the way it currently is, I know for a fact that some authors prefer to refuse reviews.) So if an author chooses to refuse reviews of their story, nobody would be able to submit any review for their stories. However, the author cannot choose to accept reviews on per story basis, it’s global on/off switch for reviews for all the author’s works, not per story setting.

Anybody can volunteer to be a review moderator, but it’s not automatic, they have to go through me to become a moderator.

A reader will be able to submit a review (it goes to moderation), edit an existing review (it goes to moderation before the changes take effect), delete an existing review that they have previously submitted and view the status and edit the reviews that they have submitted that are not on the site yet.

Each review will need to be approved by two moderators, so a moderator will be able to give first or second approval to a review, or reject a review. If a review is rejected by one moderator, it does not go to a second moderator and if it is accepted by the first moderator and rejected by a second one, then it does not go to a third moderator.

Authors cannot submit reviews of their own stories.

Authors who volunteer to be moderators cannot moderate reviews of their own stories.

Moderators cannot moderate their own reviews

Request for Suggestions

Of course, your suggestions are most welcome. If you think that I missed something important, don’t be shy.

Update – 2004-11-09: 3:15pm

• The voting system is going to stay the same. There is no need for a change.

• Each review must have two approvals or two rejections

 

Scoring System

I knew that as soon as I create this Blog, that the issue of scoring on the site will be brought up into the discussion of every article that I ever post.

So, I thought I would address the issue with my first actual article.

I hope, whether you’re an author, or a reader, you’ll read this article with an open mind and a bit of understanding. I’ve dealt with the subject for the longest time, since the scoring system started on the site in 1999. And rest assured, ever since the start, the subject is brought to my attention by an author or a reader a minimum of once a week, if not more. Needless to say, I’ve put quite a bit of thought into it over the years, and in this article, I’ll try to share the results of these thoughts.

The concept of scoring an artistic work like stories by people is a difficult one to tackle.

Ideally, scores on the site would enable any reader to take a story’s score as a measure of what to expect of the story in any facet. The score should tell how well the story is written, how well the story’s story is told, how extensive the plot is, etc… So theoretically, the score should help you pick out the best stories for reading and theoretically, it should be infallible.

From a reader’s point of view: the scores should tell him/her how much they’re GOING to enjoy the story if they decide to read it.

From the author’s point of view: the scoring mechanism should give him/her 10s and everybody else a 6 or lower. Of course, unless the author is trying to find a story to read, then the score should reflect this need, temporarily at least.

But the reality is, there could never be a way for a single score to reflect every facet of the story. There could never be a single way to account for different people’s tastes and education levels, let alone account for their personal fetishes. For example, some people like cheating wives stories and score them higher than anything else, others think those stories are an abomination and the issue should not be discussed without mentioning the stoning of the wife at the hand of the raging masses; so they always score any story that fails to do that with a 5 or less vote. Does that mean the story is not good? Does it means it’s not written well? Does the score take into account your personal stance on the genre?

History

I wanted to build that ideal system mentioned above. So the first iteration of the scoring mechanism, who’s existence is probably not remembered by anybody, was a reflection of the ‘Celestial Reviews’ scoring method. The reader gets to select three separate scores, one for each of the criteria: ‘Technical quality’, ‘Plot & Character’ and ‘Appeal to Reviewer’; the points range was from 1 to 10.

This mechanism stayed on the site for a whole week and, at the time, the site had about a thousand stories. During its existence for that whole week, the system gathered exactly 5 votes. Yes, you’ve read correctly, that’s ‘FIVE’ votes in total.

I can’t really tell why it was such a failure. Was it because it required too much work from the reader? Was it because most readers didn’t think they were qualified enough to make judgment on subjects like ‘Plot and Character’ and ‘Technical Quality’?

I don’t know, but I know it was a miserable failure, and it needed rethinking on my part to make it work; it required me to lower my expectations from the readers.

The result of the rework is the current system. As soon as it was online, scores came pouring in. So the participation problem was resolved.

Soon after, like within a week, I came to the realization that there is something extremely wrong with the scoring mechanism. Scores were fluctuating wildly, and it was evident that what I built at the time was a weak system, and it was being abused. At the time, the system that I used to run the site was very easy to use and implement, but very limited in what it allowed me to do. The only thing that I could do at the time is to track the last IP address that cast a vote for the story and stop them from voting again, but since it tracked only the last vote, when somebody else voted, the first voter could vote again. Not perfect, but better than the initial system. That system stayed online from the fall of 1999 to June of 2001. At the end of that period only 4 stories had a score higher than 9.

It took me that long to bring up my skill level to use a more powerful system. In June of 2001 I finished the re-implementation of the site using PHP and MySQL and I instituted user registration. Now, the site could tell who’s voting, and could keep track of who voted for what story and stopped them from voting again. So while most people think that I use the site to harvest email addresses to sell to spammers (which I don’t), I had to do it so that people couldn’t abuse the voting system easily; it’s not perfect, but it works well enough to make it a viable system.

The most recent change since then was about a month ago, when I changed the location of the voting mechanism to above the ‘To be Continued…’ or ‘The End’ line and changed the wording of the form itself. This recent change had a tremendous effect on the rate of participation. The rate of participation went up to almost double the rate of participation pre-change.

Remarks about the system

Inflation of scores

Many people are pissed about the seemingly artificial high scores, and have commented on how unrealistic they think they are.

Unfortunately, there is no easy way around this one. It has to do with human psychology. People tend to be nice and illogical simultaneously.

First, they don’t like hurting authors on purpose, so if they like the story, they give it a high score, and if they don’t like the story, many people would abstain from voting. I’ve been told by many people that they won’t vote at all if they don’t think that the story is good and deserves a 9 or 10. So the tendency for votes to be on the higher side of things can’t be avoided for this very peculiar phenomenon.

Second, they tend to be affected by existing scores for the story in question and the scores for other stories. So for example, if John Doe is reading the story ‘Mary Had a Little Caboose’ and the story has an existing score of 9.2. And John Doe had already read the story ‘House in the Mountains’, and it has a score of 9.5. John Doe liked ‘Mary Had a Little Caboose’ more than ‘House in the Mountains’, then his vote will be a 10 automatically, regardless of what he really thinks of the story ‘Mary Had a Little Caboose’ on its own; maybe a 9. He’s now looking at it in context of the other story’s score.

That effect causes an ever rising top score on the top list and cannot be broken unless new readers are introduced into the site, and they bring with them a fresh perspective.

Score Manipulation

I’ve often received messages from authors that somebody must be tampering with scores, or somebody has a grudge against them, and casting multiple low votes to lower their stories’ scores. While this is possible, it’s not possible at the scale that some people think.

When story scores change, they change because of valid votes 99% of the time. So if you’re an author and your story’s score drops, take it like a man and accept the fact that somebody doesn’t like your story. No story can please every person on every level.

Author and Reader Suggestions for Scoring Enhancements

Over the years, I’ve received many suggestions to what I can do to make the scoring system ‘better’.

  1. Forcing readers to vote: Does not work and would make the site less appealing. Nobody likes to be forced to do anything.
  2. Formulas that involve download counts, size of the story, number of chapters of the story, consistency of number of downloads from one chapter to another; They all don’t work. Most authors post their stories on other sites, so a reader can read one chapter here and one chapter on ASSTR and one chapter on EWP and another on ASSM. So the counts would be skewed and unreliable at best. Number of chapters? Well, how does that affect a long and repetitive story in comparison with a shorter story that is built tighter? How does the chapter length get counted? Can’t force a format on the author, so that is out of the window. Everybody knows that length and chapter count don’t correspond in any way, shape or form to the quality of the story. Every formula has an optimum variable combination, which would boost a story’s score to the max and that will have no real reflection on how good the story is. So we’re back to square one.
    Anything that somehow messes with the reader’s evaluation of the story is not acceptable, after all, the score should reflect the readers’ opinions and not a morphed version of it.
  3. Allowing people to change their scores. Well, it’s feasible, but very resource intensive and messes with the site’s ability to thwart manipulation. However, I implemented the system to allow a reader to vote again after 125 days, so you could cast a different vote if the serial degenerates over time or improves over time.
Conclusion

For now, the scoring mechanism is staying the same way it is. There will be no change, unless someone can come up with a scheme that somehow can read the minds of all those who’ve read the story reflects their thoughts and opinion of the story in perspective of the reader who’s checking the score and trying to decide whether to read or not, in one, easy-to-understand, decimal number.

Although, it doesn’t simply end here.

I need to change the reviewing system on the site to enable any reader to submit an extensive review of any story. My thoughts so far have explored re-implementing the reviewing system in parallel with a second score for stories that is derived from those reviews. So each story will have two scores: the existing score, let’s call it the popular score and a new score and let’s call it a reviewers’ score.

So far this idea has shown merit, and it beats all other ideas that’s I’ve had myself, or were suggested to me.

However, such a system presents its own challenges and limitations, so for now it’s in the analysis phase. It may or may not proceed into the planning and then implementation phases.

Update: 2004-11-01 : 11:25 am: The future reviewing system will be accessible to anybody (whenever it’s done). At the end of a story, the reader will have the current voting form and a link to submit a complete review.Reviews will be moderated, as in somebody have to approve the review before it shows up on the site and affect a story’s score; we don’t need reviews like ‘complete crap 1, 1, 1’ or ‘I liked it 10, 10, 10’, there is no point in those. The only way to really avoid silly or stupid reviews is to moderate them. Reviews should contain valid constructive criticism or a developed opinion.

No anonymous reviews, the reviewer’s user ID will show up on top of each review.

While the idea of restricting reviews to premier members does have merits, it would be too restrictive, and many of the qualified and willing to post people would be shutout. So the feature will be practically crippled; so no restriction on who can and cannot review.

New Spot Online

As you can see, I have a new blog online.

I know it sounds and looks a little strange for somebody with a site like storiesonline to have a blog on another service. I have good reasons to have this little blog here.

  1. I don’t want to add more load onto the site’s line, it’s already loaded enough.
  2. What I will be posting here may or may not have anything to do with going ons on the site.
  3. I don’t feel like creating a full blogging system with comments and everything that goes with it, the services that I want to use are all ready available here, and I do want a comments section for you to post replies and discuss things.

So, hopefully, this blog becomes more useful with time and a spot for interaction with me, something that is not really possible on storiesonline the way it is now (and which I don’t really want to put in the effort to change).