As the final test for the new digs, we have moved Finestories.com to the new server farm. It seems to be working well enough.
Our tentative schedule to move SOL is now set for tonight. Keeping fingers crossed.
As the final test for the new digs, we have moved Finestories.com to the new server farm. It seems to be working well enough.
Our tentative schedule to move SOL is now set for tonight. Keeping fingers crossed.
Since the Sunday right before Labor day monday, the sites have been slow. And today, Sunday the 9th or September, the whole server farm went offline from 1:00am EDT ’til 9:40am EDT. Even now that the sites are online, things are flaky with lots of slowdowns and dropped connections.
After much investigation, we found that our oldest two servers (circa 2003) have given up the ghost and the rest are not keeping up with demand. It seems that the electric power in the facilities have something to do with the issue.
Here is what we will do to remedy the situation:
This week sometime, the servers will be moved to a different facility and the dead servers will be replaced. We don’t know the exact time of the move, but it will be in the coming few days. We will make an announcement few hours before the actual move and as soon as we have a firm time.
During the few hours before the move, the site will be in limited functionality. No new postings and no payments accepted.
During the move, the site will be completely offline, and it will last for a couple of hours at least. After that, everything should be back to normal.
Meanwhile, until that happens, the server farm has decreased in capacity, so expect lower performance from all the site that we have, including SOL and Finestories.
We apologize for any inconvenience that this will have. Hopefully, we’ll restore the service to a better than ever shape.
Although I swore I would never discuss scores again because I’ve done it literally hundreds of times since the end of 2006, I will attempt to clarify things here again since there is a lot of misinformation floating around.
So the purpose of this post is to keep this info available for the future.
First point of many: All scores are adjusted. All of them. And it’s scores, not votes. Individual votes are never touched. A story’s general score gets weighed by the system.
What makes it look like the above is not true is the fact that the older stories’ scores, the ones posted before December 19, 2006 (that’s when I brought the new system online), had ridiculously high scores, so high that even with the insane amount of weighing that these stories’ scores are subjected to, some of those scores are still unbelievably high compared to new scores.
The one reason that mandated the need for the new system is Human Nature!
One would think that a rating system for stories would be fairly simple; Get votes, average them out and be done with it. Sadly, that turns out to be very naïve. So how does human nature make this naïve?
Consider the following:
– Not all readers vote. So voters are self selective and not necessarily a real representation of the general readership.
– People are nice in general (unless they have an agenda). They don’t like to punish or discourage. The general tendency is to only vote when they like a story.
– People are lazy. They don’t vote on stories they don’t finish. To finish a story, they have to like it. So the general tendency is to get votes from those who liked the story in general.
– People are easily influenced. If most scores are higher than 8, then voting less than 8 becomes psychologically impossible for most people. How can you give a story a vote lower than the average when it’s fairly readable?
All the above would drive any system’s scores ever higher. It’s an unescapable trend.
Voting on SOL started way back in 1998 and went through few changes. By the end of 2006, the above described trend had pushed the scores on the site to a ridiculous level.
Consider this: the ‘average’ story, which on a scale of 1 – 10 should get theoretically a 6, was getting no less than 9. Most readers said that they never read anything below 9.25. Most readers started to think of the voting system as a silly joke. When I first checked the median of scores for the year 2006, I was shocked to find it around 9.15 (I don’t remember the exact number).
To have a median of 9.15 means that half of the stories are on a point spread of less than.6 points and the other half is on a spread of more than 8 points. That can’t be right, not logically anyway.
Something needed to be done. Fix the system or scrap it. No other choices.
So, after brainstorming for a while, I came up with the new system and after having it on the site for more than five years now, I’m convinced that it was the right choice, and considering all factors, the best system that could be used. Yes, it has flaws, I know them all. Some can be fixed (and will be), and some can’t (and must be lived with). But even with these few flaws, it’s still the best.
There is one negative to the system, and yes, only one negative. It pisses authors off.
Nobody likes anything that lowers their impression of their own work. So to have a system that lowers their scores is a major turn off. So, I allow authors to opt out of the scoring system selectively. They can opt out completely or partially or not at all.
The system’s calculations are actually quite simple. All it does is it shifts the median of the stories’ scores to 6. That sounds complicated, but it is actually quite simple. If you sort stories by their weighed score and by their average score you get the exact same lineup of stories.
So it gives no advantage to any story.
Why do I mention ‘stories posted in the same period’ often? it’s because such a system must consider the factors that affect how and why people vote.
Changing the voting form’s wording affects votes. Even the placement of the voting form affects votes. Before 2007, the voting form had nothing more than number choices from 1 – 10. After the end of 2006, the choices changed. So it wouldn’t be fair to have stories posted before and stories posted after the change be treated the same. Something needs to compensate for the change.
So, now whenever I change something, I start a new period. Later this year, when I scrap the TPA and start keeping individual votes, I’ll need to start a new period.
The system takes all the stories posted in each period and finds the median of their scores (the basic scores calculated by averaging all their votes). That median becomes the focal point of the calculations and the system recalculates the average scores to create weighed scores whose median is 6.
The brilliant thing about the current system is that it kills the psychological effect higher and higher scores and compensates for shifts in voting trends regardless of their source.
If votes trend higher for some reason, the system keeps the scores in check and doesn’t allow the psychological effect of bigger numbers to influence future votes.
And if votes trends lower, the same thing happens the other way around.
Again, the system works on the whole set of stories being posted. So high quality stories will still get high scores and low quality stories still get low scores and the readers’ collective opinion is still represented exactly the same. Which story is more liked by readers is preserved by keeping the same story lineup whether the listing is sorted by average score or by weighed score. Just the representation of numbers is affected.
Love them or hate them, weighed scores are here to stay.
A hint of things to come:
The TPA is going away. I may, I MAY, keep it for authors to use to give other authors feedback, but it will be separated from the score and will have no influence on it.
I will be keeping individual scores, so removing outlier votes from the results will become possible.
Few things to keep in mind:
SOL has competitors that send minions to do some sabotage. It’s often in the form of bombs of 1s.
Some people are irrational about certain subjects and will either give a lot of 10s or a lot of 1s to those stories that touch on those subjects.
If you need clarification about some points, please don’t hesitate to post your questions and comments here. I’ll do my best to respond and I’ll update the article with anything that may be missing.
I’ve made an important change to the site today. Tonight will be the last night that weekly counters are all reset to zero. From now on the system will keep track of the last seven days individually for each story.
So, the weekly counters will now reflect a story’s downloads in the last seven days regardless of which day you view the listings.
Previously, the site kept track of weekly downloads starting from the weekly reset time of Sunday night. So, if you view the download listings on Tuesday, you get the tally of two days’ worth of downloads. With the new system, viewing the top downloads list on Tuesday will give you the cumulative downloads since last Wednesday.
The current system had the effect that any story posted around midnight on Sunday, had the biggest chance of staying on top of the download list of the rest of the week. Some authors took advantage of this by posting their updates and new stories on Sunday after 9pm EST; which of course is completely understandable, who wouldn’t want to have the biggest possible advantage.
With the new system, that changes completely and closes a long standing loophole. Posting time has no advantage at all anymore. And posting in the middle of the week has no more disadvantage.
The full effect of this change will only be felt next Monday. The system doesn’t have the daily stats for the last 7 days. It starts counting tonight. For this week, the weekly downloads will behave exactly like the previous system.
I want to thank everybody that commented and contributed to the discussions in the three previous blog entries, and who made suggestions. It has been educational.
I’ve made up my mind on how to proceed next with regards to the voting system.
While I don’t like to lose any authors or readers, the current state of the scoring system is pitiful, and if left as it currently is, then it will only get worse over time, which, to me, is unacceptable.
The suggestions by many were interesting to say the least, but a lot of it, while could be useful and helpful, is not feasible to implement. Everybody has to remember that the site is a busy one and I don’t have unlimited resources. The site serves millions of pages per day and any change in the page, in the processing, in the data stored can have a huge effect on the site’s performance and its ability to cope.
For example, keeping scores indefinitely, in order to allow for dropping a certain percentage of votes, or comparing previous voting patterns for each reader, is not possible. While the data stored could be relatively small, the cumulative numbers are huge. And it’s not just about storing the info; processing power needed to handle all that grows exponentially. Searching a database of millions upon millions of data rows is expensive in processing power. Also, backing up the data and shuttling it around the net for off-site backup gets really expensive, really fast.
Also, anything that requires processing of each bit of info on the fly is also a no-no. For example, the suggestion that the story’s highlight color changing with the score requires the evaluation of every story’s score against the median for each page displayed. That requires multiple IFs for each row of each table. The site serves over a million listings page per day, with each page having 10 rows or 20, that means for each additional IF that I add to the process, the server farm has to execute it 10 or 20 million times per day, that means each listings page would need more than double what it already needs, that means I would need at least double the processing power that the site uses now and that’s without taking future growth into consideration.
So, any solution would need to be simple, fast to process, easy on storage and easy on processing repetitions to be acceptable.
I understand what the system needs to do, and I understand what it takes to do the things that needs to be done. And, I’m the only one to know what it costs to do each little additional thing.
So, in order to balance the needs of the authors, with the needs of the readers and the resources available, I’ve decided to implement the following changes:
1 – A rewording of the current voting form. Not as drastically different as the previous rewording, but something more sensible. The change will be implemented in two stages. First stage the wording will be changed while keeping the number values associated with each description. 90 days later, the numbers will be removed and the voting form will have descriptions only.
2 – The previously noted vote weighing system will be implemented (it is very necessary), however, both scores will be displayed side by side for a whole year. In the first six months, the listings pages ordered by score will use the old score, and then in the next six month it will be sorted by the weighed score. After a year, the average score will not be shown anymore (remember, displaying both scores requires more bandwidth).
3 – The voting form will have two optional variations. The current reworded form will be the default one. Readers will have the option of using a more elaborate form that has three separate criteria to judge: Plot, Quality and Appeal.
Which form to see in the story by default will be an option in each user’s preferences and will have a switch in the form itself.
I haven’t decided how I’m going to display the scores from the elaborate forms in the listings pages yet, or how it should affect the sorting in the listings pages ordered by score.
Two options in that regard, display the average of each value or have a single combined representation. We’ll see.
In the beginning, I will keep all the scores from the elaborate voting forms. Depending on how many people use them, I will decide later if I can afford to keep that data indefinitely or not. Users who choose to use the elaborate voting forms will be able to change their votes as long as the votes are kept.
Authors will be able to view how many of each vote their stories received.
Chapter voting, I’m not sure I’ll implement it because of abuse possibilities. To stop abuse, I must track some data from each vote for each chapter. With over a hundred thousands possible users, combined with over 50,000 individual chapters currently on the site, the size of the data can be huge.
On the other hand, individual chapter voting can be done for authors’ eyes only and each reader can vote as many times as they like for each chapter, which in return, makes chapter voting almost useless, but if authors really want it, then it can be done and please don’t suggest to see vote value distribution, that would require keeping a lot more data.
Eventually, I will implement the capability to allow paying members to vote after the fact. They can download stories to read offline, so they should be able to vote on their next visit to the site for the stories that they downloaded in their previous session.
No mandatory voting. No mandatory comments while voting. No default voting for non voters. Non-voters outnumber voters 20 to 1, no matter how wild the actual voting is, it wouldn’t make a blip if non voters were counted as average. Let’s say the default vote is a 6, that means no story could score higher than 6.19 (with all 10s) or less than 5.76 (with all 1s).
As they say, hindsight is 20/20.
I now realize that the changes proposed previously were too much too fast. Hopefully, with the new gradual phasing in approach, people, both readers and authors can acclimate to the coming changes.
This is a follow up to my previous blog entry about the changes to the scoring system. If you haven’t read that one, please check it to see what this whole thing is about.
This follow up is to address as many of the comments that have been received so far as possible.
A simple clarification: the voting system itself is not really changing. It still works the same way. It’s the results representation that’s changing to allow for a clearer distinction between tiny variations in the scores. I’m just shifting the median for all scores from whatever it is now, to an artificial one of 6. For example, the current top scoring serials on the site contains eight stories with a score of 9.77. The new representation would simply magnify smaller variations within the .77 bit.
As for suggestions offered, there were plenty, and that’s good.
Few things to clarify with regards to the nature of the site, to shed light on why some things are the way they are and why I can’t/won’t change some things related.
The site gets accessed from all over the world. In most places, internet access is not unlimited. Many, many readers pay for every minute that they spend on the site. So a large chunk of the story accesses are for quick downloads to read offline. Can’t force those readers to vote. Voting works only when reading a story while on line.
Many of those world wide readers don’t have English as their first language, hell, I don’t have English as a first language. More than half of the readers don’t feel and definitely aren’t qualified to judge the grammar and sentence composition of the text their reading. Can’t force them to cast a grammar vote. However, they can tell whether they like a story or not.
Things not doable:
* Forcing readers to vote: Not good.
Readers should never feel that they must vote. This action would cause a lot of junk voting. It would be worse than not voting.
* Forcing readers to comment: Not good.
5% of all readers vote and less than 1% actually comment; trying to force those numbers up will drive people away from the site. Not good.
* Dropping a certain percentage of votes from the top end and the bottom end: That would require keeping individual votes indefinitely. Not doable; requires too much resources.
Unless everybody is willing to chip in for a larger disk array for the site ($15,000 +) and for the cost of hosting it ($600 per month), then it is not possible to keep votes indefinitely.
* Allowing readers to change their vote later and allowing voting for stories previously downloaded: Not doable.
The site has a system in place for blocking score manipulation. Those changes would break it completely and make scores open to easy manipulation. That’s a bad thing.
* Changing voting method for an additional criteria like grammar: Not exactly fair.
Older stories that had their votes cast already would be at a severe disadvantage. Plus it would require readers to vote for multiple criteria.
* Disallowing votes for serials until their completed: Not fair.
Many authors rely on votes to give them motivation to write. No votes means way less feedback.
Plus, doing that would create even more bias towards serials. If scoring is only allowed after a story is completed, the only those who stuck with the story till then end would vote, which by default means they liked it and their votes would automatically be very high.
* Automatic vote casting for non voters: Not Good.
Since there is such a large difference between the number of downloads and the number of votes, casting a 6 for each non-voter means that the scores will never go above 6.5 or below 5.5, that’s even worse than it is now.
Adding an additional voting panel for individual chapters. The results of this panel would be simply sent to the author, but not displayed on the site.
Adding an additional voting panel for grammar and stuff. The results of this panel would be simply sent to the author, but not displayed on the site.
Thanks to Aleph Null’s suggestion. He provided the solution that I needed for the new system to be more fair for older stories. It’s so simple, I can’t believe that I didn’t think of it first. The new system will calculate the median for each year and then calculate each story’s weighed score depending on when it was posted or last updated.
Everybody seems to think that I’m doing this as a spur of the moment thing. I’m not. I created the initial code more than a year ago. I knew it would piss a lot of authors off. After all, having your scores go down from 9+ to 7+ is a bummer.
I’ve been thinking about the issue and monitoring the median for the last year and a half. And now, the median has reached a ridiculous level. The effect of the extremely high average of the scores was evident in the comments posted. Many said that they don’t read anything that has a score below 9. Why is that?
From the authors comments, it was clear again that the authors’ expectations of the system are misplaced. Every author wants the system to be the equivalent of the film critics. Unfortunately, it’s not and it can never be. It’s like a poll at the door exit from the theatre.
Just look at the reviews. There are 30 people on the site able to post a review for a story. I would invite everybody to count how many reviews are submitted per month.
I tried the multi-criteria voting system in 1999 where it asked readers to rate three things: story line, quality and appeal. In its first week 8 votes were cast. Just Eight in a whole WEEK!
It was an abject failure.
People don’t want to think about it. It has to be a single easy choice. Anything other than that would be used by a slim minority of those already voting.
The new display method would be closer to showing what people are thinking instead of showing what they’re doing.
As for the ‘Impossible to improve’ option. I know it sounds ridiculous, but it’s on purpose.
The reason for it is best illustrated in Stormy Weather’s response:
Under the old system I rated stories 9s or 10s … and sometimes 8s. With the wording of the new system, the stories I read will be getting 9s and 8s and 7s. With the way 10 reads now, I can’t see myself giving it anymore… unless there’s something out there that really knocks me out of my chair.
The new wording is meant to keep the 10 for those who knock your socks off with their work. How do you really reward those authors that put so much work into their stories and have a great creativity that results in truly great story? Do you give them the same as you’re giving everybody else?
Is a 9.5 really meaningful when almost everybody is getting over 9.2?
I want people to stop and think for a bit before casting that vote.
And it seems that the new wording is being fairly effective. From a sampling of the most recently posted stories, the scores seem to be a bit more realistic.
The first two weeks after the new score display is implemented will be very rough, especially on me. I know I will hear about some extreme displeasure with what’s happening, and I’m definitely NOT looking forward to it.
We’ll all just need to get used to the new numbers. Lower our mental line in the sand for the new scores from 9 to 7 and everything will be fine soon.
Since implementing the scoring system on the site in 1999, it has been the only controversial part of the whole site. Many users found it useful, some didn’t trust it and some ignored it.
To authors it’s even more troublesome. Many authors expect the score to tell them how they did in their writing, they wanted it to reflect the effort that they put into the story regardless of the story’s content and its subject’s appeal to the readers.
Those concerns and expectations are not something that I can really address. Authors need to simply realize that the score simply reflects how much a reader liked the story and whether they recommend that others read it too. It’s like a thumbs up signal.
However, there is a problem with the scoring system that I can address: Score compression.
Score compression is when votes, like they are now on the site, tend to be mostly on one end of the scale. Last check revealed that the median for all scores on the site is 8.62!
A median of 8.62 means that half the stories on the site have a score of 8.62 and more. That means about 8000 stories have about 1.2 points spread. That means anything below 9 didn’t get a good score. 8.62 is so close to the top, it’s making scores meaningless.
The reasons for this compression are multiple.
So, I’m introducing two changes to the system to be rolled out gradually.
The first change is the wording accompanying the number scores in the vote form and I’m removing the numbers. I’m proposing the following as the new list:
Amazing; Impossible to Improve
Some Good, Some Bad
You Call this a Story!?
This way it’s not mixed signals. The old list was a bit misleading to authors as it implied that the score may represent the readers’ judgement on how well the story is written. Words like ‘Needs Work’, imply that the reader noticed the errors in the story and commenting on them.
This list is not final. I’m open to suggestions of a better wording that improves the distinction in your minds about the meaning of the score you’re casting.
The second change is the more drastic one. I’m replacing the current scores with weighed scores.
The new scores shown on the site will reflect the story’s score relative to the median of all scores on the site. This will have the effect of lowering all scores. I’ve implemented the formulas that calculate the weighed scores and here is a sample of scores and their new values:
Old Score -> New Score
(average) -> (weighed)
10 -> 10
9.85 -> 9.56
9.5 -> 8.55
9 -> 7.10
8.62 -> 6
7 -> 4.93
6 -> 4.28
5 -> 3.62
4 -> 2.96
3 -> 2.31
1 -> 1
One thing to remember, the weighed score is relative to the current median. So a story’s score may change even if it received no new votes. If the median changes, then the story’s score will change.
Hopefully, the wording change will make the votes that readers cast more reasonable, so that automatic 10s change to something more meaningful.
One problem I don’t know how to address is the fact that the more recent the story is on the site, the higher the average score is; this is related to the psychological effect of higher and higher scores. So if you have a reasonable solution, I’m all ears.
I know that scoring is a controversial subject but let’s all try to be as objective and reasonable as possible. I’m trying to make the system work for everybody the best possible way. I appreciate everybody’s contributions.
And before you fire off your reply, one thing I will not do, I won’t ever scrap the voting system. So don’t even suggest it.
I made few changes to the site in the past few days. I thought I’d talk about them a bit.
With the change, the form simply goes away and the text changes to reflect the result.
The transition for the voting form was almost trouble free, but the one for the mail form was quite bumpy. The reason for the bumpiness is mostly my inexperience with the new technology that I’m using in the new forms; I’m learning it as I implement it. Many readers had the mail form fail to send or return any results and it took me a couple of days to sort out why it’s happening and how to go about solving the issue. But it also took longer to sort because very few people reported the problem; only two people reported it.
To help me fix any current or future problems with the site, I would appreciate it if those problems were reported to me. The webmaster link on the site provides an easy way to do it.
So, if the new forms are not working for you, let me know. In the Webmaster’s contact page, select the message type as Bug Report.
Today, Saturday, August 19th, 2006, marks the deployment of a new version of the site’s submission wizard. It’s not a complete re-write but a significant one nonetheless.
If you’re an author, give it a whirl and tell me what you think. Of course, bug reports are appreciated as well as suggestions for possible enhancements.
Well, the new reviewing system is mostly finished.
I revamped the review display page completely and made it accessible to everybody. The old reviewing system has been integrated with the site. No more need for a separate log in page and different ID. Reviewing is easier than ever for those with reviewer accounts.
A new reviewers section has been added to the site and it’s only
accessible for those with reviewer accounts.
The funny part about this change is that it came with no real planning. I know I had discussed a new reviewing system on this blog a long time ago, but it seemed that I could never really plan it and implement it properly within a reasonable amount of time. So I kept putting it off for later.
The way this started was with me thinking that the old reviews listing page is not particularly useful to either authors or reviewers if only premier members could access it. So I decided to give access to everybody and started by revamping the interface. Before this change, the reviews page was just a list by reviewer. It was static and listed all the reviews without any particular sorting, so new reviews could not be easily found by interested readers.
Bit by bit, the changes came and before I knew it, I had finished
reintegrating the old reviewing system, which had separate login and separate everything, into the site’s current structure.
The old plans for a review system accessible automatically by everybody have been changed into a review system with set number of slots for reviewers.
I’ll start with 10 new reviewers. If you’re interested in becoming one of the reviewers contact me and let me know.
All reviews for now will be moderated by me. There are few rules to stick to. Basically, reviews have to be positive. Not necessarily glowing with praise. They have to be at least positive in that they aim to either guide the other members of the site to good stories, or at least help the author with constructive criticism. So it must help. If it doesn’t help anybody, and is aimed to put down the author or drive them away from writing, then the review will be rejected. If it warns the reader to avoid the story without giving a valid reason why, then it will be rejected.
If I reject more than half of your reviews, your reviewer privileges will be revoked and the opening will be available for another that wants it and want to do good with it.
The system supports more than one review per story, so even if a story has already been reviewed and you have a differing opinion or more things that needs to be said about the story, you can still review it.
Be aware that a reviewer account needs a valid email address. And that email address will be visible to readers and authors. It could be different from your main email address for the site. Editors and authors will have the priority in becoming reviewers. And there is a limit to the time allowed without submitting a review. If you go 90 days without a single review, then your slot will be given to somebody else.
If the new system proves to be useful and helpful, I may expand it later and implement the missing part, which is review moderation by other users.
So if you’re interested in becoming a reviewer, use the site’s webmaster contact form to let me know.