So, what happened?
June 15, 2009 11:04 am StoriesonlineAgain, with a problem that took the site offline for a considerable amount of time. So, what happened?
Well, it turned out that the hosting location that I have the servers in, is in the process of being closed down. They didn’t give me prior warning. So there were people moving servers and racks to a new location (our turn will be on Friday).
Somebody knocked the power cord from the database server’s array and that took it offline. We spent the afternoon and the evening trying to recover and rebuild the array and each time it starts to work then reports a failing disk. After the third time and a report of a different disk failing, we figured that it’s the array itself that’s hosed and not the disks.
At midnight, we removed the array and replaced it with a server with some drives in it and restored the database from the morning’s backup. Due to this, we had to forego the Content Search function until we get a new solution.
So the net result is that we lost nearly 12 hours worth of data and an expensive piece of hardware.
Since we’re moving on Friday, we will deal with the Content search problem and the new hardware after the move.
So this is the first announcement that on Friday, June 19th, the site will be offline for few hours. We will need to move the servers physically to a new location and have the DNS entries updated to point to the new IP addresses. So that may take anywhere from two to six hours. Hopefully as close to the lower end as possible.
After that, we’ll need to get new hardware to replace the old one and when that happens, we’ll need to take the site offline again to make the replacement. But that will be about 15 days or so after the move. I’ll make the appropriate announcement when the time comes.
The lesson learned?
You can never have too much redundancy and backup backup backup…
Oh yeah, before I forget, the site will be offline again on Friday June 19th. Sorry for the inconvenience folks!
June 15th, 2009 at 12:03 pm
Will give a more exact time for the START of the down time? I understand that it must happen but I could plan my day around the down time.
Thanks.
DAle
June 15th, 2009 at 12:05 pm
I don’t know yet. I’ll have firm times soon. But it is anywhere from the afternoon till midnight.
June 15th, 2009 at 12:18 pm
I clicked to read a story and got the announcement that I had exceeded the 16 story limit and that priviledges will be restored in minus 22 hours at some time on June 14th. As it is now June 15th I have no idea what to do. Is this a result of the outage/server migration?
June 15th, 2009 at 12:19 pm
We thank you for the information…sounds like the buttheads your servers were at really didn’t respect your business all that much…Sorry you had to go through all of that. At our house we have our own server…that way we don’t have to worry…it aint fancy smancy high dollar or high end…but for what we do & need…hell it suits us fine…again…thabnks for the word…hang in there !! esteban
June 15th, 2009 at 12:26 pm
Richard, you just need to log out and log back in. Yes it is related to the missing data.
June 15th, 2009 at 12:47 pm
Sound like a royal pain; thanks for keeping us informed.
June 15th, 2009 at 1:21 pm
A small hiccup in a well-done operation. It is mox nix.
June 15th, 2009 at 1:21 pm
The second lesson learnd: No matter how good the equipment is. It wil still fail when a idiot trip over and pull out the plug.
June 15th, 2009 at 2:02 pm
Hey Lazeez!
Thanks for the explanation! Was wondering what had gone wrong. No problem with the outages! You run one of the best sites on the Internet and due to your preparedness, this was a minor glitch. You’re right of course, about backing up. It’s not *if* your equipment is going to fail, it’s only a matter of *when*. And some guy named Murphy usually decides the when…
-mitche
June 15th, 2009 at 2:33 pm
so we lose our storytime for a wee while but the result will be the all nice and clean back again SOL. will have to resort to what are they called again oh yes paperback books love the site though I had upset something when I could not get online phew so just a numpty with big feet can cause all this pain regards
June 15th, 2009 at 3:05 pm
I agree this is the best site I know of. Lots of choices, I hope it stays for a long time. Thanks for the site and explanations.
June 15th, 2009 at 3:19 pm
Like they say in road construction — Temporary inconvenience for permanent improvement.
And it really proves that you can’t engineer for stupidity. Thanks again for the site, and the ever mounting amount of time it takes to keep it running.
June 15th, 2009 at 3:29 pm
Maybe it’s time to go to a secure co-location site provided by your carrier. Might reduce the local loop cost, too.
June 15th, 2009 at 3:33 pm
Well I must be a schmuck cause I never noticed. i guess i need to get over here more often. Anyhoo thanks for the update and I won’t get annoyed on friday.
June 15th, 2009 at 4:32 pm
Sorry for your problems and until you had the problem I had not remembered how much time I spend connected to SOL .
I do sympathize with the problems you are having and cheer for you to get it all finished. Good luck in getting it all completed
Roy
June 15th, 2009 at 4:35 pm
Nope
It’s “Smith’s Law” – not Murphy’s Law
And Smith’s law states simply = that –
Murphy was an optinist!!
June 15th, 2009 at 5:05 pm
I had da shakes man. My Jones was Really
Really gettin’ to me. Ya came back up just in time to save me from full blown withdrawal.
June 15th, 2009 at 5:27 pm
Bummer. I hate it for you and wish I could help, but if I did you’d have more than one plug on the floor.
cmsix
June 15th, 2009 at 5:53 pm
Thanks for the Info/Update. You keep this place running as best as possible. A++
June 15th, 2009 at 5:55 pm
Just before the crash I had just rec’d a dozen 10-10-10 votes on assorted stories and then they were lost. So—I was wondering if …
June 15th, 2009 at 6:01 pm
It’s great to discover that I am not the only one dealing with idiots.
Good luck on the 19th and the next date,
June 15th, 2009 at 8:14 pm
I can live with it. Thank you for letting us know what happened and what will be happening. That little bit of consideration goes a long way.
June 15th, 2009 at 9:21 pm
Ok, Your back up and running. Now you just have to make plans for future issues. Of course you have to remember the old saying – “The best-laid plans of mice and men”. You have a great site and have done an excellent job of keeping it up and running. Don’t let the Mice win.
Winger
June 15th, 2009 at 10:03 pm
Good luck, Lazeez. Do what you gotta do and I’ll light a candle in church tonight. Actually, I’ll blow someone else’s candle out and light it. I like getting God’s attention when I ask Him for something important.
rache
June 16th, 2009 at 12:20 am
Would it be possible for you to have a presence on Twitter or something where users can go to check if they can’t get in. A simple “Yes we are down folks. Hope to be back by —- gmt” would help
June 16th, 2009 at 1:12 am
Engineering to prevent some clod from pulling out a power cord…That’s chancy.
Keel haul the bozo in question!
June 16th, 2009 at 6:00 am
evening lazzez,thankyou for the warning on the server shift on friday.been a fan of S.O.L for nealy 2 yrs now & i recon you,ve been busting a boiler to keep the site going,this ozzie appreciates your effort.if murphy or other cretins try to spoil things for you,kick their BUTTS out the door (LOL LOL).kindest regards from ozzie land,geoff
June 16th, 2009 at 8:25 am
@25
We have a twitter account:
http://twitter.com/storiesonline
However, I haven’t been good at keeping it updated. I’ll try to do better in the future. This blog and the site’s discussion board are hosted on other companies servers so they don’t get affected by server outages.
June 16th, 2009 at 11:09 am
We hope that the Move will go smothly with no added problems. Thanks for hard work you do to provide Storiesonline to the web
Blessed Be!
June 16th, 2009 at 1:44 pm
Lost ya Sunday while reading new story on bluedragonauthor’ pages, still haven;t gotten all of his stories restored including the one I was reading will try again Friday Nite.
June 16th, 2009 at 3:58 pm
Was bookmarking some stories and used the control key to enable me to add more than one story to my Library – after I bookmarked four stories by the same author the screen presentation started to shrink and is now almost to small to read. I use the latest Firefox as my default browser. I am unable to get the screen back to normal size – can you help
DAVE
June 16th, 2009 at 4:39 pm
Good luck with the transfer.
The best stories site will never fall again!
June 16th, 2009 at 7:03 pm
I assume it has something to do with the server crash and restore, but it appears that most of the LazLong stories are now missing. Was reading the Fountain of youth and fountain of youth 2 series. (Had just read the Wagons Ho! universe stories, and they are gone too.
June 16th, 2009 at 7:11 pm
@ Dave Smith,
Since the library system does not support bookmarking more than one story at a time, I’m assuming that you were using the bookmarking system in Firefox. Unfortunately, I don’t have a windows box right now, so I have no idea what you did. Check the ‘View’ menu and select the ‘Reset’ item in the Zoom submenu.
June 16th, 2009 at 7:12 pm
@ Tandaari
Nope, has nothing to do with the crash. The stories are there.
June 16th, 2009 at 9:51 pm
Aw shucks, some clumsy doofus triped over the cord.
And I was just thinking that one of the alphabet soup gang did a raid…
June 16th, 2009 at 10:21 pm
Phew.. thought my fix of stories was gone.. great to hear the probles being sorted
June 16th, 2009 at 11:51 pm
If you need to go off line for a few days, Lazeez, then that’s how it goes. Reminds me of the old days when you used to turn the server off at night so you could sleep!
Best of luck with the move and the repairs.
If you need a hand announcing to all the authors, give me a shout.
Henrie
June 17th, 2009 at 9:45 am
Good luck on the move. Having been there, done that, got the scars, from the move I know that Murphy works overtime and it never goes the way you want it to, as fast as you want it to and as smoothly as you want it to. And good luck with the array!
June 17th, 2009 at 10:20 am
Is something else wrong today or is it that no one is posting anything?
June 17th, 2009 at 3:50 pm
I really enjoy your site and most of the stories contained on it. Hope you get your problems resolved quickly. I do understand that things happen but hope they ate few and far between.
June 17th, 2009 at 8:09 pm
I am already going into premature withdrawal due to lack of my SOL Fix.
Keep up the very good work
June 18th, 2009 at 5:08 pm
I know how the cord pull issue works… I was in our server room removing an old server from a rack. I touched a network patch cable, and we lost 24 workstations simultaneously. Bad patch cable, that I now need to replace. I pulled the cable and put it back in to get the stations up. But that took 24 people screaming “What did you do?” and 15 minutes to diagnose the problem.
June 19th, 2009 at 5:27 am
@dave smith
im going to guess you have a mouse wheel?
what i think you did is held the control button then moved the mouse wheel which will increase or decrease the font, try it and see if you cant undo it.
June 19th, 2009 at 1:07 pm
Lord I feel your pain.
An electrical storm took out my computer in spite of my supposed ’surge protector’.
Well, here’s hoping thing smooth out for everybody.
And Thanks btw for the site and the efforts you guys are always making to keep it going!
I know its not much to hear but it is appreciated.
June 19th, 2009 at 2:43 pm
Thanks for the heads-up, Lazeez. I wish I could afford to help more financially, but I’m strapped for funds to just pay my bills… really don’t have anything left over.
Good luck with the move and if you don’t mind, can you post the new IP here, in case DNS doesn’t propogate quickly enough?
June 19th, 2009 at 3:58 pm
Dreaming Bear,
Having a surge protector for a lighting strike is like having good shocks when driving off the cliff
June 20th, 2009 at 6:43 am
@Maxx: As long as he drops the TTLs on storiesonline.org 24 hours in advance, the domain will propogate in 5 minutes. Considering that the site is going to be down for longer than that, I don’t think the main bottleneck is going to be propagation.
June 20th, 2009 at 8:37 am
This is such a GREAT sight for so many of us that a little down time is nothing we will worry about. We all know you are and will do everything you can to keep everyone happy and up-to-date. — Thanks for the GREAT work.
June 21st, 2009 at 12:06 am
I can imagine the sweat dripping down your brow as drives started failing… we’ve all been in technical hell in some capacity. Technology is great until it stops working (or the electric plug gets pulled.)
Those of us who have been around a while back-up a lot. Good advice to hear again however.
I wish you luck on the move and getting your array replaced. Hopefully the usual techno rule of ‘cheaper, faster, better’ (than the previous RAID array) will hold true for you.
We all appreciate your site, Lazeez.
June 21st, 2009 at 10:22 am
Well what is a few hours when yoy have such a great site that is up most of the time.
Thanks for a great site
Diego
June 25th, 2009 at 8:00 pm
Lazeez, I just wanted to make a brief comment. I have been a devoted player of “World of Warcraft” for about 9 months. In that time, servers are taken offline on a weekly basis. On occasion, crashes have caused specific servers to be down for many hours.
In contrast, I have been a regular user of StoriesOnline for (I figure) a good 5 or 6 years now. In all that time, I think that I can count periods of significant SOL downtime on a single hand; in fact, this recent event was the only one I can remember in all that time.
Thus, I wished to pass along my most sincere thanks for having such a steady, strong and dependable service.
June 27th, 2009 at 11:28 am
I have done moves of all sorts of electronic equipment over the years. The one thing that stands out is that no matter how well planned, the feces WILL hit the rotating air mover. All a being can do is try to minimize the outage. Y’all did GOOD. Thank you for your efforts.
July 3rd, 2009 at 2:52 am
Lazeez,
Very good Job….Iknow problems can be.
Everything is working fine
Mike
August 9th, 2009 at 10:32 pm
The problem with making things follproof is that we always underestimate the ingenuity of fools.
August 16th, 2009 at 8:14 am
Never Noticed the downtime, which is the nice thing about having multiple tabs open to track large stories.
October 28th, 2009 at 4:31 am
You’re doing a hell of a job. Now that my left “great toe” has been removed maybe I can start adding to the burden you have by posting more.
cmsix
February 8th, 2010 at 8:35 am
As of 2010-02-08:13:32 (GMT) The site appears to be unresponsive. Looks like the system accepts the initial connexion but then the webserver never responds.
The storiesonline.org site seems to take a long time to respond, but then comes up with the usual home page.
Sorry for the slightly innapropriate use of commenting, but I’m not at my usual computer and I’m accessing the site via Lynx (which is a whole new ‘experience’ in itself).
Regards, Arty