Lazeez’s Corner

We’ve moved!

Alright, so we’ve moved. The move yesterday took about two hours from when the servers were shut down to the site being accessible again. It was fairly uneventful.

However, the site is not back up to to its old normal self. We’re still missing the hefty hardware required for content search. We’re running on a single database server with large memory. The current speed comes mostly from the database being completely cached in the RAM. So disk access is strictly for flushing MySQL’s memory back to disk.

We’re now looking for either a decent disk array to hook up to the server, or a server with decent hard drive space and high processing capability to replace the current one. So content search (where you can search for stories according to text contained in them) is still missing.

So, what happened?

Again, with a problem that took the site offline for a considerable amount of time. So, what happened?

Well, it turned out that the hosting location that I have the servers in, is in the process of being closed down. They didn’t give me prior warning. So there were people moving servers and racks to a new location (our turn will be on Friday).

Somebody knocked the power cord from the database server’s array and that took it offline. We spent the afternoon and the evening trying to recover and rebuild the array and each time it starts to work then reports a failing disk. After the third time and a report of a different disk failing, we figured that it’s the array itself that’s hosed and not the disks.

At midnight, we removed the array and replaced it with a server with some drives in it and restored the database from the morning’s backup. Due to this, we had to forego the Content Search function until we get a new solution.

So the net result is that we lost nearly 12 hours worth of data and an expensive piece of hardware.

Since we’re moving on Friday, we will deal with the Content search problem and the new hardware after the move.

So this is the first announcement that on Friday, June 19th, the site will be offline for few hours. We will need to move the servers physically to a new location and have the DNS entries updated to point to the new IP addresses. So that may take anywhere from two to six hours. Hopefully as close to the lower end as possible.

After that, we’ll need to get new hardware to replace the old one and when that happens, we’ll need to take the site offline again to make the replacement. But that will be about 15 days or so after the move. I’ll make the appropriate announcement when the time comes.

The lesson learned?

You can never have too much redundancy and backup backup backup…

Oh yeah, before I forget, the site will be offline again on Friday June 19th. Sorry for the inconvenience folks!

Small changes in posting

Due to some new constraints, I have had to make some changes to the scheduling of delayed postings.

From now on, instead of posting delayed stories and chapters at midnight Eastern time, it will be at 9pm Eastern time.

When it rains, it pours!

Well, it does. When it rains it seems to come from all sides.

Last night, a network outage 10 days after the last one.

Tonight, a catastrophic hardware failure in the database servers’ raid at 8:15 pm on the 27th. We’re scrambling to find and configure a replacement. It will definitely take a while and some data has been lost.

Sorry for the inconvenience folks. We’re doing all we can to get things up and running again.

Update 1:55 am: Well, we’re piling it on. On top of the hardware problems we’re having, the replacement server raid that we got turned out to be faulty AND Rogers (the backbone provider) is doing a network maintenance just like last night, without any warning. The tech working at the data center can’t communicate with the outside world to make sure that we get new software to install of the hardware (which is dead at the moment too).

Update 6:30 am: Things are coming along well. We’ve managed to get the database for the storiesonline.net domain going. It’s fairly operational. However, we lost everything that happened from 4:00 am EST on the 26th, so roughly 48 hours of activities have been lost (the latest back up was corrupted, the previous one worked.)

The secure connections over https (for log in and for payments) are not working. We haven’t figured it out yet. The admin pages aren’t fully working yet. So be patient about updates.

Not everything is as it should be, but it’s definitely better than being offline completely.

Update 28th, 2:25 pm: well, the site is almost back to normal, but the submission processing is broken. It needs some work. It will take until the evening to get it done and start processing submissions again.

Your patience is appreciated.

Site back up

Well, the site came back online at 4:45am EST.

It was another fiber this time, but it was a cut instead of 3 poles taken down by a truck like last time. Last time they had to lay nearly 100m of new fiber.

Down again?

Well, it’s 1:50 am EST and I just became aware that the site is not reachable. Again, my whole host’s network is not reachable, not just storiesonline. So as of now, I have no idea what’s going on. I called the tech support people and waiting for news.

SOL is unreachable

As of 2:10 pm, storiesonline is not reachable. The whole hosting facility’s network is not reachable. We don’t know anything yet. Our ISP is investigating.

We’ll keep you posted.

Sorry for any inconvenience.

Update: Not much has changed. Rogers (the backbone provider) is having a major issue with their network in the city. No info on resolution time yet.

Update 4:20 pm: No news whatsoever yet.

Update 5:10 pm: It seems that a fiber optic cable has been cut. So much for trusting ISPs when they say their facilities have multi-links. The last time this happened it took 8 hours for the facilities to go back online. Crossing fingers that this time it would take less. If it’s like before, the site should be back online around 10:00 pm EST. Crap!

Upcoming RSS feed changes

When I started implementing the site’s various RSS feeds, I took the easy road and I used a token in the URL itself for authentication. It was an acceptable solution all that time ago. However, with the proliferation of RSS aggregators and their facilities for finding RSS feeds, I found that I could easily subscribe to any member’s RSS feed that uses those facilities like Google reader.

The upcoming change will have an impact on some of you.

From next Monday on, RSS feeds on the site will require authentication. Most news readers these days support the authentication protocol required. Usually, it will only require that you enter your storiesonline’s user name and password once, and after that, your news reader should take care of the rest.

If your news reader doesn’t support this functionality, then you need to change readers.

Sorry for any inconvenience that this may cause; but there is no way around this, especially with the existence of the library RSS feeds.

Update: I’ve created additional non-authenticated feeds for the New stories and the updates pages. However, for various reasons, not all stories show up in the non-authenticated feeds. So if you choose to use these feeds, you may miss stuff. But, I figure it’s better than no feeds at all. To find the various feeds, including the non-authenticated ones, check the RSS feeds page.

Second update: Firefox does not work properly with the new authenticated feeds. It asks for authentication once and if you create a live feed, the second time around you’ll get ‘Live bookmark failed to load message’ without telling you why. The only way to make firefox work with the authenticated feeds is to follow the hack outlined below in the comments.

SOL is back up

Storiesonline.net went down at 4:00 am EDT for reason still unknown.

I am still investigating the issue. Your patience is appreciated.

UPDATE 11:30am: As it turned out, we got hardware failure in more than one server. We need to replace few servers. This will take a while folks.

UPDATE 2:30pm: We managed to get the site back up on minimal capacity at 1:00 pm EDT.

Not everything is running as it should, so the site could be slower than usual. To bring everything back to normal we’ll need to do a maintenance run. That will happen sometime this week when I get replacement servers. I’ll make an announcement on the site about when to expect this to happen.

Sorry for the inconvenience.

Small Changes

Today I implemented a couple of small changes that may be of interest.

First one is on the readers’ side. For premier members, the Universe/Series listings page now allows you to sort by multiple criteria. You can have the page sort the listings by Universe or series name (the default), by author name, by story count and most importantly, by update date. You can have this sorting be in ascending or descending order.

The second one is for authors, and it’s a minor thing. I added the ‘Flash’ story genre for those stories that would benefit from it (less than 500 words).