Lazeez’s Corner

Phasing out support for externally linked stories

We’ve had support for externally linked stories since the start of the site. However, the feature wasn’t exactly popular and generally led to confusion and frustration with the readers and misuse and abuse by some authors.

I’ve decided to phase out support for externally linked stories. As of today, the ‘New External Story’ button in the site’s submission wizard is gone.

I haven’t decided what to do about the existing, externally linked stories, yet or when. But my first instinct is to remove them after contacting their authors and giving them a chance to post on the site directly.

Expired Security Certificate Situation

Since Monday, November 9th 2009, in the afternoon, all secure pages on the site from the log in page to the subscription system are causing browsers to report an expired security certificate error.

Impact:

Not much really. The certificate still works for encryption, so no data is ever compromised. All a user has to do is to make their browser accept the certificate temporarily (allow an exception) and all will be well.

What happened?

Why such a lapse in security? Well, this is interesting.

The current (expired) certificate was obtained from Entrust two years ago. It had a duration of two years. And as it turned out, Entrust has interesting new policies.

First: They only alerted me last Wednesday about the impending expiration of the certificate instead of their usual 30 days. So I didn’t remember to renew on time. It would have been fine if we simply renewed.

Second: Entrust’s new amazing policy of not accepting any purchases for less than $1000.

A single, standard URL certificate for 4 years is $538 with Entrust (it started at $299 per year in 2003). That’s what I usually get and the site doesn’t need any more than that. So if I were to keep dealing with Entrust, I would have to buy something else; something that I don’t need. I would have to invent a domain to get a certificate for it.

So, I decided to get a certificate from Verisign instead. They are much better priced anyway and they accept my insignificant business dealings of $356 for the same type of certificate for the same period of time.

However, certifying a site requires verification of the business entity behind the site and that takes 4 to 5 business days to finish and produce the certificate.

I started the process on Thursday, and hopefully we get the new certificate by tomorrow.

So for now, just accept the current expired certificate and everything would work as usual.

Sorry for any inconvenience.

Update 2009-11-11 2:00 pm: The issue has been resolved and a new security certificate has been installed and activated.

iPhone / Small Screen Support

Today, the 21st of September 2009 I rolled out preliminary support for portable devices with small screens. This support is restricted to story display and the new stories list and the updated serials list.

I will be rolling out support in other pages gradually.

For now, if you stumble on bugs, let me know and I’ll correct them. Also, if you have other suggestions that could help, let me know.

We’ve moved!

Alright, so we’ve moved. The move yesterday took about two hours from when the servers were shut down to the site being accessible again. It was fairly uneventful.

However, the site is not back up to to its old normal self. We’re still missing the hefty hardware required for content search. We’re running on a single database server with large memory. The current speed comes mostly from the database being completely cached in the RAM. So disk access is strictly for flushing MySQL’s memory back to disk.

We’re now looking for either a decent disk array to hook up to the server, or a server with decent hard drive space and high processing capability to replace the current one. So content search (where you can search for stories according to text contained in them) is still missing.

So, what happened?

Again, with a problem that took the site offline for a considerable amount of time. So, what happened?

Well, it turned out that the hosting location that I have the servers in, is in the process of being closed down. They didn’t give me prior warning. So there were people moving servers and racks to a new location (our turn will be on Friday).

Somebody knocked the power cord from the database server’s array and that took it offline. We spent the afternoon and the evening trying to recover and rebuild the array and each time it starts to work then reports a failing disk. After the third time and a report of a different disk failing, we figured that it’s the array itself that’s hosed and not the disks.

At midnight, we removed the array and replaced it with a server with some drives in it and restored the database from the morning’s backup. Due to this, we had to forego the Content Search function until we get a new solution.

So the net result is that we lost nearly 12 hours worth of data and an expensive piece of hardware.

Since we’re moving on Friday, we will deal with the Content search problem and the new hardware after the move.

So this is the first announcement that on Friday, June 19th, the site will be offline for few hours. We will need to move the servers physically to a new location and have the DNS entries updated to point to the new IP addresses. So that may take anywhere from two to six hours. Hopefully as close to the lower end as possible.

After that, we’ll need to get new hardware to replace the old one and when that happens, we’ll need to take the site offline again to make the replacement. But that will be about 15 days or so after the move. I’ll make the appropriate announcement when the time comes.

The lesson learned?

You can never have too much redundancy and backup backup backup…

Oh yeah, before I forget, the site will be offline again on Friday June 19th. Sorry for the inconvenience folks!

Small changes in posting

Due to some new constraints, I have had to make some changes to the scheduling of delayed postings.

From now on, instead of posting delayed stories and chapters at midnight Eastern time, it will be at 9pm Eastern time.

When it rains, it pours!

Well, it does. When it rains it seems to come from all sides.

Last night, a network outage 10 days after the last one.

Tonight, a catastrophic hardware failure in the database servers’ raid at 8:15 pm on the 27th. We’re scrambling to find and configure a replacement. It will definitely take a while and some data has been lost.

Sorry for the inconvenience folks. We’re doing all we can to get things up and running again.

Update 1:55 am: Well, we’re piling it on. On top of the hardware problems we’re having, the replacement server raid that we got turned out to be faulty AND Rogers (the backbone provider) is doing a network maintenance just like last night, without any warning. The tech working at the data center can’t communicate with the outside world to make sure that we get new software to install of the hardware (which is dead at the moment too).

Update 6:30 am: Things are coming along well. We’ve managed to get the database for the storiesonline.net domain going. It’s fairly operational. However, we lost everything that happened from 4:00 am EST on the 26th, so roughly 48 hours of activities have been lost (the latest back up was corrupted, the previous one worked.)

The secure connections over https (for log in and for payments) are not working. We haven’t figured it out yet. The admin pages aren’t fully working yet. So be patient about updates.

Not everything is as it should be, but it’s definitely better than being offline completely.

Update 28th, 2:25 pm: well, the site is almost back to normal, but the submission processing is broken. It needs some work. It will take until the evening to get it done and start processing submissions again.

Your patience is appreciated.

Site back up

Well, the site came back online at 4:45am EST.

It was another fiber this time, but it was a cut instead of 3 poles taken down by a truck like last time. Last time they had to lay nearly 100m of new fiber.

Down again?

Well, it’s 1:50 am EST and I just became aware that the site is not reachable. Again, my whole host’s network is not reachable, not just storiesonline. So as of now, I have no idea what’s going on. I called the tech support people and waiting for news.

SOL is unreachable

As of 2:10 pm, storiesonline is not reachable. The whole hosting facility’s network is not reachable. We don’t know anything yet. Our ISP is investigating.

We’ll keep you posted.

Sorry for any inconvenience.

Update: Not much has changed. Rogers (the backbone provider) is having a major issue with their network in the city. No info on resolution time yet.

Update 4:20 pm: No news whatsoever yet.

Update 5:10 pm: It seems that a fiber optic cable has been cut. So much for trusting ISPs when they say their facilities have multi-links. The last time this happened it took 8 hours for the facilities to go back online. Crossing fingers that this time it would take less. If it’s like before, the site should be back online around 10:00 pm EST. Crap!