Again, with a problem that took the site offline for a considerable amount of time. So, what happened?

Well, it turned out that the hosting location that I have the servers in, is in the process of being closed down. They didn’t give me prior warning. So there were people moving servers and racks to a new location (our turn will be on Friday).

Somebody knocked the power cord from the database server’s array and that took it offline. We spent the afternoon and the evening trying to recover and rebuild the array and each time it starts to work then reports a failing disk. After the third time and a report of a different disk failing, we figured that it’s the array itself that’s hosed and not the disks.

At midnight, we removed the array and replaced it with a server with some drives in it and restored the database from the morning’s backup. Due to this, we had to forego the Content Search function until we get a new solution.

So the net result is that we lost nearly 12 hours worth of data and an expensive piece of hardware.

Since we’re moving on Friday, we will deal with the Content search problem and the new hardware after the move.

So this is the first announcement that on Friday, June 19th, the site will be offline for few hours. We will need to move the servers physically to a new location and have the DNS entries updated to point to the new IP addresses. So that may take anywhere from two to six hours. Hopefully as close to the lower end as possible.

After that, we’ll need to get new hardware to replace the old one and when that happens, we’ll need to take the site offline again to make the replacement. But that will be about 15 days or so after the move. I’ll make the appropriate announcement when the time comes.

The lesson learned?

You can never have too much redundancy and backup backup backup…

Oh yeah, before I forget, the site will be offline again on Friday June 19th. Sorry for the inconvenience folks!