CTM Outage Recap
Posted on November 19, 2007
It’s time for a review. One or two of you may have realized that this site was gone for about 4 days last week, I know I realized it.
For some background, I think CTM has been down for an extended length of time only once over the past 3 years and that was totally my fault. It was sometime around the beginning of 2006 and I attempted to make a change and totally wiped everything out. I didn’t lose any posts, but there are scars from that event. One of my most commented posts occurred in mid-December, 2005 and those comments are gone as well as many others. But time heals all wounds, or at least adds distraction to forget the pain.
Also, the first 6 months worth of posts are gone except for the very first one because I failed to take the effort to retrieve them from blogger. I manually copied the first one at some point. I should have been more diligent in moving those posts over. Again, time heals those losses, there is far more value in the last two and a half years than what was in the first six months. It’s probably good that those are first posts are gone, honestly.
So, those are the two major hits that this site has taken in it’s life, until last week. Sometime during the day last Wednesday, CTM went down. That can happen for any number of reasons and generally the problem is quickly resolved, either on it’s own or with a little troubleshooting. We resell hosting, so there are more than 20 sites on our Virtual Private Server (VPS), so the first thing I checked was some other sites. I honestly hoped they were down as that would indicate a server issue that could be easily resolved. Unfortunately, the ones I checked were up and running.
From there, between Mary and I, we spent many hours Wednesday night, Thursday and Friday troubleshooting and ultimately doing everything we could imagine to get things together. On Friday, Mary devoted all morning to get it resolved and finally concluded she couldn’t fix it. At that point, I knew it was bad, she can generally fix anything website related.
The original database was still on the server with all of my posts and comments in it, but anytime I tried to connect to it, either the original blog or the newly installed blog, it threw errors.
So, by Friday afternoon, when I had a clean version of WordPress 2.3.1 running with an empty database and a MySQL database with 3 years of posts in it that I couldn’t attach to, I concluded that I would restore the database one table at a time.
Saturday morning, I got up early to take on this blog one last time and see who emerged the winner.
First, I restored the posts table from the original database. It worked! No errors! Next, I restored the comments table. It worked! No errors! By then, it was time to wake my daughter up so we could leave for our Saturday morning date.
Looking back, we did get a database corruption notification at about the time that this site went down but I don’t honestly know the cause of that. I almost suspect that the site was hacked but I can’t be sure. The site was being redirected to run under a folder named with a specific string of numbers, yet I couldn’t access the folder or even see it on my server.
Three times between Wednesday night and Friday night, I had a blog up, a fresh new blog with at most, one post. These variations were not up for more than a few hours at a time but the first two times, I received several encouraging comments from people checking in, seeing what’s up, letting me know they are praying for us as we work to get the site back up. The third time it was up, I got no comments, which surprised me based on the first two iterations. But when I checked the RSS feed, it had an error, so no one was notified. I realized that on Saturday morning, so I just blew WordPress away and reinstalled it, then brought everything back.
I have a new appreciation for what I’m doing here and I am thankful for having things mostly restored. I’ll be working to restore categories and tags soon, but honestly, I am behind on some other work I need to get done.
Thanks again for your support and encouragement!
Welcome back, CTM!
Popularity: 12% [?]
Other posts by Jim Walton
»
Comments
4 Responses to “CTM Outage Recap”
-
Subscribe via RSS
RSS Subscribers
Comment Subscribers
Email Subscribers
I knew you were down! Just to encourage you… Kept dropping by the see what the deal was. Glad to see you back and that things are up and running.
Wow, what an ordeal! And a good reminder to grab a WordPress database backup of my own blog. And the other five or six I’m in the process of creating/managing or running!
If you aren’t updating your WordPress install using Subversion, you might want to look at a script I found called EasyWPUpdate. It’s a shell script that will download the newest WordPress version, make a full directory backup and database dump for you, do the upgrade, and tell you when it’s all done. It will even update multiple blogs on the same server at once if you want it to! I’ve been using it for a while with good success. I believe there is a Plugin that will do the same thing now, but I don’t know if it includes the backup that is so nice. But there’s a Plugin just for database backup, too…that’s the one I need to remember to run!
Glad you’re back!
Jim – you may have luck recovering those missing comments you referenced with Archive.org’s Wayback Machine
Actually hadn’t noticed the outage (mostly because I subscribe via RSS), but noticed that your Links page was wiped out. I found a copy of it as of Jun 2007 at http://www.archive.org if you wanted to restore any of the content. I had been meaning to peruse them when I had some down time and just now noticed that the current page appears to be empty.
Glad that you’re back up now.
-Pete