Last week I had 2 problems that webmasters dread, being hacked and a catastrophic double server failure, I had both on different sites in the space of 2 days.
The first thing I knew about the hacking was when one of my writing partners Keith sent me an email and screenshot showing a strange error he was getting when uploading images to one of our WordPress blogs. The blog in question was running a pretty old version of WP with known exploits so that’s how the facker got in.
I did a bit of a rush job fix on this, by deleting (after backing up) all the previous WordPress files (n.b. NOT your custom files in the wp-content directory) and then uploading and upgrading to the latest version. It’s important to delete existing files first as they could be compromised by the hacker and if the new version doesn’t overwrite them the hacker could still have an access point. I also went through all files in the wp-content directory (plugins, themes and uploaded images) to ensure that they were the correct version.
Next thing I know I get the following automated email from Google:
Removal from Google’s index
Dear site owner or webmaster of mysite.com,
While we were indexing your webpages, we detected that some of your pages were using techniques that are outside our quality guidelines, which can be found here: http://www.google.com/webmasters/guidelines.html. This appears to be because your site has been modified by a third party. Typically, the offending party gains access to an insecure directory that has open permissions. Many times, they will upload files or modify existing ones, which then show up as spam in our index.
The following is some example hidden text we found at mysite.com:
In order to preserve the quality of our search engine, we have temporarily removed some of your webpages from our search results. Currently pages from mysite.com are scheduled to be removed for at least 30 days.
We would prefer to have your pages in Google’s index. If you wish to be reconsidered, please correct or remove all pages (may not be limited to the examples provided) that are outside our quality guidelines. One potential remedy is to contact your web host technical support for assistance. For more information about security for webmasters, see http://googlewebmastercentral.blogspot.com/2007/09/quick-security-checklist-for-webmasters.html.
When you are ready, please visit https://www.google.com/webmasters/tools/reinclusion?hl=en to learn more and submit your site for reconsideration.
Google Search Quality Team
Bugger!!! When looking at the page I could see none of the links, however looking at the source and there they where (hidden using very simple CSS). I next tried to edit the spammy posts to see if the links were there and sure enough they were, however when I deleted them and clicked post all I got was a blank page, this also happened when I tried to create new posts.
I started researching the problem and came across a few (what looked like) useful tips on WordPress.org, however none of these fixed the issue. So I went back and double checked all plugins, even after disabling them all posting and editing still did not work. The only thing this left to check was the database and sure enough that’s where the problem lay.
In the wp_options table, there was an innocent looking record called wp_links, however the option_value of it contained all of the spammy links that we’d seen earlier and the field infact shouldn’t exist, so this was quickly deleted. We also found our upload path had been changed to:
So this too was corrected. What was causing the post to fail was the hacker had also edited the active_plugins field, so that a rogue plugin would insert links whenever a post was edited. For some reason 2.51 prevented this hack from working but it also prevented posts from being made (so if you’ve just upgraded to WordPress 2.51 and get a blank screen when clicking post this could be why.). The plugin never appeared on the WordpPress options page but I was able to manually remove it from the database. Must admit I’m still not sure how this part of the hack worked but I’m hoping that deleting all reference has fixed it, if you know better please let me know.
The lessons learnt here are pretty obvious, keep up to date with versions and regular backups. I’m now about to send off a re-inclusion request to Google and I’m really thankful for that automated email, even though it did scare the shit out of me.
My next post will focus on different backups strategies that you can implement and a real life example of what can happen if you don’t, bugger!