Webmasters nightmare #1 - WordPress blog hacked

Last week I had 2 problems that webmasters dread, being hacked and a catastrophic double server failure, I had both on different sites in the space of 2 days.

Hacker Error The first thing I knew about the hacking was when one of my writing partners Keith sent me an email and screenshot showing a strange error he was getting when uploading images to one of our WordPress blogs. The blog in question was running a pretty old version of WP with known exploits so that’s how the facker got in.

I did a bit of a rush job fix on this, by deleting (after backing up) all the previous WordPress files (n.b. NOT your custom files in the wp-content directory) and then uploading and upgrading to the latest version. It’s important to delete existing files first as they could be compromised by the hacker and if the new version doesn’t overwrite them the hacker could still have an access point. I also went through all files in the wp-content directory (plugins, themes and uploaded images) to ensure that they were the correct version.

Next thing I know I get the following automated email from Google:

Removal from Google’s index
Dear site owner or webmaster of mysite.com,

While we were indexing your webpages, we detected that some of your pages were using techniques that are outside our quality guidelines, which can be found here: http://www.google.com/webmasters/guidelines.html. This appears to be because your site has been modified by a third party. Typically, the offending party gains access to an insecure directory that has open permissions. Many times, they will upload files or modify existing ones, which then show up as spam in our index.

The following is some example hidden text we found at mysite.com:

Buy Procyclidine
Buy Acetylcholine
Buy Yasmin
Buy Tolazoline
Buy Benadryl
Buy Dichlorphenamide
Buy Viagra
Buy Lyrica
Buy Oxytetracycline
Buy Tussionex
Buy Meloxicam
Buy Fenoterol
Buy Streptomycin

[…]

In order to preserve the quality of our search engine, we have temporarily removed some of your webpages from our search results. Currently pages from mysite.com are scheduled to be removed for at least 30 days.

We would prefer to have your pages in Google’s index. If you wish to be reconsidered, please correct or remove all pages (may not be limited to the examples provided) that are outside our quality guidelines. One potential remedy is to contact your web host technical support for assistance. For more information about security for webmasters, see http://googlewebmastercentral.blogspot.com/2007/09/quick-security-checklist-for-webmasters.html.

When you are ready, please visit https://www.google.com/webmasters/tools/reinclusion?hl=en to learn more and submit your site for reconsideration.

Sincerely,
Google Search Quality Team

Bugger!!! When looking at the page I could see none of the links, however looking at the source and there they where (hidden using very simple CSS). I next tried to edit the spammy posts to see if the links were there and sure enough they were, however when I deleted them and clicked post all I got was a blank page, this also happened when I tried to create new posts.

I started researching the problem and came across a few (what looked like) useful tips on WordPress.org, however none of these fixed the issue. So I went back and double checked all plugins, even after disabling them all posting and editing still did not work. The only thing this left to check was the database and sure enough that’s where the problem lay.

In the wp_options table, there was an innocent looking record called wp_links, however the option_value of it contained all of the spammy links that we’d seen earlier and the field infact shouldn’t exist, so this was quickly deleted. We also found our upload path had been changed to:

/../../../../../../../../../../../../../../../../../tmp

So this too was corrected. What was causing the post to fail was the hacker had also edited the active_plugins field, so that a rogue plugin would insert links whenever a post was edited. For some reason 2.51 prevented this hack from working but it also prevented posts from being made (so if you’ve just upgraded to WordPress 2.51 and get a blank screen when clicking post this could be why.). The plugin never appeared on the WordpPress options page but I was able to manually remove it from the database. Must admit I’m still not sure how this part of the hack worked but I’m hoping that deleting all reference has fixed it, if you know better please let me know.

The lessons learnt here are pretty obvious, keep up to date with versions and regular backups. I’m now about to send off a re-inclusion request to Google and I’m really thankful for that automated email, even though it did scare the shit out of me.

My next post will focus on different backups strategies that you can implement and a real life example of what can happen if you don’t, bugger!

About Al Carlton

Al quit the 9 to 5 rat race in January of 2007, before then he was a software engineer and systems architect of financial system. Nowadays Al spends the days running his various businesses and experimenting with different ideas and opportunities.
Al can be found on twitter at AlCarlton.

Comments

  1. Most likely it was done using the same hacking attempt I faced on my site. Check out the link in my name for full details.

  2. Al, you must be number 100’000 who got hacked this way. But looks like you have erverything under control now again. Good!

    Looking forward for your post about (automated?) backup strategies…

  3. Ouch, you have been a busy guy Al, hope you can iron out the wrinkles and get everything back on an even keel.

  4. When one of my sites got hacked it didn’t even use a CMS, they got in using ftp. Still no idea how the hell that works.

  5. fortunately wordpress is not a complicated updgrade compared to other scripts I’ve used in the past, as long as your theme is compatible and any plugins you reply on are compatible of course.

    • I do have a plugin which doesn’t work with 2.51 though 5 think I’ll have to bite the bullet and either drop it or find an alternative.

  6. I guess this is one of the serious drawbacks of being famous or having a popular site - your chances of being attacked are much higher.

    I was recently targeted, and once I fixed the issue Google had put all my pages back in the index within 2 weeks. You probably don’t need the reinclusion request, just resubmit your sitemap. Only some of your pages were taken out, not your whole site.

    • The funny thing is (funny is probably not the right word) it was on one of my lesser known sites, so I think it was just some random hack.

  7. Sounds like your lucky you’ve got the technical skills to fix this, with wordpress being so easy to setup I can imagine a lot of people just wouldn’t have a clue how to get rid of this.

    • You’re absolutely right Carl, upgrading and restoring files is pretty easy but you do need some technical knowledge to both edit and know what to edit in the database. wonder if there is a market for a tool to do this, there’s an idea for someone 🙂

  8. Backup! Backup! Backup!

    I’ve just moved my server to a new hoster and bought the automatic backup plan immediately, with weekly and monthly retention.
    Also, I download the full backup to my office every week. (offsite backup)
    Costs me a few dozen bucks more, but I can sleep at night…

  9. What was the purpose of the hack? Am I missing something? I don’t know anything about WordPress internals or hacking, but if all the links inserted were invisible, how could the hacker expect a site visitor to click on them?

  10. Right, but Google takes a very dim view of invisible links. Surely someone capable of a relatively sophisticated hack would know that the pages would get flagged. Are they hoping other search engines would pick them up? And why disable the blog posting mechanism? Wouldn’t it make more sense to make the hack undetected so that more links make it onto more posts?

  11. Scary stuff. I would never have found those links myself. I knot that technical. I’ll go upgrade all my blogs right now!!

  12. That’s awful! I hope you it didn’t hurt one of your big earners. After a while you start to think that your pages are safe, but this juts proves that you have to stay vigilant. I’ll definitely be looking my sites over this weekend.

  13. Matt Cutts put up a post on his blog about this very topic, detailing Google’s response to hacked blogs. Interesting stuff.

    http://www.mattcutts.com/blog/helping-hacked-sites/

  14. The hack was most likely a scripted hack. Then when they hit pay dirt they go in an manually set-up their spam machine. Bascially MyCanadianPharmacy (MCP). They use Javascript to hide the URL and actually you should double check your box (if linux) to see if the following processes are running: tswapd and irqd

    If they are you can’t just kill them. Read more here:
    http://pharmalert.zoomshare.com/0.html

    For us they got in and did the same thing to the database of a wordpress site. Come to find out we left root on MySQL without a passowrd. Don’t think we are stupid as we locked it by IP to access. To affect our DB they found another app we were running and did SQL injection through the app. Once they write the SQL statement to insert the /../././././tmp they are calling a jpg file. This jpg file is actually Magic Injection Shell. With this they are able to have shell access as apache.

    Feel free to contact me if you want more info. I would double check your tmp folder or any of our media folders for a file named mis.jpg.

    • Thanks for commenting and the great article techustle, I’ve got over a a fair articles like yours now and even switched servers so I’m pretty sure all is clean, fingers crossed.

  15. Hey Al, I share your pain.

    My site was nicked on May 22nd. My solution was to remove WordPress all together and write my own CMS platform. I understand that programming is not everyone’s job (or hobby) but for me, it was a faster solution that auditing a ton of php code.

    Cheers!

    Mario’s last blog post..VoIP Differs from Data Communications

  16. Had a very similar experience few days ago. new kind of attack, was very lucky to discover it. Read here http://www.prelovac.com/vladimir/warning-website-virus-attack

    Vladimir’s last blog post..Users of Amazing Grace theme

  17. Ughhh Why can’t hackers foul up their own back yard?! I had a membership site of mine compromised a couple of weeks back and the hours of headaches it generates is something no webmaster needs!

  18. Does that mean anyone can sabotage a well ranked site? Am I missing something here?

  19. This experience just goes to show you: it’s better to build a site on a domain you own instead of relying on WordPress. Why not spend a few dollars every month to build your primary web presence. Blogging is the wrong approach to building a business web site. Blogs lack what you need to make money online, which is the goal most webmasters persue.

  20. OMG.

    Does this mean all of our sites are open to this kind of attrack?

    I am hoping it was fixed in a new update.

    Talk about bad luck….

  21. I am sorry to hear that one of your websites got hacked. Thanks for giving us suggestions on how to stop this from happening. Greg Ellison

    Greg Ellison’s last blog post..Is the Iphone for me?

  22. fortunately wordpress is not a complicated updgrade compared to other scripts I’ve used in the past, as long as your theme is compatible and any plugins you reply on are compatible of course.

  23. I download the full backup to my office every week. (offsite backup)
    Costs me a few dozen bucks more, but I can sleep at night…

  24. i was wondering if there are webmasters who manages several thousand websites at a time.:~”

  25. There are a number of wordpress add ons that make backing up your blog a whole lot easier

  26. Unfortunately, keeping up to date doesn’t always work. I had a site that was hacked the same day it was upgraded to WP 3.0. Not fun! Keeping up to date with the latest releases is definitely wise advice - but unfortunately it doesn’t prevent hackers from doing their thing.

  27. I truly despise people who doesn’t do anything but create havoc to people who work hard and make proper and decent living. You work hard to keep your page and then suddenly, you’ll wake up one morning and you were hacked.Pitiful!!!

    • Agree. My first ever blog i made was hacked and whast worse, it was just a blog about me and my family with all of my friends visiting it. He deleted all of my posts and i was panicking for hours before i found out i can get back my posts trough feeds. I never understood how can someone do something like that…

  28. I usually back up my site every other day in the evenings just to be safe. I couldn’t image losing all the data we have in our portfolio… nightmare!

Leave a Reply