Webmasters nightmare #1 - Wordpress blog hacked
Last week I had 2 problems that webmasters dread, being hacked and a catastrophic double server failure, I had both on different sites in the space of 2 days.
The first thing I knew about the hacking was when one of my writing partners Keith sent me an email and screenshot showing a strange error he was getting when uploading images to one of our WordPress blogs. The blog in question was running a pretty old version of WP with known exploits so that’s how the facker got in.
I did a bit of a rush job fix on this, by deleting (after backing up) all the previous WordPress files (n.b. NOT your custom files in the wp-content directory) and then uploading and upgrading to the latest version. It’s important to delete existing files first as they could be compromised by the hacker and if the new version doesn’t overwrite them the hacker could still have an access point. I also went through all files in the wp-content directory (plugins, themes and uploaded images) to ensure that they were the correct version.
Next thing I know I get the following automated email from Google:
Removal from Google’s index
Dear site owner or webmaster of mysite.com,While we were indexing your webpages, we detected that some of your pages were using techniques that are outside our quality guidelines, which can be found here: http://www.google.com/webmasters/guidelines.html. This appears to be because your site has been modified by a third party. Typically, the offending party gains access to an insecure directory that has open permissions. Many times, they will upload files or modify existing ones, which then show up as spam in our index.
The following is some example hidden text we found at mysite.com:
Buy Procyclidine
Buy Acetylcholine
Buy Yasmin
Buy Tolazoline
Buy Benadryl
Buy Dichlorphenamide
Buy Viagra
Buy Lyrica
Buy Oxytetracycline
Buy Tussionex
Buy Meloxicam
Buy Fenoterol
Buy Streptomycin[...]
In order to preserve the quality of our search engine, we have temporarily removed some of your webpages from our search results. Currently pages from mysite.com are scheduled to be removed for at least 30 days.
We would prefer to have your pages in Google’s index. If you wish to be reconsidered, please correct or remove all pages (may not be limited to the examples provided) that are outside our quality guidelines. One potential remedy is to contact your web host technical support for assistance. For more information about security for webmasters, see http://googlewebmastercentral.blogspot.com/2007/09/quick-security-checklist-for-webmasters.html.
When you are ready, please visit https://www.google.com/webmasters/tools/reinclusion?hl=en to learn more and submit your site for reconsideration.
Sincerely,
Google Search Quality Team
Bugger!!! When looking at the page I could see none of the links, however looking at the source and there they where (hidden using very simple CSS). I next tried to edit the spammy posts to see if the links were there and sure enough they were, however when I deleted them and clicked post all I got was a blank page, this also happened when I tried to create new posts.
I started researching the problem and came across a few (what looked like) useful tips on WordPress.org, however none of these fixed the issue. So I went back and double checked all plugins, even after disabling them all posting and editing still did not work. The only thing this left to check was the database and sure enough that’s where the problem lay.
In the wp_options table, there was an innocent looking record called wp_links, however the option_value of it contained all of the spammy links that we’d seen earlier and the field infact shouldn’t exist, so this was quickly deleted. We also found our upload path had been changed to:
/../../../../../../../../../../../../../../../../../tmp
So this too was corrected. What was causing the post to fail was the hacker had also edited the active_plugins field, so that a rogue plugin would insert links whenever a post was edited. For some reason 2.51 prevented this hack from working but it also prevented posts from being made (so if you’ve just upgraded to Wordpress 2.51 and get a blank screen when clicking post this could be why.). The plugin never appeared on the WordpPress options page but I was able to manually remove it from the database. Must admit I’m still not sure how this part of the hack worked but I’m hoping that deleting all reference has fixed it, if you know better please let me know.
The lessons learnt here are pretty obvious, keep up to date with versions and regular backups. I’m now about to send off a re-inclusion request to Google and I’m really thankful for that automated email, even though it did scare the shit out of me.
My next post will focus on different backups strategies that you can implement and a real life example of what can happen if you don’t, bugger!
Related Posts:
Most likely it was done using the same hacking attempt I faced on my site. Check out the link in my name for full details.
Angsuman, great post thank you. I so wish it had come up when I was searching for answers.
Al, you must be number 100′000 who got hacked this way. But looks like you have erverything under control now again. Good!
Looking forward for your post about (automated?) backup strategies…
Ouch, you have been a busy guy Al, hope you can iron out the wrinkles and get everything back on an even keel.
Cheers Scott, it’s getting there last week was bad but picking up now.
When one of my sites got hacked it didn’t even use a CMS, they got in using http://ftp. Still no idea how the hell that works.
That’s nasty Patrick, all I can think of is an old OS, weak password or brute force, bastards!
fortunately wordpress is not a complicated updgrade compared to other scripts I’ve used in the past, as long as your theme is compatible and any plugins you reply on are compatible of course.
I do have a plugin which doesn’t work with 2.51 though 5 think I’ll have to bite the bullet and either drop it or find an alternative.
I guess this is one of the serious drawbacks of being famous or having a popular site - your chances of being attacked are much higher.
I was recently targeted, and once I fixed the issue Google had put all my pages back in the index within 2 weeks. You probably don’t need the reinclusion request, just resubmit your sitemap. Only some of your pages were taken out, not your whole site.
The funny thing is (funny is probably not the right word) it was on one of my lesser known sites, so I think it was just some random hack.
Sounds like your lucky you’ve got the technical skills to fix this, with wordpress being so easy to setup I can imagine a lot of people just wouldn’t have a clue how to get rid of this.
You’re absolutely right Carl, upgrading and restoring files is pretty easy but you do need some technical knowledge to both edit and know what to edit in the database. wonder if there is a market for a tool to do this, there’s an idea for someone
Backup! Backup! Backup!
I’ve just moved my server to a new hoster and bought the automatic backup plan immediately, with weekly and monthly retention.
Also, I download the full backup to my office every week. (offsite backup)
Costs me a few dozen bucks more, but I can sleep at night…
What was the purpose of the hack? Am I missing something? I don’t know anything about Wordpress internals or hacking, but if all the links inserted were invisible, how could the hacker expect a site visitor to click on them?
The links were not invisible to search engines and therefore helped whichever sites they linked to.
Right, but Google takes a very dim view of invisible links. Surely someone capable of a relatively sophisticated hack would know that the pages would get flagged. Are they hoping other search engines would pick them up? And why disable the blog posting mechanism? Wouldn’t it make more sense to make the hack undetected so that more links make it onto more posts?
Scary stuff. I would never have found those links myself. I knot that technical. I’ll go upgrade all my blogs right now!!
[...] If you need any motivation, check out what happened to this guy. [...]
[...] week was a bad week, following on from one of my blogs being hacked I also experienced one of my forum accounts being accidentally deleted by our provider. It [...]
That’s awful! I hope you it didn’t hurt one of your big earners. After a while you start to think that your pages are safe, but this juts proves that you have to stay vigilant. I’ll definitely be looking my sites over this weekend.
Matt Cutts put up a post on his blog about this very topic, detailing Google’s response to hacked blogs. Interesting stuff.
http://www.mattcutts.com/blog/helping-hacked-sites/
The hack was most likely a scripted hack. Then when they hit pay dirt they go in an manually set-up their spam machine. Bascially MyCanadianPharmacy (MCP). They use Javascript to hide the URL and actually you should double check your box (if linux) to see if the following processes are running: tswapd and irqd
If they are you can’t just kill them. Read more here:
http://pharmalert.zoomshare.com/0.html
For us they got in and did the same thing to the database of a wordpress site. Come to find out we left root on MySQL without a passowrd. Don’t think we are stupid as we locked it by IP to access. To affect our DB they found another app we were running and did SQL injection through the app. Once they write the SQL statement to insert the /../././././tmp they are calling a jpg file. This jpg file is actually Magic Injection Shell. With this they are able to have shell access as apache.
Feel free to contact me if you want more info. I would double check your tmp folder or any of our media folders for a file named mis.jpg.
Thanks for commenting and the great article techustle, I’ve got over a a fair articles like yours now and even switched servers so I’m pretty sure all is clean, fingers crossed.
[...] Luckily, the hack was discovered within 12 hours and the site was restored back to its original status. The full detail of what happened is covered by Al in SelfMadeMinds. [...]
[...] and I have access to, we can see over 100 requests daily for these various security holes. Stories about hacked blogs are becoming more and more common and the ongoing concern is that the newest [...]
The post is on TechCrunch for the moment
http://www.techcrunch.com/2008/06/11/my-blog-was-hacked-is-yours-next-huge-wordpress-security-issues/
Amit Bhawani’s last blog post..eBooks Business Steps & Planning for Serious Business
[...] Hacked! [...]
[...] いくつのWordpressブログが感染しているのかはわかっていない（以前にハッキングされたホストがまたハッキングされる、という二重感染のケースを見たことがある）。しかし指標として、TechCrunchと私自身がアクセスした10以上のWordpressブログでは、これらのさまざまなセキュリティホールのリクエストを毎日100以上発見することができた。ハッキングされたブログについての話 はますますありふれたものになっていて、現在、いつなんどき最新のセキュリティホールが見付けられ悪用されるか、ということが懸念されている。 CrunchBase Information WordPress Information provided by CrunchBase [...]
Hey Al, I share your pain.
My site was nicked on May 22nd. My solution was to remove Wordpress all together and write my own CMS platform. I understand that programming is not everyone’s job (or hobby) but for me, it was a faster solution that auditing a ton of php code.
Cheers!
Mario’s last blog post..VoIP Differs from Data Communications
Had a very similar experience few days ago. new kind of attack, was very lucky to discover it. Read here http://www.prelovac.com/vladimir/warning-website-virus-attack
Vladimir’s last blog post..Users of Amazing Grace theme
Ughhh Why can’t hackers foul up their own back yard?! I had a membership site of mine compromised a couple of weeks back and the hours of headaches it generates is something no webmaster needs!
Does that mean anyone can sabotage a well ranked site? Am I missing something here?
This experience just goes to show you: it’s better to build a site on a domain you own instead of relying on Wordpress. Why not spend a few dollars every month to build your primary web presence. Blogging is the wrong approach to building a business web site. Blogs lack what you need to make money online, which is the goal most webmasters persue.
OMG.
Does this mean all of our sites are open to this kind of attrack?
I am hoping it was fixed in a new update.
Talk about bad luck….
Sites that are running older versions are vulnerable but if you keep uptodate you will minimize the risk.
I am sorry to hear that one of your websites got hacked. Thanks for giving us suggestions on how to stop this from happening. Greg Ellison
Greg Ellison’s last blog post..Is the Iphone for me?