Bringing Down Millions of WordPress Sites | Hacker News Reader

Bringing Down Millions of WordPress Sites

jeremyaboyd.com

215 pointsjermaustin163 comments

Back in 1999 when I was learning to code (PHP obviously), I was stoked to discover the mail() function. Up to that point I'd only been doing HTML and JavaScript coding so my newfound ability to send emails was a pretty big deal. I asked my friend what would happen if I put the mail call inside a while(true) statement. He said it would run until the server hit the 30 second timeout. I asked how many emails my server could send in 30 seconds but he didn't know. I decided to to find out, directing all the emails to my hotmail account. Little did I know, that my ISP & hosting provider had disabled the default 30-second timeout so the script kept running. A few hundred thousand emails later I was having second thoughts about my poorly planned experiment. The next day I got a phone call from my ISP. Turns out Microsoft perceived it as a DDOS attack and blocked all traffic from my ISP. And so for a day basically no one in my city could access their Hotmail accounts or any other Microsoft properties. Oops.

I was on a Cathay Pacific flight a few years ago and was looking at all the games they had in the in flight entertainment system. This was an older plane with really bad games, one of which was a number guessing game. I was bored and so wondered what would happen if I entered a negative number as my guess. The entire in flight entertainment system on the plane was down for a couple of hours until the crew figured out how to restart the system properly.

I think you mean Cathay Pacific?

Edit: Glad to help! I was going to delete after you corrected, but it looks like I can't :/ Interesting note: Google doesn't even give me the "Did you mean" option when I tested "cafe pacific". Must be a pretty common mistake.

I've always wanted to fly Cathay Pacific, as I've heard the service is fantastic. Never had the opportunity yet, though.

I do indeed :)

A lot of those big airlines are a mixed bag. If you get one of their newer planes they are world class, but we were on an aging 747, it creaked and groaned, had the aforementioned in flight entertainment system, and the toilets stunk up all the rows near them. But some of their planes are beautiful and the service is top notch. This was a few years ago but even in economy they gave everyone a little pack with toothbrush, ear plugs, travel socks, and those masks you put on to sleep when it's light.

I've had a similar experience on Lufthansa where one flight was an amazing plane with a bar downstairs, and all the mod cons, and then I had a flight from Frankfurt to LA which was 12 hours without any back of seat in flight entertainment (it just had a few screens hanging from the ceiling). This was only three years ago.

That's why I try and fly Air NZ wherever I can, it's not the best airline in the world but it's consistently good across the whole fleet.

I was worried you where going to say it took the plane offline

There was that story a year or two ago about someone who took control of a plane through the in flight entertainment systems. I don't know if it was scare mongering or not but you'd expect that the entertainment system and flight control computers were on separate networks.... wouldn't you?

I don't just expect them to be on separate networks - I expect them to be fully air-gapped!

Entertainment systems typically can show flight info. They receive it through something like ARINC one-way serial link.

Air-gapped might get in trouble too in case of depressurization. It'll be almost vacuum gapped then.

This reminds me of the (possibly urban legend) story of the F-16 navigation system that turned upside down on crossing the equator.

https://catless.ncl.ac.uk/Risks/3.44.html

There is another one that is not an urban legend, in 2007 a group of six F-22 completely lost their navigation and partially lost communication when crossing the international date line. They had to visually follow their tankers back to Hawaii.

http://www.dailytech.com/Lockheeds+F22+Raptor+Gets+Zapped+by...

Pretty certain this isn't an urban legend. I was stationed at Hahn in 1983, the first base in USAFE to get the F-16, and it was related to the troops by a General Dynamics technical guy. This was found in the simulator during testing, and the pilot had flown south of the equator and the plane inverted. He said "Huh, that's strange." and flew back into the northern hemisphere, and the plane corrected itself.

Would the speed of the inversion have killed a pilot? That wasn't mentioned, but given that humans have different G tolerances depending on the direction ... maybe.

I love testing things like that.

I was a caller for my University's Phonathon. Somebody realized that the software had never been tested, so they asked a number of the frequent callers to test the system. I wondered what would happen if I put a negative number in for a pledge. The system accepted the input, but the systems it was tied in to did some really weird things with it. The pledge was automatically marked as completed, because the amount paid (0) was larger than the amount owed (a large negative number). Recognition for gifts isn't based on paid amount though, it's based on amount "credited" to the donor. This makes sure that people get recognized for matching gifts from their companies, but the companies get the tax documents, etc. The recognition amount was stored as an unsigned integer with 4 decimal places. When a negative pledge was entered, an integer underflow occurred and you suddenly were the largest donor to the university by a huge margin. Funny thing was that anybody could have messed the system up, because you could also enter a negative number on the online pledge page as a donor.

That being said, professional fundraisers are very familiar with the largest donors at an organization. Somebody would have caught the error immediately, and there wasn't any way to screw up the financials/bank statements, but it was a fun little error to uncover.

Funny enough, a few months ago I passed a value to a WordPress function, a function that evidently wasn't checking what was sent to it, and got something wonky back. I open a bug report. The reply was, "Well, you're not supposed to do that." My reply was, "NO. You're not supposed to let me do that." I believe the code went unfixed. Crazy.

More like a DOS attack given the one server nature of it.

OP didn't say it was a DDOS, just that MS treated it as such. Given the ISP-level block, that makes sense.

I did one of those in my first couple months of my first development job. I put a mailto in a recursive function for debugging. I thought that the function wasn't working, and since it was 5:30, decided that was enough for the day, shut everything down and went home. On Monday I came back to over a thousand debugging emails in my inbox, and everyone in the office's email was down. Well done me.

Had something similar happen with log4j and sending emails on errors. One day the database was down, and one of the categories was such that it would log database issues. You know what happened next.

What a time, when you got a stern phone call and a scolding for mistakes instead of a full swat steam and some ventilation.

  I asked my friend what would happen if I put the mail call
  inside a while(true) statement.

Amusing. I did something similar in 1995. The difference being that I used mail(1). IIRC, it was something like:

  while : ; do
    mail ...
    sleep 1s
  done

(I did this to a friend who refused to stop forwarding trash messages. I did this so that his inbox could run out of quota.)

We did this one better in 1993/94 at NAU.

Taking our lessons of shell scripting while loops, redirection from a file, and the write command, we'd flood our friends terminal screens with endless scrolls of ascii penises.

What I found horrible was kind of an opposite experience with a bug. There was a developer who built a really powerful project on an aging framework. You could do amazing things with this software; the results were always impressive, but the underlying framework and its tools made the experience a bit less than "slick" on modern computer systems.

So the very nice community that had risen around this software kept shrinking due to competing projects drawing people away. Eventually, things got really stagnant, but the developer just didn't seem to notice or care and kept shipping code, slowly as usual. Finally he pushed some big changes to the software, made an announcement, and members of the community _really_ woke up. "Finally, the features I've been waiting for!" "Wow, thanks for squashing all these bugs!" These people had been subscribed to the mailing lists for over a decade and thought it polite to reply and express appreciation and admiration at this recent--and huge--development.

Six months later, the developer posted to the general list. His message was brief and to the point. There had been a showstopper bug, he said, and today he was providing an updated release with the fix.

Looking closer at the announcement, I realized that the bug meant the latest version of the software _could not perform its intended function_. At all. Completely broken.

So: No one--not a single person (self included)--except the main developer had even given his big release a try. And _he_ hadn't even tried it for ~6 months!

This made me feel really awful for the developer. Fortunately, today people still use the software and have since shared their results & appreciation. But I will always wonder if the developer felt that experience as hard as I did, or even worse.

Is it a Tcl project by any chance?

Well, if it was on Linux, it could have been that people were excited for the release, but it wasn't available on their distro's repo yet. I've definitely been really excited for a feature and then discovering it doesn't work once it's available in my repo 6 months later.

As much as I like the repo system (and C), I was pretty impressed by an experience I had a little while ago. I found a bug in Atom after a new release. I was able to track down the cause in about an hour, and the person who made the commit that caused the bug fixed it an hour later. The bug fix release was available within 24 hours or so. I don't know JS or Coffeescript, but it was verbose enough that I was able to track down the bug and get it fixed without having to wait for the bug to be accepted upstream, or download and compile the package. Compiling a package isn't hard, but I only do it for the software I really want the most recent version of.

Timthumb is how Wordfence came about because my own site was hacked via the vulnerability when it was an 0day and that's how I got involved in WP security.

http://markmaunder.com/2011/08/02/technical-details-and-scri...

Props to Ben. When I worked with him to fix timthumb I don't recall him mentioning that it was someone else who wrote the bug.

There have been plenty of vulnerabilities just as bad or worse since - and it keeps our site cleaning team busy. Besides Timthumb, the other top WP vulnerabilities/exploits are Revolution slider, Mailpoet and Gravity Forms.

You rarely see a site hacked via an old timthumb these days but it still shows up. Most hacked sites are from the other three newer remote code exec vulns I mentioned.

One big problem with Timthumb was that several themes came pre-bundled with it and to make matters worse, they often changed the name of the file, so that it wasn't enough to just scan for the filename but also for variants and file contents.

At the time it was the exploit that kept on giving, since you would suddenly find another timthumb related file on the server that was being targeted, followed by another, ... In the end, a lot of the hosts simply blocked requests to timthumb related files, in order to try and stop it on the server level and not the individual account level.

As you mentioned, these days the sites that I'm being contacted about that have been defaced or were hosting abusive content, are usually related to severely outdated plugins. I haven't seen a timthumb issue in quite some time, but had to restore a site that was affected by the original(!) Revolution slider exploit recently, due to having a several years old version of it installed.

Hey, thanks for Wordfence man! That was right near the top of the list of "must instal plugins" when I used to do WordPress...

3 plugins that are on everyone's "must install before you write a single post" lists.

I never spoke to Ben outside of the comments thread for timthumb to introduce my modifications, but in 2009-2010 (then I lost my site to a bad host who didn't do backups), I know I hosted my cross-domain timthumb and it received tens of thousands of downloads before the exploit was found.

If I recall correctly, the 0day patch was just a list of approved external domains that you had to open the code to add, but it was 5 or 6 years ago, so it is hard for me to remember everything perfectly.

My worst was probably when I rebuilt an old, broken recurring billing system. Unfortunately, in the real world, the retry on failure limit didn't quite work as intended. If a charge failed or was declined, it was supposed to try again a couple of times over the course of several days and send an email to ask the subscriber to enter new payment info. It worked as expected in testing.

Unknown to any of us, the code that triggered it on the production server was being triggered every few minutes by an outside system instead of once a day by the server (as it was supposed to be). So after deployment, thousands of failed transactions were being retried over and over every few minutes. People woke up to scores of emails about the transactions failing and called their banks to find out what was going on. Meanwhile the company that ran the system got some rather unfriendly calls from the banks as it must have looked like they were trying to do something nefarious.

The good news was that no one was overcharged, it was only failed transactions that were repeating (and repeatedly failing). Tripped a lot of alarms though. I added safeguards against the code being triggered by external systems and also explicit chronological limits that didn't rely on the server configuration. It was a learning experience.

A very similar thing happened to me on my first payment processing system. Thankfully it maxed out at 3 tries then sent a final email that the account was suspended until the payment method was updated. The paypal portion worked as expected... they have better devs than I ever will be, but the Authorize.Net portion would fail out within 3 hours.

The bug was simple to fix, the production database's schema failed to update in the deployment. Once we figured that out it was 10 minutes to rerun the deploy script.

This might be considered along the same lines of "unintended use of code".

When I was around 13 I wrote a bash script that would automatically backup your browser data to an FTP server of your choosing. This included your the passwords database, and I made it work with just about every browser possible.

I released the script on some IT forum (I forget which) and it was a big hit for some reason, but then someone re-released it on hackforums and told people to hardcode the server credentials instead of using a settings file. Tricking someone to download a tiff, exe, whatever would upload their password databases from all their browsers in a matter of seconds.

It got quite a bit of attention (many thousands of views) and I've always felt bad about the amount of credentials that were possibly stolen.

It was 1995, and everybody was excited about this new technology called BGP. I was a young sysadmin on his first job at a NOC.

...

Ten minutes later, most of Fairfax County is stuck in a routing loop, and we're getting angry phone calls from NOCs in California, the UK, and I think even Hyderabad. While I didn't actually take down the Internet, it gave me a lot of appreciation for how vulnerable its glue is to human error.

I get contract work from time to time about making a wordpress based site for customers. And they all have custom requirements on top. Granted I haven't done it for years - it's probably better now - but to me, it's just a shinier Joomla.

I couldn't help but facepalm at this article. I understand deadlines and all, but get a friend developer to overlook your code before releasing in the wild, because you know the caveat of "hey I haven't tested it properly" isn't going to make anyone itch.

I've found that the Advanced Custom Fields solves most of the custom needs I get from clients.

I'm not a fan of Wordpress' innards. I usually avoid themes or plugins (other than ACF and a few 'crucial' ones). I'm not a fan of the theming system to the point where I often override or circumvrent the out-of-the-box queries and templating rules. I use my own get_partial() function that properly isolates my template variables and have even used React for the view layer once.

But I still use Wordpress quite often because 1) so many of my clients need exactly the features that wordpress offers, 2) they're familiar with WP and often insist on using it (and it saves me support), 3) I've found that the CMS-part of the whole thing is by far the most work, and building something as slick as WP' admin interface is just too much reinventing the wheel, and 4) there's still an advantage to being able to just run the site on 'typical' hosting (speed, pricing).

Furthermore, the inconvenience to me is a huge advantage to the client, because if they need small changes or additions they're now free to hire any (shitty) 'webmaster' to make css changes or install crappy plugins that magically offer them the features they need. These webmasters are a lot cheaper than I am.

All that said, I generally prefer the kinds of projects where I don't feel Wordpress is the right solution. But for a number of reasons the WP projects are still the most common and relevant ones right now.

You'll probably really enjoy Timber for themeing https://wordpress.org/plugins/timber-library/

Integrates nicely with ACF as well.

That's interesting! Thanks for the link. Any other suggestions?

I'm honestly considering using React when I have the luxury of hosting on my own server. I find it difficult to go back to other templating systems after the nice 'componentized' approach of React. Hard to justify though.

I do a lot of WP work.

Personally, I like things like Timber, but IMO, you really want to avoid getting very far out from the most basic stuff if you ever want to hand the project off to rando WP folks... I stay away from anything even as complex as sage/roots for those reasons.

Though I don't even like ACF, as I have a bunch of snippets for building post meta interfaces and ACF is large enough that I haven't read all its code. I'd rather just write 200 lines into an MU plugin.

> I usually avoid themes or plugins (other than ACF and a few 'crucial' ones). I'm not a fan of the theming system to the point where I often override or circumvrent the out-of-the-box queries and templating rules.

I'm not alone! :)

I used Wordpress 10 years ago. Then I used it again last year. I was shocked that it is still as unreliable as ever. I consider myself lucky if plugins only stop working after an update. Sometimes they break the whole website.

I made the mistake to build my theme on top of the popular Origin theme. That one breaks or changes such it breaks my theme all the time. Over time I adapted my code similar to your guidelines to make it more resilient against other changes.

While it might be in 'paranoid' terroritory, I even avoid using the ACF theme functions and just use the vanilla get_meta() one (or whatever it's exactly). Just in case ACF breaks.

For me, the biggest Wordpress fail, is a file called xmlrpc.php. This file is just a menace and should not exist in the default installation.

My first two things I do when I install a new wordpress server is add this to `.htaccess` :

    Redirect 301 /xmlrpc.php http://127.0.0.1

Otherwise you will have a very easy DDoS vector that can bring your website down, immediately after some bot finds out this file exists on your host.

WordPress 4.7, due in a couple of weeks, includes a brand new REST API which means we can finally start removing some uses of XML-RPC and maybe even someday disable it.

One of the major issues with XML-RPC functionality in WP is that it's enabled by default and opens up a wide target for brute force attacks[1]. I'm assuming this will not change with their REST API. Let's face it, how many WP sites out there really need POST/PUT/DELETE endpoints to be enabled for their family blog?

[1] https://blog.sucuri.net/2015/10/brute-force-amplification-at...

On a blog network I ran for a few years, I spent a lot of time hardening my xmlrpc. I started by adding a required parameter to every call, and if that parameter didn't exist, it would redirect you to a static HTML page. When that was eventually DDoS-ed (all the sites are on the same giant server), I added the location /xmlrpc.php to the nginx server config that blocked it for anyone not coming through my proxy server.

I used to run random feed attacks on the aol/yahoo account fishers... basically feeding and submitting their forms with random names/emails, etc... so that they'd get so much bad data the good would be harder to sift through.

Then, iirc around 2001, there was some change in the law that made that action potentially a felony (regardless that my target was also probably illegal), I stopped doing it...

Overall more of a grey hat thing, but it was cool to play with... would set a new set targets almost every day, as I received fishing emails.

I previously worked for a state government, and owned a project that served districting information. It was used by constituents to find their elected officials and to display different types of districts on maps.

I pushed out a bug that took down the mapping functionality. Normally this would not have been that interesting, but at the time the state was in the middle of a heated redistricting debate. Some newspapers around the state picked up on the outage and attributed it to shady dealings from the majority party at the time.

Slightly different, but it's not often that I have the opportunity to check in code that will lead to a conspiracy theory.

Far smaller scale but this one still makes me giggle:

Worked at large enterprise, at small project team working on the successor for a dated customer support system running in production. Apart from that vision, we had little idea about the domain and about what we were doing. Another team, located in France, was running the existing system, and they were required to cooperate with us. In reality think that they were quite unimpressed with our efforts.

Anyway, we wanted to import their data into our system -- partly for indexing, partly because we saw this as the future -- and we managed to obtain credentials to use their SOAP API. There was a lot of data, and it was paginated by the API, for obvious reasons. Hence, we wrote a script which, when our system was spun up, would import the data from the legacy system by querying all pages of it in rapid succession.

During development we imported from a test system of theirs which contained only a small amount of data but for alpha testing we wired it to their production system. After about 15 minutes we got a phone call from France demanding to know why on earth we were forcing the company customer support system to its knees in a DOS attack. By that time I think there was no doubt in their minds that they were dealing with clowns.

This is the scary part about dependency-heavy projects. Sure, you trust the developers of Rails, but what about that gem 5 levels down your dependency tree? Maybe you've seen the name as you glance through your lockfile, but you don't know what it does, when it was last updated, who (if anyone) maintains it, how competent they are, or what processes they have in place to keep bugs out.

https://github.com/rubysec/bundler-audit

Yeah, but that problem exists with all levels of software development. The best I've seen so far is signed packages in Linux. But even then you must trust someone.

Anyone here ever checked in a bug that infected large numbers of users?

In fact I have. I hit Enter before completing the WHERE clause, and because I'm OCD I close parentheses and quotes before filling them in, and put the ; at the end of SQL statements before filling in the clauses.

Well, I _did_ a long time ago. Now I treat the semicolon key like I treat the Enter key. Lesson learned.

I am also much more cautious now. When I write my UPDATEs I leave out the SET until after writing WHERE clause. And for DELETEs I actually write it as a SELECT first, then once all the rows I expect are accounted for, switch SELECT to DELETE.

Since I use MSSQL, the semicolon isn't required, only a new line, so I actually start every .sql file I write with 3 new lines and an open multi-line comment this way if I accidentally hit Ctrl-E without selecting the EXACT statement I want, it throws an error.

All of these things to combat my stupidity.

I once checked in a database migration that was run automatically as part of the continuous deployment process we have for a client.

It ran through testing and UAT without issue. When the deployment ran for production everything died.

The database change was the addition of a new column to a database table which was set as NOT NULL with a default value. The production table had over 25 million rows. I learnt a valuable lesson the hard way. Never add columns large table with NOT NULL and a default value with MsSQL server. Leave it nullable.

On a much smaller scale, I published a small change to a WordPress plugin I develop called Thank Me Later. Unfortunately, I had used a closure which was only supported by PHP 5.3 and above. At the time, there were still a large number of users using WordPress with PHP 4.something. Being such a small change that couldn't possibly cause any harm, I got on the train and was away from the computer. An hour or so later, I opened my email inbox full of messages like "Help! My website is down", etc. I estimated I knocked at least 100 websites offline.

Reminds me last summer when I was working as a junior software developer in this rather small or medium sized IT company. We used our custom framework written in PHP, and the funny part was that the framework didn't have a single test. Everything had always done by testing manually and hoping for the best. Had I known this beforehand, I would have applied somewhere else.

Anyway, our customer was (and still is) one of the biggest wholesale in some part of the Europe. We had developed online store for them like ten years ago, and the project had become more or less a legacy project where everything was outdated and nothing ever worked. If the customer asked some changes or updates, they were always implemented without proper refactorization nor testing. So you can imagine, if everything is done like this for the past decade with this custom framework, the code base just becomes too complex to maintain, not to mention developing it any further.

So, I had worked in the company for about 3 months when I am given this project. The former developer had left huge changes unfinished without too much commenting nor even briefing me about them. Took me like 3 months to get myself familiarized with the project, finish up the changes and carry out the rest of the requests. The summer ended and everything seemed to work fine in the local as well as in the staging environment. It was Thursday morning I pushed the changes live and everything crashed. Quick rollback, some research, couple fixes and a new try. Same thing. New rollback, some research, some fixes and push. Still broken.

Took me about 10 hours with the help of 2 developers to find out what was wrong. The framework was fundamentally broken in one part, it somehow didn't include files correctly. Still to this date, we don't know why it worked both in local and staging environments but not in production. When I got this fixed, new problems arose: customers had lost products from their carts, logins didn't work, the site was slow... After banging two weeks my head to the wall, everything finally worked. I was so relieved.

These problems affected thousands of users throughout Europe, and I can imagine how pissed some customers might have been. I was so ashamed on behalf of our company, our "framework" and how things were done that I quickly left my resignation.

If I learnt something, discuss the tools and how things are done in the company before applying for the job.

Does anyone have a good guide/article on hardening wordpress from an hosting perspective? What is listed at wordpress.org is too thin for me. What tools are available that can be run on the server instead of in wordpress (WordFence alternatives that I can run on the CLI via cron etc)?

For example I have fail2ban rules to catch break-in attempts etc.

the best you can do is to use a static site generator plugin and serve the generated html. This is the only secure way to use wordpress. The wordpress software should be installed in a not public network from where you can edit (e.g. home computer, vpn).

From a hosting perspective you want to isolate Wordpress as much as possible - i.e. putting it in a namespace with php-fpm (easiert way is using firejail) and running PHP as a seperate user. You also want to only give it the minimal required access rights to the database.

You also need to keep up with security updates and run a supported PHP version - there are still crazy amounts of PHP < 5.6 installs out there :(

From Wordpress.org own stats: https://wordpress.org/about/stats/ http://php.net/supported-versions.php

That's 70% of Wordpress installs that run on EOL and unsupported software!!!

Besides that update your kernel as soon as possible (reboot or hotpatch e.g. ksplice) and keep userland up to date. There are tons of local root exploits for Linux that help an attacker to attain some serious persistence.

Some other ideas I've read is keeping most of Wordpress readonly for the PHP user and use wp-cli (http://wp-cli.org/) or similiar tools that update from the shell. You can do this some group membership trickery.

It's also possible to almost completely forbid access to web clients to wp-content/plugins and wp-includes if your plugins don't do something stupid. But that's error prone and something you should test. For a simple site with a theme you control and few or none plugin it should work fine through.

As a general rule be very conservative regarding plug-ins and themes and spend some time to look at what comes with them.

Spin it: how I created more IT jobs than the Obama administration.

I like that idea, at our Job we sometimes Joke that "Taskrunner" should be a Job description for some people.

Disclaimer: Title is misleading. This is not a 0-day nor a new vulnerability on WordPress.

The author is just talking about some old stories that happened years ago.

Everything is fine. There is no emergency and no need to go update your wordpress setups ASAP. :D

Apparently this years theme was for everyone to keep telling me database IDs are unique that aren't unique...

Reply on news.ycombinator.com