Hacker News Reader

Comment by vinceguidry | Hacker News Reader

I don't know how you could expect QA to be able to have a rigorous process to catch security problems of this type. It's one thing to audit the strength of crypto protocols, quite another to rigorously test every conceivable attack surface for privilege escalation. That space is vast.

View on HN

Actually entering blank passwords and automated password entry should be Test Cases #0 and #1 for any thing that has a login. OS and other critical infrastructure vendors should go beyond that and explore the vast space to make sure nothing like this ever happens.

They may have even had a test case that entered "root" and no password and hit OK once, but that wouldn't have caught it. What if you'd had to hit "OK" forty-one times to trigger the bug? It's deeply unsatisfying to just say "the problem space is so vast that it's hard to even know what to try beyond the basics that every QA person knows" but I don't personally have a better answer than that. Maybe those that study the space do, though?

Fuzzing inputs

Where are all the unit/integration tests for the APIs that this damned button is calling? Hindsight being 20/20, but I cannot imagine not asserting that a newly created/re-enabled root user has a non-empty password.

I don't think empty password on root is the issue here; enabling root when the user did not intend to and did not have permission to is. Isn't empty password for root a default state for systems where direct root login is not enabled? And isn't enabling root with no password the intended behavior when booting into Single User Mode?

Exactly, testing the UI may be hard, but whatever API that UI is calling is amazingly horribly broken.

And fascinatingly wasn’t broken until very recently...

Following and furthering your logic, what the hell could they have been doing in the codebase to revert a control mechanism that was effective up to and including 10.12.6, but unsafe as of 10.13.0 onwards???

A graceful upgrade mechanism towards a new password hashing algorithm.

Change is so, so risky.

There's a lot that can go wrong, which is why writing testable code and then /actually/ testing it, matters so much.

The code was not supposed to enable the root user in the first place, so I'd be surprised if someone added an assertion that anything about the root user's account was set. It was only supposed to check the root user's password.

I suppose you could write an assertion that the code didn't enable the root user, but I'm pretty sure that no password-validation routine anywhere in the history of the world has ever had a test case to make sure it didn't modify the account while validating it.

These are unknown unknowns. If you knew enough to write the right test, you wouldn't have written the bug in the first place.

But if the 2nd attempt does something different, there must be some sort of logic happening, and whoever built this magic (maybe it's an if-block like: (if loginAttempts++ > 0) ... ) should've tested it.

We cam see that the function f(a, b) { return a+b; } should return the same value for the same a and b, so testing that once is enough. But if there's something in there that reads a global variable, maybe check who changes that global variable, and what happens with different values of that global variable?

The "logic" in this case is the first attempt failed because the root user was disabled, but as a side-effect it enabled the root user. The second attempt worked because the root user was now enabled. There's no explicit logic in here that says the second attempt should have behaved differently.

Oh it's never that easy. It happens 20 layers deep, with configuration set a certain way... only every Friday the 13th.

I.e. MacOS is based on UNIX/BSD, root is all powerfull. So root is disabled form login with <star> password, but it's still there. Then there is the Apple autentication framework on top of that, here root is disabled. But if you try to log in with root, it gets enabled but can't login because password is still <star> an disabled. Then somebody decides new password hashes are safer and we should update old hashes whenever possible. So now when a disabled root account gets enabled, the update routine mishandles <star> password and updates the root password hash with <empty string>.

And I'm sure they've had privilege escalation vulns before and a workflow already in place for catching a lot of them. I mean, it's not like Apple's totally asleep at the wheel here. OSX may not be a front-burner project anymore, but they're still supporting it better than Microsoft manages to support Windows.

But that's the nasty thing about InfoSec. It's a never-ending process. There's always realms you haven't considered yet. I'd bet that they had a process in place that they thought could catch these, but it managed to fail somehow. You're always in a hindsight-is-20-20 situation.

> but they're still supporting it better than Microsoft manages to support Windows.

I don't know what metric if any you're using to make that statement but any person working Enterprise/Corporate IT will tell you what MS manages to pull year over year with Windows, Office and bunch of other stuff is fairly monumental given the demands of their operating space. Only way Apple can survive there with their current Engineering setup/culture is if they only did simple stuff and told the customers no interop and you only get to do what we think is best for you!

I bet Microsoft really does do a great job supporting corporate users. Apple does a miles-better job supporting consumers.

(I am not a developer, nor a sysadmin, but a casual programmer and intensive power user, and I get the instant bleah factor when I come across consumers or enterprises using it.)

I’m a longtime NeXT-then-MacOSX user with an experience in commercial UNICES ante-2000 and a predilection for Linux and DragonFlyBSD for my server needs. That said, I am gaining a begrudging respect for Microsoft. They make you pay for your nose, and their stuff looks and feels clunky, but the solidity and backwards-compatibility is utterly amazing.

No way. Microsoft has teams of people who own features. Their errors tend to be regressions related to some other workaround they need to support.

Apple leaves stuff to rot, then marketing cooks up some new requirement and a few programmers rush to build it.

Disk utility is the canonical and obvious example of this. There are many others.

> Apple leaves stuff to rot, then marketing cooks up some new requirement and a few programmers rush to build it.

Is this actually true? I have a hard time believing that. Are Apple developers really jumping around product to product? I'd be very interested to get some insight into their internal processes.

I worked on mobile games few years back. Our tester always did everything that was sane and most things that were not sane. For every version. He tried to break the game. He was randomly and wildly tapping the screen. He was repeatedly going through menus. The most things he did us developers thought: who would do that? We caught lots of bugs this way.

It was a company that you never heard of. It's a normal thing for me to require such things from QA.

I also worked in big corporation that you heard about, developing embedded software. Same thing happened.

Testing is doing things that most users do. But most of all testing is monkey bashing trying to break the thing. That's the mindset that QA must have.

EDIT: typos, thanks

I've frequently seen people use "break" when they mean "brake", but I don't think I've ever seen someone get it the wrong the other way around.

Do you realize for all the unexpected bugs your manual tester caught, an equal number were simply never found?

No one said that you can't do automatic tests. Manual tests do not exclude automatic tests. But I dare to say that manual tests are essential, because that's how the software will really be used. Especially for release candidates. Apple has resources to do both repeatedly.

In this big corporation I mentioned tests were also done automatically. A device receives via infrared programmed set of commands in a loop for 24h. The test simulates a user with a remote.

Even so called gravitational tests are useful. A test where "only" gravitation acts on a device for 24h.

Those may seem dumb, but bugs can be hunted this way, that no automatic test could find. Because real hardware and release environment is messy.

That approach works for behavior and UX, but is woefully inadequate for security, sadly. The space for shenanigans is exponentially greater.

What do you mean? That is literally the job of QA. Even the worst QAs I've worked with test every input field into oblivion.

Reply on news.ycombinator.com