Hacker News Reader

Comment by jacquesm | Hacker News Reader

That is a very expensive mistake, but it could have been a lot worse, at least they made it to orbit. The orbit reached is a stable one and if it were a crewed mission at least at this point in time the crew would be fine. If they can't reach the ISS (which is what it looks like right now) they'll have to deorbit a bit sooner, there is a lot of money riding on them making it down in one piece, if they fail at that then you may want to trade in some (more) of your Boeing stock.

It's been a very tough year for Boeing, and SpaceX makes it all seem so easy that you tend to forget this stuff is very hard and that success is not at all guaranteed.

View on HN

Not sure SpaceX makes it seem easy. To me it's more like they get a free pass for a lot of failures because people like what they're doing and think the company is cool.

The story "cowboy innovates rapidly, kills someone" sells a lot easier than "old crusty firm attempts to take zero risks, floats on past success, still kills someone"

One thing I learned from a recent Everyday Astronaut video is that NASA has a long history of trying to calculate the chances of failure for a vehicle before it has ever flown. They got it wrong with the space shuttle for example. And this method requires a lot of extra factors of safety and expensive parts. An alternate method is to test in flight, and this is what SpaceX does. Boeing uses the other method. So Boeing systems are supposed to be relatively failure free, while some SpaceX systems may at some point be expected to fail. Of course SpaceX is supposed to have their testing done before humans get onboard, so those systems should still be reliable.

But a big difference is Boeing tests on the ground and SpaceX tests in flight.

People don't even remember the SpaceX failures. SpaceX CRS-1's secondary payload was left in an unstable orbit and ORBCOMM's $10 million prototype burned up in Earth's atmosphere about 4 days later. CRS-2 had an issue involving the RCS thruster, but was able to recover. CRS-7's launch vehicle catastrophically failed, the cargo capsule ejected, but its parachutes didn't deploy (had it been a crewed mission, the astronauts would have died). The $200 million Amos-6 satellite was lost in a launch pad explosion and Musk, being the paranoid weirdo he is, initially blamed it on imaginary United Launch Alliance saboteurs—for which there was an investigation by the Air Force’s 45th Space Wing at Musk's request.

Notably, ULA has a nearly perfect record (they've never lost a vehicle, only one or two payloads didn't reach the correct orbit) and ULA has launched nearly twice as many times compared to SpaceX. While the SpaceX's record is very bad by modern standards.

This is a highly disingenuous post.

https://en.wikipedia.org/wiki/SpaceX_CRS-1#Secondary_payload

> Both SpaceX and Orbcomm were aware, prior to the mission, of the high risk that the secondary payload satellite could remain at the lower altitude of the Dragon insertion orbit, and that was a risk that Orbcomm agreed to take given the dramatically lower cost of launch for a secondary payload.

As for this...

> CRS-7's launch vehicle catastrophically failed, the cargo capsule ejected, but its parachutes didn't deploy (had it been a crewed mission, the astronauts would have died)

Had it been a crewed mission, the parachutes would've been active. It's not standard practice to have a escape rockets or armed parachutes on ascent for cargo flights, SpaceX or otherwise. Standards are different for crewed flights.

For CRS-1 there was a risk of a lower altitude obit, that wasn't the outcome—they lost the satellite.

For CRS-7 Wikipedia says "SpaceX officials stated that it could have been recovered if the parachutes had deployed, but the software in the capsule did not include any provisions for parachute deployment in this situation."

> For CRS-1 there was a risk of a lower altitude obit, that wasn't the outcome—they lost the satellite.

No, that was the outcome. The lower orbit has much more atmospheric drag, resulting in deorbit within a few days. SpaceX and Orbcomm both knew this was a possibility. From Wikipedia: "Both SpaceX and Orbcomm were aware, prior to the mission, of the high risk that the secondary payload satellite could remain at the lower altitude of the Dragon insertion orbit, and that was a risk that Orbcomm agreed to take given the dramatically lower cost of launch for a secondary payload."

They're also a very young company in the space, with a fraction of the experience that the incumbents have. It's more understandable to make mistakes when you're that new to such a complex industry.

Is there any real US experience left with creating manned capsules for space? I would guess there are no Apollo engineers available, and Shuttle engineers for the same had a very different configuration. For general space experience, also the ability to maintain controlled processes for existing space vehicles is not a fully overlapping skill set with developing new space vehicles. In a way, every company is starting from new using historical data.

> Russian government Soyuz makes it all seem so easy that you tend to forget this stuff is very hard and that success is not all guaranteed

FTFY

> SpaceX makes it all seem so easy

To be fair, their version of this capsule blew up rather dramatically.

https://www.youtube.com/watch?v=xe4ee56aHSg

After that exact capsule had visited the ISS, incidentally, which must've caused a little butt-clenching at NASA.

As you said that was a recovered capsule from an ocean landing that they were testing. SpaceX will not be using recovered capsules for crewed missions.

To be fair they have not stated why it exploded. If it was damage during landing or retrieval then oh well. It could be a design flaw.

Edit: It turns out they did explain and I missed that news.

> To be fair they have not stated why it exploded.

They actually did explain:

https://www.businessinsider.com/spacex-crew-dragon-explosion...

> To be fair they have not stated why it exploded.

Yes, they have.

https://www.cbsnews.com/news/spacex-explosion-destroyed-crew...

They redesigned the capsule to not need the valve.

Well I clearly missed that news update! Whoops.

The capsule blew up while firing the abort motors. If the abort motors fired while the capsule was anywhere near the ISS, it wouldn't have mattered if it exploded or not.

> The capsule blew up while firing the abort motors.

To be clear, it blew up when pressurizing those motors' systems, not during firing. As you note, that makes it unlikely to have happened in orbit, but it's still a major concern for future crew - it's a step that'd happen in every launch. Hence the redesign.

To be clear, "pressurizing" is part of the "firing" process. It occurs ~200ms prior to ignition. It is not a step that would happen during every launch. It would only happen if the SuperDracos were fired (in an abort scenario).

This was a serious flaw in a critical system, and the redesign was very important. It also advances the state of the art in engineering in these sorts of systems... the failure that took place was something that no one considered possible (specifically the Ti/NTO reaction).

I was just responding to the fact that you mentioned the capsule had visited the ISS, which is totally irrelevant to the failure.

No doubt this will still advance the state of the art in aerospace systems, but there's some old data by Boeing and NASA that seem to indicate some potential compatibility issues with Ti in the presence of a strong oxidizer.

See PDF pg. 66 (not report pagination) https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/197200...

See PDF pg.11 https://apps.dtic.mil/dtic/tr/fulltext/u2/866010.pdf

Or the classic book "Ignition!" pg 70-71.

On December 29, 1953, a technician at Edwards Air Force Base was examining a set of titanium samples immersed in RFNA, when, absolutely without warning, one or more of them detonated, smashing him up, spraying him with acid and flying glass, and filling the room with NO2. The technician, probably fortunately for him, died of asphyxiation without regaining consciousness.

There was a terrific brouhaha, as might be expected, and JPL undertook to find out what had happened. J. B. Rittenhouse and his associates tracked the facts down, and by 1956 they were fairly clear. Initial intergranular corrosion produced a fine black powder of (mainly) metallic titanium. And this, when wet with nitric acid, was as sensitive as nitroglycerine or mercury fulminate. (The driving reaction, of course, was the formation of TiO2.) Not all titanium alloys behaved this way, but enough did to keep the metal in the doghouse for years, as far as the propellant people were concerned.

Different oxidizer from SpaceX, but still useful as a data point.

Big difference between blowing up during a test on the pad or blowing up while going up to mate with the ISS.

Certainly - there's no reason that system would've been pressurized while in orbit. That won't have prevented NASA from getting mental images of the ISS blowing up, though, and it would still have been a concern for crewed launches where the abort system would have been pressurized; it's good they redesigned to prevent the possibility at all.

There is a big difference in the consequences, but the fact that it was a test does not, of course, mean one can dismiss the question of whether, and how, a significant problem had escaped all efforts to eliminate unnecessary risks up to that point. Once something has reached the point where it is considered nominally ready for use (or has actually been put in service), the failure of a test conducted within operational parameters should be treated as seriously as a failure in actual use.

Not that it shouldn't be, or wasn't, treated seriously. But it was very far from operational parameters. It was a previously flown capsule that had been dunked in sea water. Crew capsules are not re-used.

There's no indication in the post-mortem that the issue had anything to do with reuse or immersion.

> "We believe that we had a liquid slug of the (NTO) in the pressurization system," Koenigsmann said. "When we opened the valves and pressurized the propellant system, we think that this slug was driven back into the check valve. ... That basically destroyed the check valve and caused an explosion."

Hence the redesign.

There's also been discussion of reusing Crew Dragon on unmanned CRS missions. https://www.teslarati.com/spacex-no-crew-dragon-spaceship-re...

> Although the OIG report in question never specifically states it, some of the language used to describe Dragon 2’s cargo configuration does seem to imply that Cargo Dragon 2s will predominately (if not exclusively) be derived as slightly-modified Crew Dragon capsules, seemingly indicating that SpaceX’s CRS-2 missions may only ever launch flight-proven Crew Dragon capsules.

That might or might not have been relevant, and the fact that something was changed as a consquence of the investigation argues strongly for the latter.

I do not understand the effort being put into arguing that this failure was largely inconsequential because it was a test. Clearly - and rightly - neither SpaceX nor NASA took it that way.

> I do not understand the effort being put into arguing that this failure was largely inconsequential because it was a test.

That's not what I'm attempting to argue... and why my post starts with a disclaimer that I'm not trying to argue that.

I just think it's important to keep the facts straight.

It is also important to identify which facts are relevant.

And yet it's still important to not misstate facts regardless of whether or not they are relevant, because other people will take those supposed facts and apply them in other ways then you did.

And yet it's still important to not merely present facts regardless of whether or not they are relevant, because other people will take those facts and suppose they are relevant.

Note the issue that unites these two statements - it is relevance. A test does not have to be a simulation of actual use to be relevant.

Let me say this in a different way. As soon as you misstate a fact in a public forum where people might not know the truth of the matter, that misstatement becomes relevant to many readers. Because otherwise people will read your supposed fact, assume it is true, and apply it elsewhere.

That doesn't make it necessarily relevant to the original point of whether or not it was an issue that needed to be taken seriously, but that's why I explicitly started my post by stating so.

I honestly can't believe that you're defending a serious misstatement of the conditions of the test on the internet by saying "well it doesn't matter that the premise of my post was wrong". Of course it matters, even if I agree with your conclusion anyways.

Could you clarify what fact I misstated?

The capsule blew up. True.

The capsule blew up after already having visited the ISS. Also true.

Wasn't much more to my post.

Not you, mannykannot, the person I replied to in this thread.

> the failure of a test conducted within operational parameters should be treated as seriously as a failure in actual use.

Everything before the "should be".

The premise of my post is not wrong, because if you read what I actually wrote, you will see that nowhere do I state that this test was conducted entirely as if it was a real manned mission to the ISS - of course, it wasn't, including for all sorts of facts that do not have anything to do with the previous history of the capsule. What I actually set out, in my first post, is a principle for regarding tests as being as significant as real operations, and it so happens that this principle still largely holds even when a test deviates to some extent from a real operation. I did not state this obvious extension because I did not anticipate a pointlessly pedantic response.

Until the failure had been diagnosed, it would have been correct to consider whether the problem was entirely the consequence of the capsule's previous history, but by now, we are all - even you - agreed that the test did reveal a potentially serious problem that could be fixed. It turns out, fortunately, that tests do not have to be equivalent to live operations to be useful, because relevance does not depend on them being so (as this case demonstrates.)

> the failure of a test conducted within operational parameters should be treated as seriously as a failure in actual use.

The premise, that the test was conducted within operational parameters, is not true.

Undeniably, this does not say that this test was conducted entirely within operational parameters - as I wrote before, it sets out a principle for taking tests seriously. It also, intentionally, suggests that you consider this test as being close enough to be relevant, and you have offered no argument against that.

> significant problem had escaped all efforts to eliminate unnecessary risks up to that point.

This test was part of the process of identifying and eliminating those risks.

There is no difference between a failure during a test and a failure during operation, in terms of identifying needed changes in space craft design.

There is a difference when it comes to evaluating the design and testing process of the company. SpaceX seems to be very good at balancing moving quickly with design, then testing safely, using those test to find weaknesses and learning from their failures to iterate rapidly.

Despite some early failures, it seems SpaceX is rapidly on its way to having the best launch safety record in the industry.

All I am saying is that by the time you are flying the thing, you should have high confidence in it. A failure at that point should lead to a serious reevaluation of that confidence, regardless of whether the failure occurs on a test or an operation (the exception being those tests that are to destruction, or at least well outside the operational envelope, and only then if the failure occurs at about the point you expect, or beyond it.)

I agree that SpaceX is looking very competent.

> All I am saying is that by the time you are flying the thing, you should have high confidence in it.

It seems like there is a balance between the level of confidence you achieve prior to a flight test, the costs of a failed test, and the speed at which you can implement and iterate designs.

> A failure at that point should lead to a serious reevaluation of that confidence,

If, as in this case, the failure is due to non-considered risk, you certainly want to make changes to the assumptions you make during the design process in the future (in additional to the changes you make in your current design.)

I don't think you need to generally question your confidence in your components that have not only passed all their operational tests, but have been used successfully to complete operational missions.

Only because they were shaking it to bits while trying to light the emergency abort engines as a shakedown test for Nasa, reusing a capsule they had used before.

So I don't think that's at all a fair comparison.

Vibration testing is common practice on accepting a particular engine design. All engines will typically perform this type of test and be expected to pass.

I agree, but these are very different kinds of error.

I agree if you mean the difference between the MET error and the vibration failure. I tend to disagree if you mean the vibration failure mode in a test stand vs. a potential vibration failure mode on a mission.

Yes that's what I mean.

"Our emergency abort engines explode if you shake them a lot" is not a great feature in an emergency abort engine.

That they wound up redesigned indicates NASA and SpaceX agree.

This is why we test things.

Sure, but that doesn't mean you don't hope to pass the test.

Well yeah, but at the same time, you want anything that could possibly go wrong to go wrong before you finish testing. A test that finds a design flaw is a successful test.

> 176x144@10fps MMS quality video of the SpaceX Crew Dragon anomaly

Is this a covertly taken video? Why so low quality?

Yes, it was a leak, from a NASA employee/contractor.

https://www.space.com/spacex-crew-dragon-explosion-nasa-memo...

If I remember correctly, it was cell phone video taken by someone in the control room at the time of the test.

That was a on-ground test, specifically intended to minimize launch miss-hap. Huge difference.

This program revenue wise is a drop in the bucket for Boeing, much less than 1%.

It's expensive because Boeing's reputation is very much at stake these days.

Your Boeing cost prediction don't include the possibility of catastrophic RUD during reentry (still 2 days out). At that point does NASA cancel starliner?

There is simply no way that the US government will let Boeing fail as a company. Even if this thing blows up, I think it will still be funded, especially given their commercial airline troubles.

In terms of reputation that would be very bad news for Boeing.

SpaceX is basically the Google of aerospace at this time, Boeing is like IBM. Their interview process is incredibly rigorous while Boeing's is a joke. I've worked with people from both places (former aerospace engineer), their respective workers are not in the same league.

Reply on news.ycombinator.com