After six months on Mars, NASA's tiny copter is still flying high | Hacker News Reader

After six months on Mars, NASA's tiny copter is still flying high

phys.org

347 pointsPikkie82 comments

One of the most interesting things about Ingenuity is the amount of standard open-source code used. Github has a badge for if you contributed code to a project used by it, and they put up an explanatory page at https://github.com/readme/featured/nasa-ingenuity-helicopter.

Having that badge would give anyone very cool bragging rights about how they wrote software that's running on Mars.

It makes perfect sense that NASA used a lot of open source software for this. If some high quality, widely used and tested code already exists, why not use it instead of trying to reinvent everything?

> If some high quality, widely used and tested code already exists, why not use it instead of trying to reinvent everything?

This is precisely the spirit of free software and the motivation behind movements like public money public code. Govermnets immensely overpay for software (projects).

I highly doubt it's running on Mars. For one thing, sibling comment has a list of repos that have the badge, and it's a lot of python packages. I suspect it's more along the eath-based infrastructure and data processing side of the equation. Still a pretty cool get, though.

Apparently some of the non-mission critical experimental components that were sent to Mars run your standard Linux systems with some Python software, as well. I believe I heard in an interview with a NASA engineer that they were using Python for ML and scripting.

Well, I just checked and I got the badge! Go me! I wonder which of the tens of open source libraries I've written was actually used by NASA!

Oh, the badge tells, you... Hmm, let's see, it says "attrs". That's not mine, I must have submitted a PR at some point, let me find it.

Ah, right. I changed "serious business aliases" to "serious-business aliases" in the README.

You guys are all welcome.

Mine was some CMake fixes for cURL when using it as a subproject in a "superbuild" project.

Still better than adding a hyphen...

I literally worked on the heli, but my badge came from a documentation patch to numpy :)

I once spent a week troubleshooting a pair of routers that were part of a NASA Mars mission. They were using firmware over a decade old at the time, which was causing issues with the internet facing edge devices, which had automatically updated a few weeks prior, and that started causing massive packet loss.

There were redundant paths with redundant hardware, and the node I worked on wasn't particularly important, but I got a taste of what the engineers and scientists at NASA must live for, the sense of contributing to something historic, profound, and deeply human.

The fix was simple at the end, a firmware rollback and disabling further updates, and most of the time was consumed in conversation and carefully assessing each step and action, but I count it as a high point in my career. Just to touch the edge of it was special.

The people whose code and engineering get sent to space are contributing to the betterment of mankind, literally making the universe a better place. Kudos to those guys. To write code that's part of a critical system is a meaningful in a way

And a list of the repositories. Very cool.

https://docs.github.com/en/account-and-profile/setting-up-an...

It’s been six months already? How time flies…

Well it's been nearly 11 months in Martian time... :)

But Mars takes 1.88 earth years to make one orbit. So, 6 earth months is closer to 3/12 of a Martian year.

Yea, same reaction, I had to double check it's true. Damn, we're getting old.

Only 4 and a bit months since its first flight though, when it was all over the news.

Maybe after the shift from space-grade parts to commodity parts in (some) satellites, this will mean the same for parts of future mars missions. A Snapdragon 801 as used here is certainly much cheaper, easier to work with and powerful than anything space-grade.

It really depends on the space environmental work conditions, though… that’s the fundamental trigger for space grade quality.

From my limited knowledge of electronics-in-space, isn't ECC memory a hard requirement to correct for bit flips due to solar radiation/particles? But DDR5 having ECC be standard across the board may make this a non-issue.

Kinda, yes, but at the same time, servers here on earth have moved away from (expensive) ECC in favor of resilient software, disconnected services (job queue based) and redundancy. Cheaper to have a dozen computers that sometimes break than three with lower risk (but they still break).

For consumers no but for things like large simulations or even deep learning you want to have ECC memory.

Intel only enables it on Xeons but AMD doesn't officially support it on their consumer chips but sometimes it works.

It's locked down on AMD customer APUs, only not forcefully disabled on AMD customer CPUs that do not have integrated graphics. (Ryzen PRO parts have it officially supported)

It’s also such great marketing. I wonder why the Zigbee alliance doesn’t communicate more that it’s used for communication between the rover and the helicopter.

Probably because they'd get ridiculed to beyond the solar system that they can make a Mars rover fly, but it's a nightmare to get two Zigbee products from different vendors to talk to each other.

There's also some RAD hardened components to check for bit-flips and restart the hardware that quickly it can do it mid-flight.

Here's the full paper. https://trs.jpl.nasa.gov/bitstream/handle/2014/46229/CL%2317...

That is so smart. I know we're talking about NASA, but I am still in awe.

I read a bit of the paper, it's actually even more redundant than what you said:

- snapdragon is only used for higher level functions of the flight: The SnapdragonTM processor has a 2.26 GHz Quad-core SnapdragonTM 801 processor with 2 GB Random Access Memory (RAM), 32 GB Flash memory, a Universal Asynchronous Receiver Transmitter (UART), a Serial Peripheral Interface (SPI), General Purpose Input/Ouput (GPIO), a 4000 pixel color camera, and a Video Graphics Array (VGA) black-and-white camera. This processor implements visual navigation via a velocity estimate derived from features tracked in the VGA camera, filter propagation for use in flight control, data management, command processing, telemetry generation, and radio communication. The SnapdragonTM processor is connected to two

- for flight control they use 2 (redundant) automotive chips from Texas Instruments: TMS570LC43x high-reliability automotive processor operating at 300 MHz, with 512 K RAM, 4 MB flash memory, UART, SPI, GPIO.

- these are all controlled by a radiation hardened board, the 3rd level of redundancy

They also used a $129.99 altimeter from Garmin (Lidar-Lite-V3), no fancy fancy. https://www.sparkfun.com/products/14032

edit: https://www.garmin.com/en-US/blog/general/garmin-on-mars/

I wonder how much hardware marketed as "space grade" (or whatever the terminology is, I'm just a layman) is actually as hardened using these techniques and redundant as you'd expect. Or are they simply certified because the manufacturer's process went through some checkboxes. In the aviation world, for some components, the difference between aviation grade and non- is 1. cost and 2. a Certificate of Conformance from the manufacturer that says "this thing is provably aviation grade, trust us bro."

Anyone can call their stuff "space grade", but it is only marketing copy if it isn't flight qualified. I don't track new stuff because my interest is in hardware developed in the 80s and 90s, but for that time frame there are long lists of qualified parts and published testing methodologies. I know that for military grade stuff of that time you had a combination of distinguishing characteristics that set one part number apart from the commercial grade offering: top bin draw and complete performance characterization. At some point that was relaxed to where it is now just mostly about binning and traceability.

I believe it is much cheaper to put something in LEO then it is to get it to Mars, so I don't know if the economics of Mars Missions have fundamentally shifted to cheaper probes the way they have for satellites.

I suspect we're limited more by communications capability then anything else: pretty much a hand-held radio will reach up to LEO if someone's listening (short-wave definitely does and that's off-the-shelf at consumer prices pretty much).

Whereas Mars requires the Deep Space Network to reasonably be able to maintain communications - so your very cheap probes still need super-highgrade radios to be useful (the landers for example still generally pack an antenna that can reach all the way back to Earth even when they use uplinks - the helicopter is relaying via the lander).

What we're missing on Mars is reliable communications for probes: there's no common backhaul to Earth that you can just hook into. But I'd be willing to bet that's coming: if Starlink works on Earth, there's no real reason it shouldn't work on Mars. At which point you can go with the "swarm of cheap hardware" idea because it can hand off the need for the power, hardware and logistics and backing to communicate inter-planetary.

> Whereas Mars requires the Deep Space Network to reasonably be able to maintain communications - so your very cheap probes still need super-highgrade radios to be useful

That's not technically true. They maintain fancy radios that can reach Earth as a backup. Data transfer is done via the Mars Relay network.

https://mars.nasa.gov/news/8861/the-mars-relay-network-conne...

Ingenuity uses this as well.

> if Starlink works on Earth, there's no real reason it shouldn't work on Mars.

Ground-to-low orbit comms relays have been in operations for half a century. And decades at Mars already. Starlink is for high throughput communications from ground to ground on the same planet with always-on connectivity and global coverage. Their sattelites are engineered cheap and replenished with cheap lift from SpaceX. Totally different problem calculus at Mars and doesn't address "backhaul" to Earth.

You'd need longer-lived high throughput optical terminals or better, preferably at Areostationary orbits. Checkout work by Briedenthal and Edwards on the matter.

I'm not a space expert at all but my wild guess is that doing small, careful tests for consumer grade hardware (and software) to be used on Mars is a way to lower costs of a possible future settlement, by leveraging economies of scale here on Earth.

Anyone knows if the copter suffer already from issues and reboots already?

I actually just wrote the policy on this topic for a space project I work on.

It really depends. It’s often not cheaper to use such commodity stuff unless you can tolerate a lot of risk. A lot of normal and automotive components are used in space, but it’s mostly simple stuff that’s relatively easy to qualify. A snapdragon cpu will basically never be fully qualified as a single component because you cannot control each state, let alone test it in each state (I am oversimplifying but bear with me). What you can do is apply overcurrent detection, external watchdog timers, redundancy and other techniques to mitigate the effects of radiation.

Then, you can almost fully test that mitigation in theory but not really because it will cost you millions of dollars with huge technical risk and the part will probably be obsoleted or made by a different fab next time you need it. So, you choose older proven easily-hard stuff where you can, or for anything mission critical or related to safety. Better to just buy the $200k rad hard cpu and save your millions.

Then, you are left with only bonus items such as this helicopter that simultaneously do not have many reliability requirements (because it’s not mission critical) and are not possible with traditional space grade technology due to performance requirements.

“Easier to work with” is only true when you take risks too. Yeah, you can start developing right away using a devkit and Linux, but if you find your project needs reliability or safety requirements, you may find you are spending person-years of software engineering time if it’s even possible at all. You might be required for the system to be deterministic and avoid dynamic memory allocation, for example.

Interestingly, students making cubesats don’t “know better” yet. And while the overall failure rate is probably bad on a per mission basis, it’s probably good on a per kg or per dollar basis. There’s a ton of cubesats and cubesat payloads that have raspberry pi’s inside for Leo missions.

Yeah but part of why they used a high speed arm, is so they could do realtime AI fo determining where to land and what not. If you use a RAD hardened chip you are limited to what 400-500mhz chips from over a decade ago. Fine if your doing a rover that doesn't need to do realtime detection and avoidance of stuff.

However they still use a rad-hardened FPGA to run some of the critical aspects. This paper discusses a lot of the internals.

https://rotorcraft.arc.nasa.gov/Publications/files/Balaram_A...

A bigger effect is that, by the time the next helicopter is ready to launch, SpaceX Starship is likely to drop the cost of delivering cargo to Mars by an order of magnitude or more. That will massively disrupt the mass vs. capability trade-offs we see in all Martian rovers and aircraft.

About the size: the blades are really huge "in person" (in VR :-)

I added both the Perseverance Rover and Ingenuity to our VR puzzle game Peco Peco[0], and I was totally surprised both times by how bigger than I thought they were. (Players can assemble the puzzle for Ingenuity at full scale, and at both full scale and 1/3rd of scale for the rover)

So if you have a VR headset, I highly suggest to find a way to discover Ingenuity at full scale in VR.

[0] https://pecopecogame.com/

That looks cool! How do I play it if I don't like giving Facebook any data?

This is a good lesson in under-promise, over-deliver. NASA seem to be masters of that technique.

I always took it to be a budget defense method since they never get as much money as they as for, and have to expertly manage expectations to ensure they continue getting the “little” money they do.

Little is quoted because it’s a hotly debated and subjective word.

I saw first-hand how it comes to pass, from collaborating with NASA engineers years ago.

In order to ensure that a critical mission-goal is met, with, say, 95% confidence, every subcomponent must be much more reliable, as there will be many ways to fail. The net effect is that often the overall perceived reliability turns out to be much better than the requirement.

This is especially so for systems that have already achieved some measure of success, as a number of those subcomponents will have already done their jobs completely.

I suspect it's because of a solid physical engineering culture. They probably have enormous safety factors and redundancies or work-arounds, since you can't change anything physically once it's launched.

So if things go alright, those provide lifespan far beyond specifications.

Eh, at this point it's a boy crying wolf situation. Whenever there's a new space gizmo you know they're going to say "we only designed it to work for eight minutes", and then six months later it'll still be chugging along, because the actual design life was measured in years.

They could at least say "the doodad is designed to work for two years, but there's a 5% chance that it'll be DOA, because so many things can go wrong in space". That would be more realistic.

The problem is people hear what they want to hear. Much like telling a PM that a project is probably going to take 2 weeks but might extend to 2 months. Guess which number they are going to remember and hold you to?

Not really. They are putting experimental hardware into an environment that is not well understood. It is also being done by very public facing organizations with very large budgets. They basically have to guarantee that they will deliver what they say they will deliver. From a public relations standpoint, it may even be easier to justify a complete failure than an 80% success since a string of 80% successes will look like incompetence while a few complete failures are quickly forgotten.

It is always an exercise in horrible estimates and it gets really tiresome. If you always greatly exceed the estimate (by orders of magnitude!) it wasn’t a good estimate, it was PR. I assume NASA has real estimates they use internally. No one could be this bad at it and still have a job there surely.

you don't deserve to be downvoted (this place is ridiculous) because saying that there's a "boy crying wolf" aspect is true, and it was the whole point of learning that story which point apparently some people missed.

but there is also a position I'd like to point out between the one most people here are taking ("NASA over-engineers every part individually to avoid failures")

and yours ("here's our reasonably expected life, but could come up short")

and that is: they need to set a bar that evaluates a reasonable amount of science they need to accomplish to justify the budget.

It's not the minimum number ("crashed in flames but we learned some things")

and it's not the gloriously optimistic number ("this thing's still going after 12 years, it's really worth it!")

and it's not a fake marketing expectations number ("we'll fake the budget numbers so we can razzle dazzle by blowing past the expectations")

rather it's just, "for X hundred million we expect to spend a month taking pics in every direction and gathering and analyzing some rocks from different promising places, and that mission alone is worth it" because that's what congress is approving

> This is a good lesson in under-promise, over-deliver. NASA seem to be masters of that technique.

And Tesla seems to be master of the exact opposite technique with Autopilot.

It's far from clear that NASAs general conservative nature has been a good thing relative to a more risk-tolerant one. If we accepted more failure, could we accomplish a lot more?

Particular with SpaceX on the scene, it's time to start rethinking risk, because launch costs will be so much lower.

Have you seen the recent FSD videos? It's really quite impressive. It still makes occasional mistakes (that humans also make) but I find it quite astonishing how far it is.

Interesting craft in that it's essentially just a larger version of the R/C toys you can buy. I wonder how they tested the blade design. Did they fly it at 100,000ft on Earth?

Four words: really big vacuum chamber.

I assume it was tested in a pressure chamber on earth to simulate the pressure and composition of Mars atmosphere.

Obviously it wouldn't be able to fly in the pressure chamber due to increased gravity, but by suspending it on elastic you can still test that the blade functions as intended.

The only bit you can't really fully test is the constants for the control loops in the flight control algorithm, but I assume they chose them with a lot of stability margin.

> I assume it was tested in a pressure chamber on earth... by suspending it on elastic you can still test that the blade functions as intended

They did exactly that. NASA has a big (adjustable) vacuum chamber for this kind of purpose.

This comment links to video: https://news.ycombinator.com/item?id=28432811

It did fly in the vacuum chamber. It was a tethered flight and the difference between Earth‘s and Mars‘s gravity was compensated by an appropriate pull being applied to the tether.

Just a vacuum chamber: https://www.youtube.com/watch?v=tMCJGfwj3rY

Didn't it also use Zigbee (for home domotics) as a control protocol?

Veritasium has a good explainer about the development and testing: https://www.youtube.com/watch?v=GhsZUZmJvaM

I'm really interested to see how far consumer electronics will actually be proved useful in space. There's this perception that designing for space is ultra-difficult and requires n-tuple redundancy and specially fabricated radiation-hardened processors etc. but I wonder how much of that is just because when you're spending a billion dollars on a mission you REALLY don't want to lose it to some random bit flip and so you drop $100mil on goldplating the absolute heck out of everything.

I think people who fly tiny whoop quadcopters would find the description of a 1.8kg copter as 'tiny' quite amusing. I'm guessing the primary driver for the weight is the power to lift this in the thin atmosphere, secondary to that the weight of the strong materials used to make it durable.

Not just the power to keep it up, but also the rotors. Not sure how heavy Earth quadcopter rotors are as a proportion of total vehicle weight, but the Ingenuity ones are absolutely enormous.

Saw this Veritasium youtube video a year ago and the technical complexity required to fly on Mars was shocking to me. Since the air density is super low the blades need to spin at a speed which nears the speed of sound at the blade tips. Crossing the speed of sound causes mini sonic booms which could disrupt the flight entirely. Kudos to NASA scientists for this technological breakthrough! Sharing the video: https://www.youtube.com/watch?v=GhsZUZmJvaM

Can't wait for a successor to Ingenuity to capture aerial video during its flights.

I wonder what the upper limit on weight will be. The article mentioned multiple kilograms being possible in the future.

Have a swarm instead/as well, much more capable at covering range.

JPL is planning already much bigger copters:

> The current MSH concept has a mass of about 31 kg and a total diameter of just over four meters, with six rotors each sporting a quartet of 0.64 meter blades

https://spectrum.ieee.org/the-next-mars-helicopter

It's incredible to me they managed to get it to fly at all considering how thin mars' atmosphere is.

Do we have any amazing pictures from it?

Half of the pictures on the NASA website are rendered mockups. I don't know which ones are real anymore.

They always write in the image description if it is an illustration or not.

Or use the raw image library https://mars.nasa.gov/mars2020/multimedia/raw-images/

Ingenuity only has one downward facing color camera and b/w nav cam. None of them are particularly amazing but its just a tech demo. You can see all the images here, https://mars.nasa.gov/mars2020/multimedia/raw-images/

Seeing color pictures from another planet's surface is one of the things that always fills me with awe.

Depends on your threshold for amazing. It's a lot of pictures of rocks since they're cameras for navigation. On the other hand they're aerial views of another planet taken by a helicopter during autonomous flight.

One notable photo was on Ingenuity's third flight, where it happened to get the Perseverance rover and landing area together in one frame: https://mars.nasa.gov/resources/25862/ingenuity-spots-persev...

There is also the NASA blog just covering Ingenuity, which has some more notable photos along with explanations: https://mars.nasa.gov/technology/helicopter/status/

It could hover near the rover and take nice photos. But I'm sure NASA doesn't want to take a collision risk like that.

Someone has used the low-FPS, downwards-pointing color camera plus photogrammetry to reconstruct the flight environment: https://www.youtube.com/watch?v=tX89Y766D_M

> ... still flying high

Shirley you must be joking!

Jesus, has it been six months already? What is this hell world.

Wait, you're on Mars too?

It's less than four months until 2022.

Quote: "NASA is planning to retrieve those samples during a future mission—sometime in the 2030s. "

Not if Musk gets his rocket filled with 100 people on Mars in next years. Then you'll have a better way to do science on that soil.

Reply on news.ycombinator.com