Hacker News Reader

Comment by HidyBush | Hacker News Reader

It seems to me that what the FOSS world needs in order to succeed is an extreme push towards forming a class of programmers specialized in reverse engineering. We already have plenty of capable coders who have written so much useful software, but does is matter if we can't use all the hardware at our disposal? Our devices are filled to the brim with accelerators, sensors and assorted specialized hardware that we could be taking advantage of, but they are all buried behind proprietary drivers or just plain inaccessible firmware.

View on HN

What a waste of human potential that would be.

Don't get me wrong those peoe are heroes. They're literally sacrificing their life time for a good cause. But like with many such many hero stories, some bureaucrat could have resolved the issue with the stroke of a pen.

What we need is an extreme push against a culture of secrecy and disempowerement of device owners.

What is done is done. Even if some piece of legislation were to come out tomorrow all the older devices would still be completely closed off. Would we have to collectively just push to move to the new open-by-default devices or should we persist in reverse engineering the older stuff?

That's not how laws work.

Are all laws retroactive? The EU has now mandated the use of USB-C, does that mean I can bring my iPhone 4 to Apple and have them solder the new port?

Laws can say you cannot sell any more X unless it complies with Y. That's not a retroactive law.

I don't think you've read my reply correctly. If a law were to come out that said "you can't sell any more X unless it complies with Y" we would still have billions of devices that don't comply with Y. So even if a law about open hardware/firmware were to pass in the next few years, we would still need a huge reverse engineering effort to open up all the older devices. Unless you want to argue that, yes, we should forget about those older devices and all jump on the new compliant platform, I think pushing hard for good reverse engineers is a very implortant priority

> So even if a law about open hardware/firmware were to pass in the next few years, we would still need a huge reverse engineering effort to open up all the older devices.

Not necessarily. Impose a tax on all products where the manufacturers do not release all documentation even for past devices, that should serve as a pretty decent incentive.

Yes, legally it's a questionably grey area, but to be honest it's time to actually use our market power.

Not all are, but they can be. There's typically a bias (can't remember the name, but it's a known concept) to not change the legality of past actions. But there have been and could be situations where previously-legal actions were retroactively outlawed and punished. It comes down to a decision per situation: what is a higher good: the individual's ability to rely on my current action to be legal in the future if it is legal now, or rather the ability of society to deter from things even though the laws haven't been updated yet. Prominent examples of this (sorry for the Godwin spin here) are punishments of Nazis post-war, for actions that were technically legal during the Nazi years. The argument being "being able to rely on the law not being applied retroactively is not a higher good that stopping people from doing clearly immoral things."

Nazi warcrimes were punished retroactively for a number or reasons, political and not. One of the most important ones was that what the Nazis did was blatanly wrong. You can't seriously claim that killing millions of people can be justified simply because there isn't a law about it, it's something so intrinsically evil that you can't rely on pure formality.

Arguably, refusing to publish literally every single document pertaining to proprietary hardware is not on the same level of obivous malpractice as a genocide, so I think you could have proposed a milder example to argue your point.

Fair point, not sure why you're being downvoted...

I would never dream to equate proprietary documentation to Nazi war crimes. That's without question. I just used this because, like so often, in questions about these war crimes, there is no real argument to be made for "the other side" so it clearly shows the line of reasoning for breaking with a legal convention in specific cases. You said it yourself: "it's something so intrinsically evil that you can't rely on pure formality." Which is exactly the point. Protecting the perpetrator just because there's also a value in relying on their actions being legal at the time just doesn't outweigh the cost of letting them get away with such monstrosities.

A more recent and less evil example would actually be the Cum-Ex scandal. One question was whether money would have to be given back, given that what happened might be technically-legal but blatantly, and expressly against the spirit/intention of the law. But that whole thing is still being fought out so the conclusion is less clear cut.

Some laws are retroactive, some aren't. And, anyway, "you must publish the specs of any hardware that you keep selling" isn't retroactive in any way, but would solve most of the historic problem.

That is like saying it would be a waste give aid to refugees in war zones because we shouldn't have wars in the first place. There will always be push back against open hardware as long as it is profitable.

Not having a war is much harder than approving the release of a bunch of documentation PDFs that you have already.

You’re wrong about it being a waste of potential imo.

It’s real world training for reverse engineers. Having worked in that field myself I can tell you I was never surrounded by so many smart, talented and exceptional people as when I worked with other reverse engineers (especially in the video game world).

RE is a necessity, and it paves the way for some incredible talent to shine and practice.

I think the argument is that in a better world where everything came with open specs we wouldn't need reverse engineers and those smart people could spend their time on something else. It's an argument pointing out where we live away from the global optimum.

Yes and my argument is that we might lose out on the skills being developed by fantastic reverse engineers :)

A global optimum would include some other way for these legends in the making to train.

They'd be solving other low level problems instead I assume. Systems that are unintentional blackboxes instead of intentional ones.

Completely agree. I took matters into my own hands with my laptop. Wasn't able to reverse engineer everything but I got many features working on Linux and the result was better than the proprietary Windows app.

I wish I was knowledgeable enough to tackle more complex hardware like webcams. Perhaps one day I will be.

> I wish I was knowledgeable enough to tackle more complex hardware like webcams. Perhaps one day I will be.

https://lkml.org/lkml/2020/9/17/363

People who started the job on Surface webcam hardware didn't even know C to begin with ;)

Yeah that's impressive... I have to step up my game then. :)

I was able to intercept and reverse engineer much of my laptop's USB features. I hit a wall when I tried to figure out the ACPI stuff. There's some fan control functionality in the Windows app that's missing from my software but the fans are terrible and always at 100% anyway so at some point I decided to just let it be.

What about if we just form a class of programmers that exclusively work with Linux and buy Linux compatible laptops? That would be my preference.

What's the point of that? One of the benefit of having a FOSS platform is that you can keep supporting and updating hardware and software that in the normal proprietary economy would have been abandoned a mere few years after entering the market. If instead of pushing for reverse engineering we simply made more and more people buy Linux-compatible devices (which still doesn't mean they'd run FOSS) we would have a humongous amount of perfectly fine hardware locked down and unable to be properly used.

I think if say 15% of buyers demand Linux support, then it can become a standard feature of most sold equipment. Just like Windows support.

> Just like Windows support.

Completely non-existent? Either a laptop comes with windows or you get zero support. See every chromebook ever.

That's martyrdom and it isn't even for a good cause IMO. If someone buys Windows-only hardware that loses support after a few years, that's on them.

> If someone buys Windows-only hardware that loses support after a few years, that's on them.

Unfortunately, the FOSS world isn't always able to cater to professional and prosumer crowds.

If it is absolutely critical to my work that a piece of hardware has feature X, then I'm going to buy it regardless.

I'd choose a FOSS-compatible solution if it were available, but this hasn't been the case many times in my career. (Yes, I can provide concrete examples if necessary.)

If you're reverse engineering you've already lost. Long-term it's much cheaper to buy FOSS-friendly hardware (AMD+AMD).

You are absolutely correct. We need an organisation that channels lots of money into hardware and software reverse engineering. This is of course how Linux started (without the funding); developers doing reverse engineering in their spare time on the hardware that they had and wanted to use.

Judging by Asahi, FOSS will be fine.

The real question is where the hell is the open source hardware?

Engineers working for NVIDIA, or camera IP core specialists are not gods. Let's just make our own god damn camera and GPU.

On that note, I recently read that Asahi now has pretty much a finished driver for the M1 GPU. How did they pull that off that quickly? Was there extensive documentation available?

Thanks to nouveau being mediocre at best and lagging 2 or 3 generations behind until it has usable support for a GPU, I always assumed writing a decent GPU driver through reverse engineering must be next to impossible.

So what's up with Asahi? Were there just some gifted Wunderkind working on it? Is the architecture so much simpler? Is there more information available? Iirc it's based on powerVR, so not completely unknown, but then again the nouveau team got some hints from Nvidia in the past, and also they have been working on it for a really long time in comparison. Just really curious.

https://news.ycombinator.com/item?id=25873887

(“Dissecting the Apple M1 GPU, Part II”)

nouveau has the problem that they rely on nvidia signed firmware blobs for some functionality (like reclocking), and if nvidia don't want to give them those blobs, then they can't use the features the blobs enable. With nvidia's Linux kernel driver going open (but firmware and userspace staying proprietary), the firmware will be released separately and be redistributable, so nouveau will be able to use it.

> if nvidia don't want to give them those blobs, then they can't use the features the blobs enable

Nvidia is giving these blobs to Windows users who download their drivers, so why can't they write a script/etc that extracts blobs from a Windows driver binary and let the user download that binary manually?

They are giving the blobs to Linux proprietary driver users too. I'm not sure why that isn't viable, but they definitely did it for very old unsigned versions of the firmware on old GPUs. I guess they got tired of the upgrade treadmill if they aren't doing it any more, but probably they have such a script for their own reverse engineering efforts but don't want to deal with the support issues from when users try the script against newer driver downloads etc.

> I recently read that Asahi now has pretty much a finished driver for the M1 GPU

Where have you read that? According to Asahi Lina's tweet from a few days ago, they are just starting with it:

https://nitter.net/LinaAsahi/status/1559408257965309952

Hmm, I can't seem to find it, but it was some article maybe half a year ago, where in the end they managed to render some model of a rabbit with shaders and all. Am I remembering something else here?

> Am I remembering something else here?

No. That has indeed happened. It was a work by Alyssa Rosenzweig [0].

If I understand correctly, she implemented the userspace Mesa driver. The kernel driver is being developed by Asahi Lina. Together, they are meant to provide "full open source graphics stack for 3D acceleration on Asahi Linux" [1].

[0] https://rosenzweig.io/blog/asahi-gpu-part-5.html

[1] https://rosenzweig.io/blog/asahi-gpu-part-6.html

Edit: make the quote more accurate, adjust formatting

The problem is that fabricating a chip has upfront costs measured in 10s of millions for the masks, and the chips that come out aren't all uniform -- there is more work required to do to capture variations in power consumption, rejecting bad parts, etc in order to bin them to target different price points. It's hard to do at a reasonable cost per chip without huge volume.

> Judging by Asahi, FOSS will be fine.

Or rather: based on the results of other contemporary complex hardware RE efforts, Asahi will never be relevant. Think of _how many GPU hardware_ there have been efforts to RE, and how few have produced any drivers that people would be happy to use on a daily basis. How many people use Nouveau these days? PowerVR SGX (a "GNU high priority project" from a decade ago!)? Lima? Even Panfrost?

In fact, this entire IPU webcam thing is not new. We've had hardware with IPU for at least half a decade and literally _NO_ such device works with Linux, only a couple devices half-work, and very poorly at that.

There are much fewer people working on hardware support on Linux that you may think.

> The real question is where the hell is the open source hardware?

Agreed. I get so angry every time I think about the amount of Verilog and VHDL that will never, ever get released even after the companies responsible for it die and even after many years have elapsed.

Where is the 3Dfx VSA-100 source, for instance? What technical advantage could it give NVIDIA now? They purchased 3Dfx twenty years ago!

They don't necessarily own the rights to all the source. You can't release it if you licensed part of it from some other company and that company is now dead.

Reply on news.ycombinator.com