Crev – Socially scalable Code REView and recommendation system
github.com
github.com
This is even better: I only want to have a version that has a minimum level of a verified "reputation score". I could perhaps even treat it similar to a Blockchain new block validation: I want to wait until at least X validations from people with a reputation score of at least Y vouch for the package.
I think Anaconda is pretty much exactly this idea, so it's not entirely novel concept. If you squint a bit, you could also see traditional linux distros being this sort of thing for the C ecosystem and then gotten extended to everything. Big question is how could we encourage more of those? Because package maintenance I imagine is pretty unglamorous job, getting volunteers could be tricky. But maybe these problems with npm etc are becoming so aggravated that enterprises could be willing to chip in?
A traditional distro is a central authority, with fixed policies, and they don't have to agree with you which packages are worth including and what level of scrutiny they get. If you want something that's not a part of the distro, you're on your own. With the crev system you can organize or sponsor reviews of any packages you want. There can be multiple sets of approved packages, and you can decide yourself whether e.g. packages approved by an NSA employee are checked for good cryptography, or an evil govt backdoor.
Re getting volunteers for packaging: Debian exists, so it must be possible. In fact, not only Debian exists, but Debian has >1000 members. It's a matter of organization.
* Bugs → Bad
* Less bugs → Good
* Less bugs → Code reviewed
* Less bugs → Valuable
* Code review → Valuable
* GPG → Digital signatures
* GPG → Digital identity with database (key servers)
* Signed code review → Proof code was reviewed by an entity
* Signed code reviews → Valuable
* Signed code reviews → Identity verifiable, proof verifiable
* Signed code reviews → Can calculate reputation across GitHub or similar
* Signed code reviews which can have their ownership transfered via a new proof from the original author can "sell" these reviews to people. Yes, I'm talking NFTs.
From here, people can make a living making and selling code reviews / claims. "Proof of able to do useful work"
The activity in itself is not valuable, and code does not become better just because somebody looks at it. Identifying and subsequently fixing flaws -- that's what makes a code review worthwhile.
Your concept more closely fits the idea of "certifying" code and in that case, as the review sponsor, you will interested in being able to hold someone accountable. And why would you want that liability to be transferable?
Obviously the solution is a meta-reputation system with staking (ie. reverse bounties and when it turns out someone missed a bug they lose their stake, etc), and ... and a web of trust too! Oh and I want a pet dragon too!
That's a really good insight into the sort of implementation that's needed. I'm wondering how we would design the incentive structure, and maybe gamify things, to encourage positive participation.
For example, would we need people whose job it is to deliberately inject underhanded code to try catching out / testing the reviewers? These pen-testers would need a way to pre-commit to the fact that they were trying this (so an attacker couldn't turn around and say "just testing!" when they got caught), and a way to guarantee the underhanded code doesn't reach production even if the reviewers don't catch it.
I still think the distributed reputation system is the hardest part (and would be one of the most useful things to happen to the web since TLS), but there definitely seems to be other hard problems ahead after that.
cargo-crev is very barebones, a bit hard to use, but the concept seems solid.
https://web.crev.dev/rust-reviews/
> For example, would we need people whose job it is to deliberately inject underhanded code to try catching out / testing the reviewers?
That seems ethically iffy. (Reminds me of the university study (from last year?) when they tried to push bugs into the Linux kernel on purpose to test this.)
I guess life creates enough natural experiments for us anyway, so there's enough variance that allows for fishing for a signal. But of course with consent of the involved, sure, why not. It'd be a nice high-high-high level addition to the Simian Army.
> doesn't reach production even if the reviewers don't catch it
I think that's the hard part. If they commit to introducing a vulnerability, it has to be a vulnerability for the test to make sense, and ... at that point there's going to be a time window where it's present in the codebase.
Of course it might make sense for projects that have a known release cadence and/or some kind of security policy which allows for notifying the maintainers and potentially all downstream affected, etc.
> I still think the distributed reputation system is the hardest part (and would be one of the most useful things to happen to the web since TLS), but there definitely seems to be other hard problems ahead after that.
Agreed :)