Ask HN: Best way to prepare a codebase for open-source? | Hacker News Reader

Ask HN: Best way to prepare a codebase for open-source?

60 pointsEikon32 comments

Hi HN,

How would you approach the process to opensource a proprietary codebase, especially regarding things such as ensuring that no secrets are sitting somewhere in the history.

I'd be tempted to publish without history but I feel like a lot of important context will be lost.

View on HN

Keep the proprietary history to yourself, publish just a squashed snapshot of the state you want to make public, use git-replace [1] to link the public history with the (hidden) private one when you want full history, develop only on the public one from now on.

[1] https://git-scm.com/book/en/v2/Git-Tools-Replace

I've never seen git replace before, very interesting! Can you use git-replace to edit an old commit message without rebuilding the whole commit graph and instead by just adding a new commit? The article seems to imply that you can use it to "truncate" history but still be able to browse it if you happen to have the full history, or something. I'll have to look more closely into exactly what's going on.

I think this just changes your own local view of the history. So you can fix the message, but in order for anybody to see it on push, you'd need to reify the changes with git-filter-branch & rebuild.

https://www.oreilly.com/library/view/git-pocket-guide/978144...

https://blog.plover.com/prog/git-replace.html

I haven't tried, but my read was "yes" when I looked at the intro, and "no" once I got an illustration of the data structure. There is something ambiguous in one or the other.

I'm leaning towards "no." The text doesn't make sense, and the data structure does. So I think you still rebase; you can just keep both versions around.

One really important part people forget about is providing a development workflow. How should people reach out to you ? How can they make patches ? Report security flaws ? Which design principles are you following and are you open to change these ? How will this discussion be facilitated ?

Making something available publicly is a far shot from actually having an open source project in my opinion. Good luck in your endeavors, it's a lot of work!

Aside from the code: Know your intention.

What do you want to achieve by open sourcing? Are you trying to grow your username, gain trust or is it just throwing code over the fence so users can go on while you focus elsewhere. Do you hope for contributions or do you want to continue driving the project?

From there you can derive the community management you have to do. The more involvement you want from externals the more you have to invest in community management. (The more you want the quicker and more thorough you have to respond)

Then be aware of all the legal things. When using libraries: anything lgpl or gplnlicensed stuff with something incompatible etc.?

And then for secrets best is indeed squashing the history. Makes it in the beginning a bit annoying, but you might have code comments or commit messages referring to customers or have experiments with libraries of unacceptable license or whatever in there. Limiting review to recent state is a lot simpler.

BFG Repo-Cleaner (https://rtyley.github.io/bfg-repo-cleaner/) scans the history for secrets and removes them.

As well as cleaning up/squashing history; consider:

a) Make sure it's actually all your code - not code you copied from some other closed source project, or a contractor gave you 3 years ago. b) Remove the questionable comments about other employees and their managers c) If you're removing history you are removing some of the rational about why things are like they are - that does make it harder for people in the future to change things

Here's a possible practical idea:

1) run the full contents of HEAD into a space-separated list of tokens considered "okay".

2) dump out the full history into a space-separated list of tokens, filter out everything in the "okay" list, and list what's left.

You might want to set up some sort of incremental regex filter thing to chew through the list efficiently.

But if you implicitly trust HEAD as "incontrovertibly okay" this might filter out a lot of tokens for you.

I'd worry about the licensing too. What open-source license would you publish it under, and do you actually have permission from the owners and everyone who participated in writing it to publish under that license?

On secrets: you should be cycling all your secrets on a regular basis anyway. If you aren't, now might be a good time to start.

In addition to the good advice here, you might want to check for anything potentially embarrassing, such as offensive language in comments, identifiers or commit comments. Some "tech bros" can be remarkably dumb about that stuff. Of course if you developed all this yourself, no problem.

Tangential, but still relevant; Once you decide on the cleanup for open sourcing, choose a path that's comfortable for YOU, the maintainer, and clearly define contributing rules/guidelines. Take breaks, and feel free to let the community know. And remember, it's only a hobby, nothing to get stressed or burnt out by.

99.9% of people will only care about the current version. So don't worry too much about history, people mostly just want stuff that is accessible and works.

Edit for clarification: You can delete the history for the open-source version and publish it with a fresh history. And keep the original history internally in case it is ever needed.

It's not a numbers game. The remaining 0.1% can be very important too.

There are times to kill history, and there are times to bring it along. History does a lot of things.

- Giving people credit.

- If, in 20 years, you want to pull up an old (no-longer-compatible) document

- Public patent prior art

- ... and a swarm of others

"Internally" is likely to no longer exist a decade down the line.

Whenever I've open sourced, I've brought along history. In one case, we did need to rewrite history to remove a few sensitive files, but git makes that easy enough.

If keeping history is a heroic effort, by all means kill it. However, that's not a good generic best pracitce.

I've had managers tell me this when we've had to cut over from SVN to git. Maybe it's true for them... For me out on the pointy end of the spear, it's not hard to bring that history across, you just have to know how to do it, and it's invaluable when you're trying to figure out bugs.

This is breathtakingly bad advice from a security perspective. If secrets might sit around in a repository you don't just publish it and bet on people not looking for them, at least not if data security is of any worth to you.

That is not what i said. I said he should delete history and publish a new version with a new commit "v1"

You literally edited that in after my answer.

Gosh I wish there was a history available so I could understand all this context!

Read it in the context of OP's question - GP is suggesting squashing all history and not worrying about it, as OP alluded to.

Yes, I see how it can be understood that way. GP has edited it anyways to clarify this point.

History is way more important than the current version of the code.

How could history be more important than a running version of a project?

Without history the current version is the only version.

Is there a specific reason or example that supports this?

If it is an old codebase that you are not sure of the origin of all the code, then you need to do a code audit and maybe even consult a lawyer.

Do not publish the history. Fresh start for open source.

Clone it then push it to a clean repo to destroy the history. Seriously. I understand security concerns but the history could expose previous bad practices which can be applied to other products you run. Bad patterns are endemic.

Also it gives you the chance to secret scan it and remove any swearing and embarrassing comments (I haven’t seen a codebase without any yet)

clone and push does not destroy the history

Sorry I meant export and push.

I recently put something on GitHub that we wrote for our product. The repo was on GitLab. You can add a git remote and push and it will keep your git history.

I wrote the library because the issue it solves in MinIO's Python client was marked as "won't fix" and it has been useful for many people (we put it on PyPI before adding it to GitHub), and I was glad a few days ago to see that MinIO added something very similar to their Python client (they added a Python wrapper just like bmc).

https://github.com/ikodotai/bmc

CI tests are nice, unlimited use of GitHub Actions for public repos

To remove secrets you can use git filter-repo to rewrite commits

Another reason to squash into a single commit: not all of your developers want their name & work email address published without their consent.

Reply on news.ycombinator.com