Hacker News Reader

Immutability of information in lab notebooks is also important for auditability. Pen in a bound paper notebook works well for this. Missing pages or erasures are a huge red flag for inspectors and rightly so.

Wouldn’t git with signed commits work just as well? You’d have to find a good app that saved stuff as text (not Word, but probably not LaTeX either since that takes time to get the formatting right), but once you figured that out, you’d have a full history of all changes made.

Yup - that's probably a good equivalent.

Signed commits still let you falsify your own work after the fact. You need an external trusted service, a digital notary essentially, that would provide a signed record that you submitted a commit on a certain date with certain set of ancestor commits. Each time you added an entry, you would submit the hashes of all the entries in the book to the digital notary and get a signed record that you submitted those commits on that date. Inability to produce this record would be similar to having erasures or pages torn out in a lab book.

a public blockchain would do for the digital notary? or hash posted on twitter?

You can do this in a pretty cheap and low-tech way by publishing the hash of your current state somewhere that will keep a permanent record. E.g. the Bitcoin blockchain or a classified ad in a local newspaper.

This. True lab notebooks are not just personal notes, they are a legal record. Not every page of every lab notebook will matter, but it absolutely happens and is one of the many real use cases for keeping a lab notebook. If you invent something or discover something of significance, you will want this. Paper notebooks have pages stitched together and ink that can be run through chemical analysis to see if all the text on the page was written in a single session by the same pen. Forensic examiners have about 100 years of experience assessing paper documents. Digital systems can provide legal grade archival support but a sweet front end editor with copy paste and markdown support does not imply anything about traceability of the contents.

A VCS like Fossil at least makes the parent commits un-fungible, so you would only need to sign the latest commit, and not every commit every single time. Git is amazing in it's flexibility, but it's not necessarily always the right VCS solution.

Otherwise I agree with your answer about Git.

Yep! This is already done and for exactly this purpose; I believe the common term is "trusted timestamping" [1].

[1] https://en.wikipedia.org/wiki/Trusted_timestamping

That seems difficult unless it is completely automatic. Also, you will have to convince auditors that your implementation works and that your users are using it correctly and that everything has been committed correctly and in the correct order. So while it might be obvious to you that this works "just as well" it is far from obvious to others.

You wouldn't need to sign the commits, you just need to give the latest hash to a trusted (by the authorities) third party, and that will validate the existence of all history up to that point at that time, so ideally at the end of each day, you give your SHA-1 HEAD to the authority.

This is basically what the block chain does, except that instead of having somebody designated as special and trusted, they have the signers perform proof of work.

I remember that i read about a lawsuite were someone tweeted the hash of a document and proofed that way that this documenta existed at that point. I believe it was about a patent infringment and they even brought a twitter employee in as a witness to testify aboht the imutability of tweeta and the timestamp.

Cant find the article (hash is not a good search phrase when it comes to Findung an article about twitter).

This is the case in some fields, particularly life sciences where it's very difficult to reproduce results, but in others it's not important at all. I have colleagues who have to have their labbook pages signed by the group leader each day, and others that can even take their labbooks with them when they leave their jobs because no one minds/cares.

Indeed, and different fields operate under different regulatory / legal environments. When my parents were working as research scientists, they followed a rigid set of guidelines that were considered to be the gold standard at the time for use of notebooks in patent disputes.

Labs in regulated environments such as medical or pharma research have their own rules today, and an industry of vendors who sell enterprise level software for managing data in a compliant fashion.

I was working at a company on the day that US patent law was reconciled with the rest of the world, and the rules for priority changed. That was the last day we were required to keep a lab notebook.

Today, I mainly keep records so I can figure out what I did later on, and the best single tool that works for me is Jupyter.

Interesting, what about the fact people can just not write something in it?

Over the past few years I've developed a WordPress plugin that provides most (all, I think) of what you want [1]. I also a while back did a comparison of some other softwares' pros and cons [2].

At my work we've used my system for about 2 years now and there is much more engagement with the system than the one it replaced (a custom Lotus Notes application... go figure). People like that they can open the logbook from the lab, write some stuff, save it and edit/publish it later. We also heavily encourage to document anything and everything and use the categories and tags as much as possible to make searching easier.

[1] https://github.com/Academic-Labbook/alp

[2] https://alp.attackllama.com/2019/02/05/labbook-software-shoo...

If you don't mind relying on a third-party, Notion [1] has pretty much all of your requirements.

Formatting is a pleasure (including formulas and LaTeX), tables are decent, you can easily share notes, etc. Code formatting can improve a lot (only basic language semantics/operators/strings); search is decent. Above all: it's incredibly simple to use, and very customizable.

My three big missing features: 1) full API, so you can develop extensions that provide the missing functionality, 2) end-to-end encryption, 3) 2FA.

For now I'm using only for study lab notes, so encryption is not a big deal. But I wouldn't feel comfortable adding anything confidential without a more decent security model.

[1] https://www.notion.so/

but it's a proprietary format and not stored locally, unlike obsidian

Re: not storing locally, yes, hence my first comment on relying on a 3rd party. It depends on what is your intended use, given Notion doesn't have e2e encryption yet.

Re: proprietary, it exports natively in Markdown (including a zip with all images), or pdf, or html. It's not perfect (there's a bug when exporting some LaTeX formulas in .md), but for the most part it works well.

When I was in the sciences, I used a LaTeX document as my labbook

* Ability to share with colleagues - Easy to send pdfs or the raw LaTeX to others.

* Able to simply past in images - I had a folder with dated images, and a macro to insert a graph at the current location (better than floats for labbooks).

* Versioning - Each section had a date in it. Sequential versions were stored in git

* Tables - `\begin{tabular}`

* Code snippets - `\begin{lstlisting}`

* Links, tags, backlinks - `\ref{section_name}`

It could do pretty much anything. Though, you may have noticed that I omitted addressing your first bulletpoint of "simple formatting", as I am not able to keep a straight face while calling LaTeX "simple".

> It could do pretty much anything. Though, you may have noticed that I omitted addressing your first bulletpoint of "simple formatting", as I am not able to keep a straight face while calling LaTeX "simple".

One possible way of harnessing the power of LaTeX while keeping a simple formatting would be to use use `pandoc` and pandoc-flavored markdown. That keeps the formatting simple, while also making it easy to include references, equations, etc. and easy output to LaTeX/PDF/etc.

Have you ever considered Kyso (https://kyso.io) (disclaimer - I'm a co-founder)? I could be way off, and it would be a corner case for us right now, but it seems we'd fit all your requirements:

* We sit on top of git, which remains the single source of truth for your code - versioning, reproducibility, etc.

* Python & R notebooks or simple markdown (also on the web app itself).

* Show or hide code.

* Simple links for sharing reports.

* Tagging.

* Comprehensive search - by tag, keyword and content (soon).

Would be really interesting to hear your thoughts on the fit.

Maybe I'm missing something, but doesn't org-mode (on top of git or another VCS) satisfy all these requirements?

Markdown Enhanced for VSCode[1] + Git for versioning. \

[1]: https://shd101wyy.github.io/markdown-preview-enhanced/

My university has supplied labs with labarchives (and has guaranteed if it ever migrates away from labarchives it will provide an equivalent solution and handle migration). It ticks all of those boxes. My only gripes are minor. Things like no embedding of vector graphics and no good embedded spreadsheet solution. They have some MS office browser based solution that handles version control. That is the official best spreadsheet solution right now.

I generated an html spreadsheet for run logs because I wanted an embedded setup. I ran into the limitation of 64K characters per entry. It’s handled by trimming html off the end then autofixing broken stuff. It works so well that it took me a while to figure out why the number of rows was decreasing as more rows were filled out.

It’s all very tidy once enough people are on board. Whole notebook (and subsections) can be exported to html or PDF. There is even a way to have some robotic file uploading a la directory monitoring.

I think this is a problem not just for the lab community but for digital tools in general. There really isn't a good single app for doing many different styles of thinking and representation (text, drawings, formulas, tables, photos), organizing those things, and sharing them. Many apps get part of that problem right, but not all of them.

You can do all of this in rmarkdown and some grep. git integrates well with rstudio too. plus an rmarkdown file or notebook is a plaintext file so it's future proof. It's what I use and I love it.

What specific version of one note are you using? The search in OneNote included with office is noticeably better than the OneNote for Windows 10 that is preinstalled. There is an addin, NoteHighlight2016 that adds code syntax highlighting. OCR for text within images is another helpful feature that is missing from many alternatives.

I'd like a speech interface, as I often have my hands full with actual lab stuff.

Why are so many people interested in markdown for notes? I've never used it before but HN seems really interested in note taking and in using markdown for note formatting. What does it offer that OneNote or Word doesnt?

I think the difference is probably that markdown is a pretty transparent and versatile way to format things, while still being simpler than many other markup/formatting languages.

With Word, you might insert an image or try to add bullets to something and end up with your whole document looking completely different, whereas, with markdown, you know that if you put in a ">" for a quote, all that's going to change is that one spot.

I think it’s formatting as code vs WYSIWYG.

Good point. Looking back at my comment, that's basically what I meant. I just used a lot of extra words to say it, I guess...

I believe eLabFTW (https://github.com/elabftw/elabftw) would fit ALL your requirements.

Wordpress? Don't know if Blogspot or other blogging platforms offer it, but I'd say Wordpress covers most/all of them.

Plus, easy to import/export data.