He is a minor investor. He commented on the acquisition at https://x.com/mitchellh/status/2118012675582280038
765 karma · joined February 1, 2021
If you mean "how would I implement it without ptrace", there are a few options but likely the Endpoint Security API [1] would be the way to go. But it's not too great because it needs full disk access and root, neither of which I would be particularly happy accepting :(
[1] https://developer.apple.com/documentation/endpointsecurity
You can also use --compiler-traces to understand the inside of the linking process as well; these annotations are provided by lld.
AFAIK, the answer to this is no but only if LTO is doing a good job and actually optimizing well across object files.
> B) If not, why isn't the optimizer smart enough to split them up internally and optimize in parallel?
I think there was another comment by someone more knowledgable in Zig that this is something which is being discussed by the Zig compiler team.
I actually collected a build with multi-threading turned on [1]. It helped quite a bit, cutting the Zig object time from 7m49s to 4m05s. But because of the reason you mentioned (and the post was already so long!) I decided not to bring it up.
> Oh - one last thing to note, Zig has very fast debug compilation and incremental compilation, it's a real nice thing to work with that you simply can't approach in most other languages right now.
Yes, in fact I also tested just running bun run build / bun run build:release and captured traces for both! The clean release build took 3m36s for Zig versus 5m50s for Rust [2][3]. Clean debug was closer: 3m07s versus 3m27s [4][5].
I also tried incremental debug builds: adding a comment to output.zig / output.rs took 48.6s / 65.4s to rebuild. Ordinary developer builds certainly gave a different picture from CI.
[1] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
[2] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
[3] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
[4] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
[5] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
I added a Python script wrapping rustc which serializes all rustc invocations for bun_* crates; you can find the recoridng at [1]. Turns out it only increased the build by 1 minute! So clearly my intuitition that it was just crate parallelism was not correct.
One more confounding factor though is that rustc is parallelised while Zig's compiler is not (yet). So it's still possible that might be the reason.
[1] https://buildprof.lalitm.com/#!/?url=https%3A%2F%2Fblogexamp...
The fix seemed to be to split up Bun's Zig module into ~100 pieces just like they've done on Rust but how representative that would have been on anything Bun might of shipped is very much in question!
But I still think the visual view is invaluable for human understanding and for the "why is this even happening" problems which I think AI is still bad at seeing.
Provides great context on this accomplishment, what it means but also doesn't mean.
Having built a lot of protobuf tooling, I'd estimate that protobuf largely falls into the former camp; most of your time working with .proto files is operating on fields which terminate using `;` at the end of the line (though multi-line is also possible).
[1] source: I've done it for SQLite SQL at https://github.com/LalitMaganti/syntaqlite/
[2] e.g. SQL naturally has this at statement and expression boundaries which covers almost all of the cases people care about.
But then what stops the upstream tools from doing exactly the same thing and getting the same speedups? I've certainly seen AI become a huge boon in my debugging experience for random user reports for example.
FWIW if we are talking about hyper personalized software like the OP, it's going to quickly go beyond "debug and maintain" and towards bigger issues like UX, ergonomics, features etc. And at that point, if the upstream is also being developed you will have differences between your personal visions and upstream
Is the answer at that point "accept whatever AI does"? That to me seems to clash with the entire premise of hyper-personalization which is where your vision is what matters not the AI's. Do you really want to get into the game of having opinions on design of every tool you use forever?
Engineers using devtools are not so different to an average user in that they just want things to work. Maintaining a devtool is real work; e.g. suppose upstream adds some feature you want but it clashes with something you did downstream. Not in the merge conflict sense but in the UX sense. Do you want to resolve that on every release? Is it just "AI will fix this"? Maybe one day, but as of today agents can do something along the right lines but don't capture my UX sense very well; and if the whole point is hyper-personalization, I want it exactly how I want it.
"Does it seem to work?" is a fine right until it fails when you're doing something important. Do you then stop and go prompt an agent to fix your tooling, and hope it does a good job this time? What if it doesn't do a good job?
Moreover, for devtools which are fundamentally social (i.e. lots of different people looking at the same thing), there's real value in that thing looking the same for everyone. Having a baseline for teaching, auditing, verifying "we are all talking about the same thing" is incredibly valuable and is sometimes where the most value is.
All in all, maybe in some categories of devtools, this hyper personalization is indeed what will happen but it's a long way from being universal.
Excellent talk which explains the concepts really clearly and concisely.
My biggest problem comes from:
> Every constraint on the table. Tighten those enough and something interesting happens, you end up with only one possible solution. And that solution is, somewhat ironically, the perfect one.
Constraints are not rigid and unchangable [1] but flexible things which can bend to solve the problem at hand. Constraints can be: deadlines you impose, frameworks you are working within, the shape of the output schema, performance characteristics etc. And because they trade off against each other, tightening them leaves you with several solutions, each fitting a different trade-off.
Constraints also change over time as you and your team's collective understanding of the problem, the environment or priority of the problem change. This needs to be considered when building the initial system too: baking in some flexibility means giving up some of today's "perfection" to deal with future variance.
Understanding what constraints matter, which can be shaped and trading them off against each other is, to a very large degree, the job of software engineering. So no, I don't think you enumerate the constraints and find the only solution which fits; this is a very simplistic take on the complexities of building software.
[1] unless you work in a dictatorial company in which case my condolences!
1. I'm not claiming that every interaction with a user needs to be turned into a full discussion. There are lots of cases where the answer is "this is what you're looking for, here's a link to the docs". In my experience, this has gone down a lot over time because people tend now to rely on AI much more heavily for those sort of questions in the first place so they won't even reach you. But it does happen and in that case, the best thing for both sides is to point to the documentation and move on.
2. If I am going to employ this strategy, I always try and both give the direct answer to the question and, as a separate point, ask them the context. Quoting directly from the post:
> well the answer to your immediate question is X but that’s a pretty strange thing to ask for because of reason Y. Can you tell me more about the wider problem you’re trying to solve?
3. This does necessitate that you are in some sense an "knowledgeable" on the problem space someone comes to you with: I would never employ this strategy in an area I didn't already feel I was quite well equipped to give my 2c on.
4. If the person I'm speaking to I know for a fact is already knowledgeable on an area, I would be very hesitant to use this approach because I try and be charitable with assuming that people generally know what they are talking about. While yes, I'm not going to be perfect every time in judging this, I think you can get a pretty good sense from the way they ask their question and how clearly they respond to your initial answer whether they are not (ties to 3 as it requires you are good enough in the space to judge this).
5. If someone pushes back, I will always defer to their read of the situation, I'm not here to make people do what I suggest, I will simply say "well I think you should do/not do A because of B reason but at the end of the day your call" and leave it at that. I think it's a good thing to be able to have respectful discussions on topics even if both sides agree to disagree.
[1] https://lalitm.com/post/exponential-search/ [2] https://en.wikipedia.org/wiki/Exponential_search
Essentially ~all of the tests were found to be useful but in a more "smoke test" capacity i.e. they provided good "basic" coverage but it was clear that it was also not sufficient.
Which is why in the rewrite: 1) I built a TCL driver that run the upstream SQLite tests and verified we accepted or rejected the SQL in the same way as SQLite.
2) I wrote a test runner which checked for "idempotence" i.e. run the formatter over all the SQL from all the other types of tests then verify that the AST was identical in the input and output.
3) I also wrote a script which ran the formatter over the PerfettoSQL standard library [1], a real world SQLite-based codebase that I knew and deeply understood so I could go through each file and manually check the output.
> Examples of that would also be nice (I don't doubt the personal feel that waste was justified)
Some things learned concretely:
1) C was not going to work for the higher level parts of the project, even the formatter was not pleasant to read or write in C, the validator was much worse
2) Doing the SQLite source extraction in the same language meant that I could ship a really cool feature where the syntaqlite CLI could "generate dialect extensions" without people needing to download a separate script, run their own extraction on the SQLite source code, or worse yet, need to fork syntaqlite. This actually makes it technically possible for people in the web playground to dynamically build extensions to SQLite (though I haven't ended up plumbing that feature through yet)
3) Having a DSL [2] for extensions of SQLite (that e.g. PerfettoSQL could use) was the correct way to go rather than using YAML/JSON/XML etc becaue of how much clarity it provided and how AI took a lot of the annoyance of maintaining a DSL away.
4) I need to invest much more in testing from the start and also more testing where the correctness can be "proved" in some way (e.g. idempotence testing or SQLite upstream testing as described above)
[1] https://github.com/google/perfetto/tree/main/src/trace_proce... [2] https://docs.syntaqlite.com/v0.2.15/guides/custom-dialects/
I agree with you in theory but in my opinion, it doesn't work so well when you don't even know what exactly you are looking for at the start. Yes I knew I wanted a formatter, linter, parse but which language should those be written in, should they be one project or many, how the pieces should fit together, none of that was clear to me.
As I pointed out in the article, in these sort of "greenfield projects" I work a lot better with concrete prototypes and code in front of me I can dissect instead of trying to endlessly play with designs in my head.
> It's also nuts to me that he had to go back in later to build in tests and validation.
I think this is a little misleading. Yes I did do some testing retroactively (i.e. the upstream validation testing) but I was using TDD + verifying outputs immediately, even during the vibe coding phase. The problem as I point out is that this is not enough. Even when I had unit tests written at the same time as they code, they had lots of holes and over time, I kept hitting SQL statements which failed which the testing did not cover.
In my opinion, "giving me a better understanding for the architecture of the project" is reasonable technical compensation.
> Indeed, feeling is the only saving grace for a mountain of random unreviewed tests
I think I say a line or two above that this was after a review of the codebase so I did review these tests.
sqlitebrowser.org is cool but it's not the sort of developer tools I'm talking about. As I clarify in the side notes, I'm looking for a formatter, linter, LSP, not an IDE.
> https://sqlite.org/syntax.html
As I replied to some other comment, I'm very aware that there is a syntax diagram but that really only tells half the story. If you actually look at those diagrams into detail, or you look into the the actual parse.y grammar (https://sqlite.org/src/file?name=src/parse.y&ci=trunk), you'll find that they're missing a lot of information which is required for you to actually interpret the SQL into an AST.
When I say "formal specification", I'm not just talking about the formal grammar rules but also how those interpreted in practice. Something closer to the ECMAScript specification (https://ecma-international.org/publications-and-standards/st...).
The problem comes from how SQLite's upstream parse.y works. Becuase it doesn't actually generate the parse tree, instead generating the bytecode directly, the intepretation of any node labelled "id" or "nm" is buried inside the source code behind many layers of functions. You can see for yourself by looking at SQLite's parse.y [2]
[1] https://github.com/LalitMaganti/syntaqlite/tree/main/syntaql... [2] https://sqlite.org/src/file?name=src/parse.y&ci=trunk
To be clear when I say "formal specification", I'm not just talking about the formal grammar rules but also how those interpreted in practice. Something closer to the ECMAScript specification (https://ecma-international.org/publications-and-standards/st...).
[1] https://github.com/LalitMaganti/syntaqlite/blob/93638c68f9a0...
I'm sure GnuCash would also work just fine but ultimately it's also a full double entry system. I never tried it because I came across ledger/hledger/beancount first and, being command line tools, they appealed more to my sensibilities.