HNHacker News
TopNewBestAskShowJobs

emidoots

7,381 karma · joined August 8, 2014

https://devlog.hexops.com

emi@hexops.com

submissionscomments
emidoots··on Linux tool to show progress for cp, mv, dd
Wow, didn't expect to see @xfennec pop up on hacker news while drinking my coffee this morning! I don't know if he'll see this, to be honest didn't know he was still doing things - but this person basically got me into programming and game development-I really can't believe it.

xfennec (and some friends?) I think built a game engine called Raydium, and one of their games called Mania Drive-a Track Mania clone-got distributed with OpenSuse installation CDs back in the day. When I was just like 12 years old, my dad installed that on the family computer and it was all we had, Mania Drive was one of the coolest games on there. Me and my siblings played that for literally days and months on end, making crazy levels we couldn't beat without knowing every turn. It was a huge part of our childhood.

Their game engine was in C with PHP scripting, I remember posting some levels to their forums and asking, in retrospect super dumb, questions and they were so polite and friendly. I remember us joking at the time that the French seemed like these god-like game developers, it had such a profound impact on us, I even wrote about it last year and linked a video of Mania Drive first[0]. I went on to learn Python and then lower-level languages as a result. I'm not sure I'd be coding today without them, to be honest.

Sorry it's off-topic, just really blown away to see a username like that pop up in my feed. Really goes to show that kindness + some cool open source software can have profound effect on people.

[0] https://devlog.hexops.com/2021/increasing-my-contribution-to...

emidoots··on I write code 100 hours/week, here's why I probably won't stop
Ah wasn't expecting this old thing to pop up again. By far the most contentious article I've written; people ranging from "hey if that's you and you're happy then great" to people feeling like it's a personal attack on them. I'd encourage an open mind.

Anywho-it's closer to 3.5 years now that I've done this, maybe I'll do an update at 4 years.

The 2.5 year initial duration (when I posted that article) was like working two full-time jobs, 100h/week, almost solely writing code.

Past year has been more like 85h/week, my dayjob has required more time (48h -> ~60h week), with much less of that time at work being /writing code/; and ~25h/week going to my gamedev passion projects, of which a good chunk has also gone to helping others with their code/issues.

The -15h/week 'loss' has gone to caring for pets, and a gentle touch more travel/sleep/gardening. Doing well and enjoying my life, still.

emidoots··on a[low:high:max] in Golang – A Rare Slice Trick
I'm a big fan of Go, have been using it since r59 (pre 1.0), professionally working with it the past ~8 years at one of the earliest companies to adopt it.

The fact that you cannot have a generic method at all, and instead have to rewrite methods as functions.. that seems like a pretty glaring flaw.

I'm happy Go got some form of generics, definitely, but they really do feel bolted on to the language.

emidoots··on Purego – A library for calling C functions from Go without Cgo
This project isn't really about calling C functions, Go already supports that after all (CGO). Rust does suffer the same problem, which is that cross-compilation when using C code is a bit of a nightmare.

purego solves this by using dlopen and friends.

My understanding is Rust solves this through libloading (same approach effectively) and more heavy-handed approaches like cross-rs which distribute full C/C++ build toolchains for each target (in Docker images or something?)

emidoots··on Purego – A library for calling C functions from Go without Cgo
I am in love with Zig (as you know), but feel the need to say: please be careful with blanket claims like this. purego is a pretty admirable approach to fixing cgo cross-compilation without Zig (though has its own drawbacks, like no static binary for example.)

Using Zig for Go cross-compilation, although quite great, isn't bulletproof. Finding the right CC/CXX incantation can be fairly tricky, the articles on this are not super up-to-date, and you need a copy of the macOS SDK[0] if you intend to cross-compile for macOS. You may also run into a few scary linker warnings and need to figure out the right Go build flags.

I definitely think Zig and Go can be best-friends, after all they share so many similar qualities and seem quite complementary to each-other. It's definitely possible to get Go<->Zig CGO cross compilation working (I did so for Sourcegraph); But Zig needs a little more love before it'll 'just work' as claimed for the Go use case, so best not to claim otherwise until then.

[0] https://github.com/ziglang/zig/issues/1349

emidoots··on Purego – A library for calling C functions from Go without Cgo
It loads the dynamic library at runtime, instead of linking against it, which means it makes cross-compiling with CGO easier as no target C toolchain is needed.
emidoots··on Purego – A library for calling C functions from Go without Cgo
Ahh that's right, I see.

Maybe I should play with removing the stack switching from purego so that under condition of a locked OS thread you can avoid that overhead :) I might give that a shot sometime

emidoots··on Purego – A library for calling C functions from Go without Cgo
Makes sense! I also wonder (if you know): last I looked I recall that each CGO call requires switching to the system stack, but I can't recall what happens after. Does it switch back to a regular goroutine stack once the syscall has completed?

I wonder if a more tailored CGO implementation could pin a goroutine to a thread which is guaranteed to have a system stack available, so that each CGO call need not worry about that switching at all. Maybe that'd require runtime changes though?

emidoots··on Purego – A library for calling C functions from Go without Cgo
Very cool! I will definitely give this a try, I've been looking to build Go bindings to Mach[0] soon.

It looks like this would make cross-compiling CGO easier (no target C toolchain needed?)

Does this do anything to reduce the overhead of CGO calls / the stack size problem? IIRC the reason CGO overhead exists is at least partly because goroutines only have an ~8k stack to start with, and the C code doesn't know how to expand it-so CGO calls "must" first have the goroutine switched to an OS thread which has an ~8MB stack.

One reason I think Go <-> Zig could be a fantastic pairing is that Zig plans to add a builtin which tells you the maximum stack size of a given function[1], so you could grow the goroutine stack to that size and then call Zig (or, since Zig an compile C code, you could also call C with a tiny shim to report the stack required?) and then eliminate the goroutine -> OS thread switching overhead.

[0] https://github.com/hexops/mach

[1] https://github.com/ziglang/zig/issues/157

emidoots··on The technology behind GitHub’s new code search
I've worked alongside the CEO/CTO of Sourcegraph for the past 8 years, everyone else is at our company offsite so I figured I'd chime in :) nobody asked me to write this (nor did I ask) :)

The article is a top-notch technical write-up, the devs on GitHub code search should be proud of what they've achieved so far!

Honestly, we're rooting for GitHub to improve their code search, viewing them as a close peer-not a competitor. We also maintain OSS projects like Zoekt, which IIRC GitLab is maybe looking at using for their own. The more devs that 'get' code search, the better off Sourcegraph is frankly!

GitHub has a nice intuitive/simple UX, we could learn a thing or two there (though, easier to do with less features.)

Still, Sourcegraph search tech is quite a bit more powerful:

* Searching over commit messages, diffs, filename, etc. are super nice for tracking down regressions / finding 'that PR I swear my coworker made'

* Expressiveness like "find this regexp in repositories, but only if the repo has had a commit in the last month AND has a file named package.json in its root"

* Since Steve Yegge joined us, we've started thinking about ranking of search results, a notoriously difficult thing to do well in code search unless you have great factors to rank on (e.g. a semantic understanding of code): https://about.sourcegraph.com/blog/new-search-ranking

* We stream results back, so you can get a comprehensive set of results - not just a few pages, from our API.

* Works in GitHub Enterprise, not just GitHub.com. Plus on all your code hosts, think BitBucket, GitLab, Azure DevOps, Gerrit, Phabricator, etc. and even non-Git VCS like Perforce.

* Respects permissions of all your code hosts (a very difficult problem, as there are no official APIs to query this info from code hosts in general)

Having code search is one thing, but using it is another:

* Code Insights (we use search as an API to gather statistics about code, track code quality, keywords, etc. both over time and retroactively and let you build dashboards)

* Batch changes (find+replace, but over thousands of repositories. Run a Docker container per repo, run your custom linter script etc. and then draft or send PRs to thousands of repos, manage/track campaigns with thousands of PRs like that over time, etc.)

* Precise code intel / semantic awareness of code, we use SCIP indexers for this (spiritual successor to Microsoft's LSIF format for indexing LSP servers.)

I am super happy GitHub continues to push their code search effort, and genuinely believe it's a great thing for all developers and us over at Sourcegraph. Also excited to see when they do their public rollout of this :)

Anyway, that's just my take as someone who works there-other Sourcegraphers will chime in later if anything I said above feels off to them I'm sure :)

emidoots··on Learning Modern 3D Graphics Programming (2012)
samdauwe's work is fantastic. We're using WebGPU in Mach engine, and also have quite a large set of examples for how to use WebGPU from Zig[0] that are pretty much standalone (only use our engine for windowing/input), may be a bit easier for folks to run (zero-fuss installation, you can cross compile seamlessly to any OS, etc.):

[0] https://machengine.org/gpu/

emidoots··on Quite OK Image is now my favorite asset format
PNG/JPEG are not natively understood by GPUs, though. GPUs have compressed formats they natively understand, and basisu can transcode to them quickly. You'll be burning a lot more GPU texture memory as you'll be asking the GPU store raw image data instead of compressed image data.
emidoots··on An X11 apologist tries Wayland
I recently tried switching to Linux with a few different distro's (Ubuntu, Elementary OS, etc.) and use Wayland.

The thing that sticks out like a sore thumb to me, and which I've been unable to solve, is that apparently it's not possible to configure trackpad scroll speed. At all.

From what I was able to gather, Wayland/libinput say they shouldn't be responsible for handling it and instead window managers should[0][1], meanwhile gnome says wayland/libinput should handle it[2] and ultimately - several years later - it's still not possible to in pretty much any Linux distro that uses Wayland(?)

When I switch to my Linux laptop to test things, my trackpad is bonkers and I have to move my finger in like 1mm increments because otherwise I'd scroll like 10 pages in Firefox. It's infuriatingly frustrating.

[0] https://gitlab.freedesktop.org/libinput/libinput/-/issues/18...

[1] https://gitlab.freedesktop.org/wayland/wayland/-/issues/87

[2] https://gitlab.gnome.org/GNOME/gnome-control-center/-/issues...

emidoots··on VRchat bans mods, embraces EAC
VRChat mods include things like general QOL improvements, but also most accessibility features are mods. Things like preventing flashing colors to avoid seizures.

They want to ban them because they compete with VRC+, their premium service.

emidoots··on Get started with GPU Compute on the web
In case you've ever wanted an opportunity to learn Zig and WebGPU, we have Zig bindings to Dawn (the Chrome implementation) with a fair amount of examples in Zig[0] - also working on getting WebAssembly support going right now.

Can be a bit easier than trying it out in-browser (works on Linux, etc.)

I'm shamelessly posting here because we're looking for folks[1] to contribute by porting more webgpu examples over to Zig :) Can be a fun way to get started with both, though.

[0] https://machengine.org/gpu/

[1] https://github.com/hexops/mach/issues/230

emidoots··on Using Zig as cross-platform C toolchain
"So, a clang wrapper?" is a common thought, so here's how Zig differs from clang out of the box:

* Links MachO binaries for Apple Silicon via the custom zld linker it ships. LLVM cannot do this currently.

* Provides (deduplicated) libc headers for pretty much every platform, including macOS and glibc/musl. https://github.com/ziglang/zig/tree/master/lib/libc/include

* Provides a libc implementation (libSystem for macOS, musl and glibc, mingw for Windows, and WASI)

* Deals with lots of the deep depths of hell, like enabling you to target any version of glibc out of the box by building symbol mappings: https://github.com/ziglang/glibc-abi-tool/

And that doesn't mention the most important part, IMO, which is that it lets you cross compile _out of the box_. No fiddling with sysroots, system packages, etc. to get a cross compiling toolchain working.

emidoots··on Why I rewrote my Rust keyboard firmware in Zig: consistency, mastery, and fun
In Zig, you can provide a different allocator for a single data structure (or even distinct instances of the same data structure). There is no "global" allocator (what Rust lets you swap out.)

This is vastly more powerful, I have used this to tailor an allocator to specific data structures for better performance.

emidoots··on One Guy Ruined Hacktoberfest 2020
Most of them will because Hacktoberfest relies on project maintainers to flag the PR as `invalid` within a certain window and close it, AFAIK.

Though I imagine with all the outrage over this that will definitely be changing.

emidoots··on Show HN: WorkOS – APIs for enterprise features like SSO/SAML
Monday.com is the worst product I have ever had the experience of working with.

Spreadsheet with 200 rows and 20 columns? Your browser will be laggy as all hell thanks to their wonderful UI.

Try to tell support? The entire page jitters up and down as you type every single character in the support box on Chrome.

Reload the page? Well now the page isn't laggy at all, but literally it is not rendering rows that _are definitely there_.

I really wanted to like it -- beautiful user interface -- but it's an absolutely terrible product. And it was ~15 people on my team reporting the same exact behavior as we tried it.

I would consider recommending Monday.com to my worst enemy.

emidoots··on Google Open Source Code Search
You are 100% correct, I really messed up here by suggesting that option. I misread our own docs. It would only disable event counts from being sent (e.g. instead of "how many jump-to-definitions were performed in a day?" we would just send a boolean "did one or more jump-to-definition occur in a day?" based on my reading of the code[1]) -- not what I thought it did. Will send a PR to clarify the docs on this so I don't mess up like this again..

I'm human and screw up, frequently; this instance just happened to be on the ridiculously important topic of privacy -- hopefully you will forgive me for that, I wasn't trying to be malicious but certainly in retrospect I can see this being interpreted as such.. :/

The right option to turn it all off is just this one, since we only send ping data as part of the version update check you disable that and it's all off. And you can confirm this in the code as I just did here[2][3]: https://docs.sourcegraph.com/admin/config/site_config#update... And as I mentioned previously you can always firewall off Sourcegraph 100%.

As an aside, I can promise you that I wouldn't have continued to work at Sourcegraph for the last 5 years if I thought our business was selling or collecting identifiable user data in ANY form. We only collect just enough information to help prioritize what features we improve and (aside from the first admin's email as I noted already above) it is all 100% anonymous and aggregated numbers that we are extremely transparent about[4]. Our person running analytics is also constantly trying to make this more transparent[5] because we all are very security and privacy aware and know the #1 way to convince people to not run software is to make them think you are spying on them or using their data in ways they would not want.

It's obvious to me this should be more clear in our docs, I'm going to forward all of this conversation onto the rest of our team to make sure we improve our docs here.

[1] https://sourcegraph.com/search?q=repo:%5Egithub%5C.com/sourc...

[2] https://sourcegraph.com/github.com/sourcegraph/sourcegraph@f...

[3] https://sourcegraph.com/github.com/sourcegraph/sourcegraph@f...

[4] https://docs.sourcegraph.com/admin/pings

[5] https://github.com/sourcegraph/sourcegraph/pull/8930#issueco...

emidoots··on Apple's products are getting harder to use, ignore principles of design (2015)
The reason I put air quotes around "designer" was to convey the fact that I believe _good_ designers would base their decisions on logic discovered via some form of the scientific method. Studying what makes user interfaces understandable, and why.

Data doesn't need to be "track everything the user does and do A/B testing", it can come from many sources: user behavior studies done in the past, cognitive psychology, experience gained from mentors, etc.

Would you make the same "they do it every day so they must be right" argument for engineers? Product managers? Stock brokers before data-driven trading? Doctors before the rise of modern medicine?

emidoots··on Apple's products are getting harder to use, ignore principles of design (2015)
I have had conversations with several "designers", some from FAANG, and all have held the general belief that having strong authority over how the UI looks and getting eye-candy at the cost of usability and ergonomics was 'without doubt the right choice'.

In my frank opinion, many designers hold a position of subjective judgement and thrive on the power that comes with that. This is like a developer making arbitrary decisions about a product because they like the technical implementation better.

Data, and actual user preference, should always be the key decider. Not some arbitrary rule-decider.

emidoots··on Ghidra, NSA's reverse-engineering tool
Why this is important (for those uninitiated):

- Ghidra is basically the first real competitor to IDA Pro, the extremely expensive and often pirated state-of-the-art software for reverse engineering. Nothing else has come close to IDA Pro.

- Ghidra is open-source, IDA Pro is not.

- Ghidra has a lot of really cool features that IDA Pro doesn't, such as decompiling binaries to pseudo-C code.

- It's also collaborative, which is interesting because multiple people can reverse engineer the same binary at the same time -- something IDA only got VERY recently.

emidoots··on Rethinking Classical Concurrency Patterns
Sourcegraph dev here :) At Sourcegraph we've actually tried a lot of different syntax highlighters in the past. Syntax highlighting is actually a _really_ hard problem.

Syntax highlighting for basically all editors today is provided by TextMate language grammers[1] (`.tmLanguage` files) ultimately, which are just giant regex definition files, basically. These are the ubiquitous files for syntax highlighting, and you can almost always find a TextMate grammer for any programming language.

Some 'modern' syntax highlighters, mostly ones designed for use in the web, try not to use these TextMate files -- but in doing so they lose support for TONS of languages, and the languages that they do support are often very buggy / incorrect because the grammers are just simply not that well tested (nobody is testing them in their editor daily).

The annoyance with TextMate grammers is that they are based on Oniguruma Regular Expressions, which means that while they are written in very powerful regex syntax, you cannot really use them with any regex engine built in to most languages today. Almost all editors directly use the original C implementaton of Oniguruma, even editors like Visual Studio Code which are based on web tech primarily.

And even once you have a working Oniguruma regex engine, it is still very non-trivial to use it for highlighting actual code because the TextMate files themselves are pretty complex on their own.

To make matters more interesting, Sublime created their own language grammer format (`.sublime-syntax` files), which is a superset of the TextMate language grammers and supports more features. Sublime can convert any TextMate language grammer into the new .sublime-syntax format.

So when I started seriously looking into syntax highlighting about a year ago, the Pygments syntax highlighter was my first choice and it is pretty good, but I found it to be lacking support for languages we cared about such as TypeScript at the time. Additionally, Pygments was pretty slow when we tried it out. GitHub switched away from it for a similar reason[2]

This was about the time that I ran into the Syntect[3] project. For us, this had huge benefits:

- It was by far the fastest syntax highlighter we found, which meant page load times improved a lot for us.

- Its results were really, really good. I found multiple cases where the output was better than other highlighters such as Pygments.

- The built-in language support, all language Sublime Text 3 supports by default, was great out of the box.

- Because Sublime can convert TextMate grammers to .sublime-text grammers, we ultimately get support for basically all languages.

- It took me _one day_ to write `syntect_server`[4] which highlights arbitrary code for our Go application, which was a spectacular turnaround time given how much time I'd already invested elsewhere in researching all of this.

Overall, we really couldn't be more pleased with Syntect and Rust for modern syntax highlighting. It's a beautiful, fast, high-quality combination.

[1] https://macromates.com/manual/en/language_grammars

[2] https://github.com/github/linguist/issues/1717#issuecomment-...

[3] https://github.com/trishume/syntect

[4] https://github.com/sourcegraph/syntect_server

← PreviousPage 2 of 2