bertr4nd

I’ve never really thought of my procrastination as coming from perfectionism as applied to the work I’m producing, but on reflection, I realized it often comes from self-disappointment (which is perhaps a form of perfectionism). Eg, I procrastinated on writing my research papers in grad school, but it was because I was disappointed in my ideas (or lack thereof), not because I feared not being able to make the current work sufficiently perfect.

Shortly after joining the PyTorch compiler team, I was part of a team that decided that we should build our own tensor-expression compiler for PyTorch (called NNC, although it wasn’t well-publicized) instead of using an existing one like Halide or TVM.

We ended up sinking two years into it, and never ended up with a particularly good compiler (although we did absolutely crush a couple toy benchmarks).

Arguably both sides of that tradeoff were wrong, though, as the eventually successful PyTorch 2.0 compiler (TorchInductor) was based on Triton (plus some custom higher-level scheduling logic).

By “fully fused” do you mean no function call boundaries? (“Fused” is such an overloaded term)

Are you me? This is precisely what my life feels like. Honestly, one kid was actually manageable; two sort of hit the breaking point (it didn’t help that my work went to hell at the same time), but then we had twins and life now feels basically untenable with both of us working.

I love my kids profoundly, and I think in the long run I will be happier than if I’d not had them and focused on work instead (work tends not to care about you after you leave. Maybe if you’re Steve Jobs or something). But in the moment, life is pretty stressful.

Honestly I’ve been longing for a linear workday. Before kids, and before the pandemic, I used to work 9-6 and I had the most fantastic work life balance. Distance running, rock climbing, etc.

Now I have “flexible” work hours, and it’s all cramming in whatever I can late at night so that I can accommodate the schedule of kids’ daycare. If we didn’t have flex hours, one of my wife or I would have no choice but to quit to manage the kids, and we’d have less income but probably be happier. But since we have the choice it’s all too tempting to keep burning the candle at both ends and the middle.

So I’m one of those people who doesn’t know how to fix anything; I wish I did! I’ll tell you what holds me back: I didn’t grow up around anyone particularly handy, so I don’t know what I don’t know.

Simple example: I was trying to hang some shades in my kid’s room. First set went in fine, second set, my drill hit something too hard for it (brick? Metal?). At this point I’m kind of stuck. I can search Google for this scenario but it’s hard to know conclusively which situation I’m in, and I don’t know how dangerous or destructive what I’m doing could be.

This is probably laughable to lots of you who know, intuitively from experience, what’s happening and what to do, but for someone who had no idea like me, it’s pretty intimidating.

(Coda: we bought stick-on shades and they were trivially easy to install and worked great)

For what it’s worth, I’m not saying that confidence (even hubris!) is bad! Without his confidence, said senior engineer probably wouldn’t have built his successful project.

If I had any literary skill, I’d write a tragedy in which the hero’s flaw is his lack of hubris. (Perhaps it would be autobiographical - my grad school advisor said my weakness is that I’m not arrogant enough.)

I was struck by this bit in the passage about hubris: “The STEM student is taught that hubris is a useful vocational skill.” I recently asked a successful senior engineer how he was able to start an influential project, and the answer came down to a combination of hubris (he had to have confidence that his solution, starting from scratch against a well-funded team, would win out) and appetite for risk.

Re: maps. Eh. Some people tend to romanticize the pre-GPS era with stories of serendipitous adventures, but that doesn’t match my experience at all. My recollection is that wandering around lost was time-wasting, frustrating, and sometimes downright scary. I’m not sure I can recall a single positive unexpected experience that resulted. I don’t miss it one bit. If the price is that I sometimes ask for directions to a place I’ve been before, I’ll take the trade.

> I assert most people will enjoy a baked potato ... about as much as garlic bread

See I'd assert the opposite, or at least the following more nuanced version, for myself:

If I make a baked potato, I'll eat it and be quite happy -- even delighted! -- with it. But! If I have to choose between a baked potato or garlic bread (say, on a restaurant menu) I'd choose the garlic bread virtually every time, unless I'm exerting Herculean levels of willpower to choose healthier options. And if I'm served a baked potato and garlic bread, I'd probably eat the potato, and then somehow find room to eat the garlic bread anyways.

I don't disagree that healthy food can be very tasty, but I mitigate my own struggles with healthy eating by "merely" never keeping unhealthy options around, so I can focus more on the joy of the healthy food, and not have to constantly choose to avoid the high-calorie treats.

Totally hear you. We just added kids 3 & 4 in the fall (twins! Yay!) and so far I’m on paternity leave and just managing a bit of a workout (push-ups, squats, etc) but I have this feeling it’s going to be really hard to balance almost anything against full time employment. I may just end up taking a 30 minute bite out of my work day and seeing how my performance reviews fair.

Seems appropriate to quote Bertrand Russell here: “Envy consists in seeing things never in themselves, but only in their relations. If you desire glory, you may envy Napoleon, but Napoleon envied Caesar, Caesar envied Alexander, and Alexander, I daresay, envied Hercules, who never existed.”

Ah, I overlooked this explanation at first: https://github.com/rui314/mold/blob/main/docs/design.md

Thanks for the great write up as well as mold itself!

Could you comment on which data structures are most critical to mold’s performance, and what makes them so fast?

I also found this disappointing. There’s supposedly a 100x speed up to be had going from something in pandas to something using plain python lists but I have no real idea what it is or why it might have produced a speed up. I can guess, but what’s the point of writing an article that just makes me guess at the existence of some hypothetical slow code?

But what’s the back pressure to this advice? If I need to multiply matrices should I write my own to avoid including MKL BLAS? What’s the heuristic that determines when a dependency is worthwhile?

I’ve always considered taking a lecturer position as a sort of second career if I ever want something different from Big Tech. It’s kind of a shame the health insurance offering is mediocre, which probably rules this out, at Berkeley at least.

My pet peeve is `XXX:`, which I see with distressing regularity in the codebase I work in. It’s non-specific, but vaguely ominous. Is it a TODO? A warning? A bug? An incantation to ward off evil spirits? Tell me!

I love Daniel’s vectorized string processing posts. There’s always some clever trickery that’s hard for a guy like me (who mostly uses vector extensions for ML kernels) to get quickly.

I found myself wondering if one could create a domain-specific language for specifying string processing tasks, and then automate some of the tricks with a compiler (possibly with human-specified optimization annotations). Halide did this sort of thing for image processing (and ML via TVM to some extent) and it was a pretty significant success.

I was thinking this would be cool as a tree map (https://en.m.wikipedia.org/wiki/Treemapping) since that allows for nicely visualizing hierarchical categories.

Thanks!

Are there libraries/tools that people use to do Remez/Chebyshev/etc. function expansions? I can do a basic Taylor series expansion by hand but I’m out of my depth with more sophisticated techniques.

> how much better it is when you have actual control over which parts of your program are JITed

Can you elaborate? What’s the advantage of controlling which parts are jitted?

Possibly the concern with sharing answers is that students could memorize the form of likely exam questions without deeply understanding the material? But that risk exists simply by sharing the questions so I’m not sure I find that argument persuasive.

I’ve always been rather disappointed by the lack of solutions in math books. I mean, on the one hand, it’s a great feeling to crack a difficult problem unaided, and there’s something to be said for building the intellectual discipline one needs to attack a problem with no easy answer.

But also, there are some problems that I never manage to solve because I “time out,” and it’s possible that as a result I’ve missed the opportunity to learn techniques that will be useful in the future.

As someone with small children, I found myself wondering whether the parents of these Brooklyn writers have actually done their children a service or disservice by funding a financially impossible career.

While there is something to be said for being able to follow one’s interests unconstrained by the need for money, it also seems like a hollow, grasping existence to chase status and popularity amongst a clique of likeminded fellows.

I suppose one could say something similar for a career spent chasing money, but at least money sometimes vaguely (vaguely!) approximates creating value for other people.

This isn’t exactly the same as a deathbed experience, but about two years ago I had a devastating lumbar disc hernia that landed me in the ER with such pain that I couldn’t stand, or even roll over.

I remember thinking that my biggest regret was that I had prioritized my job over my health. I’d stopped exercising and was overeating because my work was in a bad place and I was trying to scrape together every moment to make progress, and I’m sure that had a deleterious effect on my spine health.

I resolved that going forward I would prioritize my health (and my family) over my job.

I succeeded at maintaining those priorities for a while, and luckily my back healed relatively well. But I’ll admit that job stress ramped back up, and as I felt physically better it was all to easy to shift back into the same old patterns.

I had a related experience with an MVP gone awry. We were building a product that had an entrenched competitor (let’s set aside whether that was a good idea in itself).

We decided to build the core technology and go after some very easy wins first, while laying the foundation for bigger, more complicated wins.

I thought this was a fairly sound plan at the time but we hit three problems that would make me rethink such an approach in the future.

(1) the “easy” use case ended up being significantly harder than expected. I’d guess it took 4x longer to ship than expected, which allowed doubt in the project to run rampant.

(2) the easy wins were smaller than expected (it didn’t help that the system we were augmenting already had some of the features we were building, just implemented in a more ad-hoc fashion).

(3) the more complex and profitable features required significantly new features in the core, the implementation of which had become very complex and the design compromised during the process of shipping the late, underwhelming MVP, and at that point there was no more appetite to continue investing in this technology.

I have not, but I will certainly consider doing so!

I’m extremely sympathetic to this viewpoint. I spent the last couple years building a JIT compiler for PyTorch, and a constant battle was that people want to write “normal” idiomatic Python, but those idioms work to defeat optimizations, unless you go to really crazy lengths. Some of the recent work in this space (torch.fx in particular) has made things easier by allowing the user to “clean up” their model, but it was definitely an eye-opening experience.

This makes a huge amount of sense. I hadn’t considered how high level managers really can’t afford to project their stress publicly (bad for morale!) so I’m probably seeing an extremely airbrushed picture of their lives.