HNHacker News
TopNewBestAskShowJobs

alexgartrell

2,937 karma · joined March 25, 2009

SWE @ Thinking Machines Previously Operating Systems, etc @ Meta

Computers are cool

submissionscomments
alexgartrell··on Five Years of Btrfs
Yeah there are a remarkable set of container runtime tasks (package downloads, rootfs creation and management, etc) that are way easier with btrfs. It wasn’t always smooth sailing but luckily Chris, Josef, Omar and others are awesome and now (and for the last while) we are asking for features rather than fixes.
alexgartrell··on Ask HN: Joining Big Tech in One’s 40s
(At Facebook, not Google) when the technical writing is used to influence a technical outcome. Particularly a strategic one. a well written memo can change the way business is done and there’s not much ambiguity around where the idea came from.

That said, being an eager doc writer in an effort to claim credit or inflate the perception of your own work is a great way to lose friends and alienate people.

alexgartrell··on Ask HN: Joining Big Tech in One’s 40s
It's also been my experience as an engineering manager that it's best to think of yourself as a manager without authority.
alexgartrell··on OpenBSD system-call-origin verification
If you can change libc then you have equal to or greater permissions than the calling process. So this strengthens the security model.
alexgartrell··on Metamath Zero, a bootstrapping theorem prover
I went to high school with Logan and with Mario. Mario was in my AP Comp Sci class with Mrs. White. I remember being proud of my forays into encryption, concurrency, and network programming.

And then I looked over and Mario had implemented a graphing calculator (as in, take "y = x^2 + 4x -10" and emit a graph) in the windows command prompt. Then, when that was boring, he started researching and implementing the graphics algorithms for drawing lines at various slopes with minimal distortion. And then, when that was boring, he made his graphing calculator support polar coordinates (something I understood only as "oops, my graphing calculator is misconfigured").

Also, he was ~12 years old (IIRC) having skipped a few grades.

I've met a lot of smart people between Carnegie Mellon, Facebook Infrastructure, and the Linux Kernel community, but Mario's one of two or maybe three at the absolute top in terms of intellectual horsepower.

alexgartrell··on How to make Linux run fast again on Intel CPUs
I don't work at AWS (and haven't) but I do work in the world of big infra and my wild-ass guess is that the user stuff in AWS doesn't run as consistently hot as the SaaS stuff that Amazon runs itself.
alexgartrell··on BPF at Facebook and beyond
Short answer: no, same thing

Original bpf is a much simpler bytecode. Ebpf extended it and made it essentially x86_64 assembly (semantically). Then we all decided to call ebpf bpf for reasons.

alexgartrell··on Memcached 1.5.18 can now recover its cache between restarts
Not every memory access to a memory mapped file is automatically paged out immediately, so depending on vm dirty bytes, etc, you may get away with it for a while.

But the worst case will be dismal and not unlikely.

alexgartrell··on Linux Pressure Stall Information (PSI) by Example
Beyond implementation, how do you prove that a new statistic is correct and meaningful? Especially when it’s going to become part of the ABI forever?

In this case, beyond the patches, we built oomd and ran the whole contraption in production on hundreds of thousands of servers. When it meaningfully improved metrics there, it became an easier thing to sell.

alexgartrell··on BPF port-based firewall for systemd services
First, there's no need to do this "on demand." You can ship a .o around and load it with bpftool. My team does a version of this at Facebook (using libbpf directly) and it works really well. The big use case for on-demand programming is in tracing where you want to capture structs, but BTF and Compile-once, run-everywhere will mitigate this need [0]

Second, I don't actually agree. JIT compilation is a generally accepted approach -- who cares what the IR is? And the bpf runtime is ULTRA constrained, so you can't slip a `system("rm -rf /")` in there (by a long shot).

[0] https://www.kernel.org/doc/html/latest/bpf/btf.html

alexgartrell··on SAT to Add ‘Adversity Score’ That Rates Students’ Hardships
I don't think people realize that SAT scores are a lot more arbitrary than they look. If you're one of the lucky kids whose parents set them up with reasonable coaching (and academic expectations in general), you do much better.

I did good but not great on the SAT and only got into CMU because I was a student athlete. But once I had my foot in the door I did very well (high gpa, TA'ed, graduated w/ honors and awards) and then continued to do very well in industry (arguably).

If an adversity score can help out other kids like me who don't happen to be "impressive-for-a-D3-school" level athletes, then I think it's a great thing.

alexgartrell··on “I Fell In Love With My Direct Report”
I would love to hear the report’s take on what happened. The “and I still love her” epilogue makes me think that she probably has a pretty different take.
alexgartrell··on Gitlab's Journey from Azure to GCP
FWIW, Google certainly led on cgroup development for a while, but the cgroupv2 maintainer works at Facebook, and we're active in upstream container development in both linux and systemd.

I don't necessarily think you're wrong; I just think your argument, as stated, is weak.

alexgartrell··on LD_PRELOAD: The Hero We Need and Deserve
I recommend bpftrace as an entry point to working with bpf

https://github.com/iovisor/bpftrace

alexgartrell··on Python 3 at Facebook
I'm actually fairly certain it was "within working hours" free time. Facebook doesn't have 20% time or anything like that, but, in practice, there's lots of extra time for whatever you want. This was my experience as an Engineer at Facebook and I certainly "pay it forward" as a manager.

Probably worth adding that we ultimately created a python foundation team, of which Jason is a member.

Here's something dumb I created in my "within working hours" free time that Facebook let me open source: https://github.com/facebookresearch/py2bpf

alexgartrell··on Killing Off Wasabi (2015)
Wasabi could compile VBScript, which was the language they had at the time. It's actually a more or less identical approach to the one Facebook took with Hack, but Facebook had a lot more resource to make it awesome.
alexgartrell··on Killing Off Wasabi (2015)
I actually had an internship at Fog Creek in 2010. Ted Unangst (mentioned in the article) was my mentor. One thing I noticed about the folks there were that they were all crazy fucking smart. Joel had built this amazing recruiting machine that could pull down some of the real best of the best -- definitely a team that rivals any I've worked with at Facebook.

The drawback to this though was that the best are easily bored. FogBugz was an important cash cow for the company, but there were few truly novel problems to solve. This was not a "web scale" effort -- it was self-hosted software for tens to hundreds of seats.

So I speculate that they okay'ed the project just because it was an interesting thing to do that solved the underlying problem (albeit in a bit of a convoluted way). The simple cost/benefit analysis from a management point of view is simple; pros: your devs are happy and you get linux support; cons: it's a little more complicated and people make fun of you on the Internet.

Over time, Fog Creek came up with better outlets for this kind of nerd energy -- kiln was fun because there were mercurial nerds around; trello was a foray into server-side javascript. There was still a bit of that "the nerds rule" bad decision making that made FogCreek an awesome place to work, but it was more constrained.

Since then (and I'm way out of it now), a lot of the old school "just badass" programmers have kind of gone their own way and Fog Creek has hired folks who are better aligned with what they do and how they do it. I'm not sure if this is because of a masterful shift in recruiting philosophy or if the diminishing recruiting brand (due to Joel's blogging pause) led to the right kind of developers becoming interested naturally.

In either event, it was definitely a very interesting culture, and I'm happy to have been able to play along for a summer :)

alexgartrell··on Implementing Go's defer keyword in C++
ScopeGuards as imagined here (Facebook's folly library has one) are near-essential for writing reasonable systems code in C++.

   int fd = socket(...);
   if (fd < 0) return false;
   SCOPE_EXIT { ::close(fd); };

   if (ioctl(fd, ...) != 0) return false;
   if (ioctl(fd, ...) != 0) return false;
   ...

   return true;
You could totally just write the FdWrapper class, but then you're still a user handling resource cleanup, but with more lines.
alexgartrell··on When Simple Wins: Power of 2 Load Balancing
You're kind of right. You can also use something like jump consistent hash [0] which only requires you to have a consistent ordering of the hosts where you're sending the information. We (Facebook) use something similar for our caches. It requires a linear array of hosts but you've already got that if you're load balancing.

[0] https://arxiv.org/abs/1406.2294

alexgartrell··on How Facebook Tries to Prevent Office Politics
One thing we do is have ICs interview leadership/management candidates. This is a piece of signal we're always looking for.
alexgartrell··on How Facebook Tries to Prevent Office Politics
I think you're absolutely right, in general, but I've been at Facebook for 5 years now, and the one thing I'll say about it is that once you establish a track record of 1) doing good work and 2) making good long-term decisions, that trumps any considerations around level. The culture is definitely such that the very senior folks (Director++) will value trust over level, so it's very much "influence over authority."

Personally, this has lead to me getting messed with my managers ~never, which is a very high-order bit in my work happiness.

alexgartrell··on We only hire the trendiest
Also in the hundreds of interviews club for Facebook and I also do not care about ++ or standard library memorization stuff. I think the recruiter is just trying to do as much as he/she can for you, because our recruiters are inherently invested in the candidates for obvious reasons.
alexgartrell··on Big IPO, Tiny Payout for Many Startup Workers
> You're forgetting something: the toll it takes on you as a person.

> Those long nights,

What nerd hasn't spent all night building something cool? I've /never/ been asked to stay at the office after hours and it's not an expectation that any of the big tech firms have.

> long commutes

No one gives a crap when you get in. I usually leave for work at like 10 AM and complete my 30 mile commute in about 40 minutes -- it's no big deal.

> (if you don't live a block away),

I used to have a shorter commute but I think SF is more fun and it's not that big of a deal...

> they can all wear you down.

I suppose if they were as you've stated them, but they're not.

> You're pressured into working more

You're really not. Also, we get like 21 full days of PTO, and I'm eligible for my sabbatical in another 6 months, which is just an additional month of free PTO.

> and you have less time to spend with friends and family

In addition to the fact that I get to hang out with friends pretty much whenever I want (no one cares about the 9-5 thing), I'm able to use my generous PTO to go visit family, which I probably do 3-4 times a year (honestly, love my family, but more would be too much)

> (work IS your friends and family! :')),

I get to work with a lot of smart people who I deeply respect and do consider to be exceptional friends. I also have friends who work at different tech companies and even outside of tech, and I get them the same was as anybody else, hanging out.

> less time for dating

We've got plenty of time for dating. My friends are going on dates all the time actually. Dinner, Drinks, Hiking... they're all totally possible thing to do.

> (and let's be honest: if you're a single man in the Valley, you're not getting laid).

Not going to dignify this with a response.

> It just sucks.

You are incorrect

> Is that worth it?

Beyond the shadow of a doubt :)

alexgartrell··on Cello High Level C: A Fat Pointer Library
I totally disagree. I've done a lot of operational and low-level systems work and handling obscure OOM scenarios is /really/, /really/ hard, and not at all worth the engineering effort.

But, even assuming you handle every failed page allocation perfectly and your stack does the right thing, you can still screw yourself /trivially/ by blowing out the network stack, or because the writeback daemon can't keep up with your filesystem writes. So now we're talking about clamping kernel resources with userspace logic that does things like periodic fsyncs and userspace tcp acks before attempting to send more -- good luck.

alexgartrell··on BBS the Documentary (2013) [video]
Jason Scott (creator of this documentary) is fantastic. I highly recommend all of his talks on YouTube. He also did the defcon documentary which was pretty good.
alexgartrell··on How to Generate Millions of HTTP Requests (2012)
> One important thing to keep in mind when load-testing is that there are only so many socket connections you can have in Linux. This is a hard-coded kernel limitation, known as the Ephemeral Ports Issue. You can extend it (to some extent) in /etc/sysctl.conf; but basically, a Linux machine can only have about 64,000 sockets open at once. So when load testing, we have to make the most of those sockets by making as many requests as possible over a single connection. In addition to that, we'll need more than one machine to do the load generation. Otherwise, the load generators will run out of available sockets and fail to generate enough load.

If the # of ports is the bottle neck, you can configure more IPs on a single host.

alexgartrell··on Facebook was down
This should be at the top.
alexgartrell··on Why aren’t we using SSH for everything?
It depends how you set up the forwarding.

Casually looking at strings /usr/sbin/sshd on my devserver reveals that there is a mode to create a tun device, which sets up a more traditional VPN-ish thing (basically, your computer sees a new interface and routes certain sections of IP space to that prefix. sshd sucks in these packets and forwards them as is). This results in two layers of congestion control, which (IIRC -- I'm not a TCP guy) Congestion Control Algorithms aren't set up to handle very well.

The more traditional mechanism for tunneling though (the one most often referenced in the posts and things I've seen) is a layer 7 forwarder. Basically it creates sockets to listen for connections on either ends on a specific port, recv's data as a traditional network user, and then turns around and sends the data over the multiplexed transport to the end server. It essentially strips away the congestion control on the first hop (because you're just going to localhost anyway) and there's no TCP/IP packets in TCP/IP packets thing going on anymore, just your application stream in their header transport thing. It's less flexible, but is good enough if you want to set up a simple proxy or something and doesn't have the same pitfalls.

If you need the flexibility, something like OpenVPN might be good (though I haven't used it myself).

And you might also want to look at stunnel as an ssl tunnel alternative that lets you have more control over what certs go where without archaic and dangerous /etc/ssh/*_config meddling

alexgartrell··on Facebook's Top Open Data Problems
Aw shucks.

Mcrouter was the first piece of software I worked on at facebook. It's nice to see that you like it! (Though the only thing that lives on from me is the umbrella protocol)

alexgartrell··on Facebook's Top Open Data Problems
That's what TAO (mentioned in the article) is for
← PreviousPage 3 of 18Next →