HNHacker News
TopNewBestAskShowJobs

jmillikin

11,783 karma · joined March 23, 2009

https://john-millikin.com/

john@john-millikin.com

Currently semi-retired; formerly at Google (2011-2017) and Stripe (2017-2022)

submissionscomments
jmillikin··on Rust's Poor Composability
Reads like someone used to working in an abstract high-level language, who doesn't like the verbosity mandated by a language aimed squarely at embedded/systems programming.

Maybe one day there will be a language that has the expressive type system of ML, the garbage collection of ML, and the predictable performance of ML. Then people who don't need to care about the stack layout differences of an async function can use that language instead of Rust.

jmillikin··on My experience being blocked by Google Safe Browsing (2022)
Port 10080 is blocked on most browsers[0] per the WHATWG "bad ports" list[1]. That particular port was added to the list due to the Slipstream attack[2] that made the news a few years ago[3].

You don't have to switch to a browser that ignores standard security mitigations. Just pick a different port for your service.

[0] I just tested Chrome, Firefox, and Safari.

[1] https://fetch.spec.whatwg.org/#bad-port

[2] https://samy.pl/slipstream/

[3] https://news.ycombinator.com/item?id=24955891

jmillikin··on Programmers, beware: ChatGPT has ruined your magic trick
Another article written by someone who thinks the difficult part of programming is the typing.

If he thinks LLMs are a threat to programmer employment, he doesn't realize the only thing programmers can't get enough of is trying to automate away their own jobs. Push out everyone who's outcompeted by a Markov chain and watch everyone who's left celebrate the reduced bug count.

jmillikin··on The roots of Japan's Indian curry
This might be a language barrier issue, but I have never seen an Indian-style restaurant advertise a dish as "curry" in any country I've visited. Even in Japan (I live in Tokyo and have several インド料理 shops nearby), even if the shop itself has カレー in the name, the menu will say what each thing is (keema, sagh paneer, etc).

Imagine sitting down and opening a menu and seeing something labelled "soup" or "sandwich"!

jmillikin··on Wat [video] (2012)
I'm struggling to phrase a response that would be polite enough for HN, so instead I'll encourage you to re-read both my post and your link more carefully.
jmillikin··on Wat [video] (2012)

  > And the problem WASM solved really would be almost
  > better instead if we just had a ring 4 for web
  > browsers. It seems clear that X86 has won, unless
  > you think that RISC V still has a chance
Mobile has a large and growing lead over desktop[0] for web traffic, and every popular mobile device released for the past ~decade has run on ARM.

[0] https://techjury.net/blog/mobile-vs-desktop-usage/

jmillikin··on Surprising Scalability of Multitenancy
I'm not referring to overcommit or actual "cpus pegged at 100%" contention, but to simple loose bin-packing.

Imagine you have three Ruby services, where each is allocated 10 cores of CPU time (via pinning with cpuset). If you give them each an 16-core VM, then there'll be 18 cores of "wasted" CPU. If you instead bin-pack them onto a 32-core VM, then they'll have the same number of cores at a lower price point.

If each service runs at 50% capacity with 2000ms latency during steady state, how much extra latency would you expect the service to have on the bin-packed configuration vs the single-VM?

My position is "very little extra latency", the other person's position was "a lot of extra latency due to hardware contention in (for example) the memory controller".

(If you're reading this and thinking "NUMA node locality", then you're operating two or three levels above where this org was in terms of optimization.)

jmillikin··on Surprising Scalability of Multitenancy
Having had some back-channel status updates after I got pushed aside, the lesson is that some people accept or discard ideas based on employment background.

On the team I was on, newly-hired managers from Amazon accepted the ideas of ICs who had previously worked at Amazon, and rejected ideas from people who had not. I didn't realize this was a pattern until I already had one foot out the door, but even if I had realized it earlier it wouldn't have helped. I was ex-Google, so the ex-Amazon folks tagged every document with the bozo bit before they'd even opened it.

My takeaway was to be cautious of companies where the culture is imported through mass hiring from single companies. I joined in the middle of the "Google wave", which was relatively peaceful (per Google's culture at the time). When the "Amazon wave" arrived it was quite a shock; their culture was much more adversarial and authoritarian than anywhere I'd worked before. By the time I left, there were signs the Amazon folks were starting to get sidelined by an emerging "Oracle wave".

jmillikin··on Surprising Scalability of Multitenancy
Both AWS and GCP offer the ability to schedule VMs onto isolated machines:

https://aws.amazon.com/ec2/dedicated-hosts/

https://cloud.google.com/compute/docs/nodes/sole-tenant-node...

The AWS offering is pretty much turn-key. I've not used the GCP version, but it seems to be similar if you're willing to create a separate "project" for each security domain.

Once your company has any PII and/or has regulatory obligations (PCI, HIPAA, etc) then it's worth spending a bit extra to make sure sensitive components are running on their own hardware.

jmillikin··on Practical Libc-free threading on Linux
I think the case of Windows is slightly different. Its architecture has the same backwards compatibility challenges of GNU libc, but where GNU took a user-hostile approach ("just recompile"), Microsoft invested thousands of person-years to mitigate and resolve issues as they were discovered.

Most folks reading this are likely familiar with Raymond Chen's blog posts about compatibility shims for older binaries, or stories like the Windows 95 work around for memory-management bugs in SimCity. That's the kind of thing that happens when you design an OS ABI with problems, but are committed to bearing the cost of those problems yourself rather than passing them along to users.

I'm sure if you asked today's Windows team if they'd have a different approach, they'd rattle off a half-dozen great ideas for backwards-compatible syscall ABIs. There's been lots of research in that area since the foundations of our current OS ecosystem were laid down 30 years ago.

jmillikin··on Practical Libc-free threading on Linux
For context, this is how it worked in old-school UNIX. If you look at (for example) SunOS, the kernel ABI is undocumented and everything in userspace depends on libc. The OSes and userspace languages of the day were deeply coupled -- C was the language of UNIX.

You can still see the remnants of this design in macOS, except instead of libc it's libSystem. Apple has decoupled their OS from C (in favor of Objective C, then Swift) but the idea of a userspace entry point to system functionality survived.

Linux is different from traditional UNIX because it's a kernel-only OS. There's no "Linux libc", and third parties have written multiple implementations of the C standard library that run on Linux. The only way this structure can work is for the kernel itself to provide stable ABI, and the standard approach at the time (mid 90s) was numeric syscall codes with positional parameters.

jmillikin··on Practical Libc-free threading on Linux
Yes, I understand the mechanism, but why would you want to structure your OS like that?

You either have to maintain a stable syscall ABI, or a stable userspace function ABI, and it's not like one is materially more difficult than the other. If anything, the Linux approach has proven to be superior -- glibc has had multiple breaking changes over the years and to this day users will resort to Docker to run binaries that depend on "old" glibc, but programs written against the Linux syscall ABI twenty years ago will run fine to this day.

Even if you decide to have a userspace trampoline/shim, it doesn't make sense for that to be libc. The C standard library is huge and requiring userspace programs to link it regardless of implementation language does nothing except add forty years of bad ideas into your process's address space.

jmillikin··on Surprising Scalability of Multitenancy
You'd generally use co-tenancy for workloads that are mutually trusted. Privileged services (authn/authz, machine management, deployable artifact builds) get put onto separate hardware, since their footprint is small enough that the extra 200% cost isn't material.
jmillikin··on Surprising Scalability of Multitenancy
At my previous employer, there was at least one person with a "staff software engineer" job title who believed that running more than one Ruby server process per AWS virtual machine would lead to unacceptable contention at the hardware level. I was never able to convince them that Linux handles tens of thousands of processes just fine, or that even if you do one per VM there's nothing stopping AWS from scheduling those VMs onto the same machine.

I guess whether you consider someone like that to be a "serious person" or a "charlatan" depends on your own point of reference.

In that case, their arguments were more persuasive to management than mine were. I found the experience baffling.

jmillikin··on Practical Libc-free threading on Linux
What alternative is there? Linux is a more minimalist OS than old-school UNIX, and libc as a userspace construct allows more evolution in that space (see glibc vs musl vs uclibc).

I could see an argument for small shims injected by a vDSO (AIUI this is what Fuchsia does), but clearly the 80s-style "libc is the OS boundary" is obsolete.

jmillikin··on Why construction projects always go over budget
A software project's design doc is like an architectural mock-up: sometimes useful, but not necessary and certainly not part of the construction process.

Programmers write the blueprint. The construction is handled by the computer (compiler, linker, tests).

jmillikin··on Ask HN: Year old CS student. Should I continue?
LLMs automate the uninteresting part of programming, which is the physical typing of code. They are not capable of reasoning or logical induction, so the job of being a programmer is still safe. They're basically just fancy codegen.

A lot of the articles you see floating around about "the end of programmers" are written by people who think software is designed by managers, which is only true in workplaces that you would not want to be in. If you can independently design and implement software, then you can work at a company that treats programmers like craftsmen, which is more rewarding both mentally and monetarily.

That said, the tech industry tends to move faster than undergrad currriculae. It's likely that any specific technologies you learn in school are already obsolete[0]. Focus on the fundamentals (CS as a subdomain of mathematics) and you'll find your knowledge has a much longer shelf life.

[0] When I was in university, the more "vocational" classes offered by the CS department tended to teach libraries and languages that had already fallen into disuse outside. Things like OpenGL immediate mode when the world had moved to shaders, or CVS when everyone used SVN and Git had been just released.

jmillikin··on Ask HN: Do you think this is the start of the new financial crisis?
If you happen to be holding any "worthless" Treasury bonds, I would be happy to take them off your hands for $0.10 on the dollar.
jmillikin··on A brief guide to Kubernetes networking
This might be a longer answer than you expected/wanted; sorry.

Let's say you have a bunch of machines, mounted in racks, installed in datacenters. You might have a naming convention like `m3r5dls1.acme-prod.com` to identify "machine #3 in rack #5 in datacenter #1 in The Dalles, Oregon". The acme-prod.com TLD is used to avoid cross-contamination between internal and public-facing identities. You'll have remote management services running on those machines, which identify themselves with an mTLS certificate issued to the machine hostname.

For services it's a little more complex, because there are at least three common use cases:

1. You have a service like `my-namespace/hello-world`, and you'd like to be able to send it RPCs. The clients don't care where the service backends are located, they just need to be able to resolve that service name to a set of (IP, port) pairs.

2. The same service as above, but now there's a locality requirement -- you want to send RPCs to a service in a specific place. Maybe for latency (closer = faster), maybe cost (local datacenter = no egress = cheaper), maybe regulatory ("all customers in India must have their data fetched from the Mumbai datacenter").

3. You have a specific instance of a service, for example when tracking the progress of a long-running operation. In Kubernetes this would be a specific pod.

It would also be nice if the services' mTLS certs are served from the same CA as machine certs, since that allows you to write tools that send RPCs without having to do their own parsing of the destination address.

The `{service_name}.cluster.local` format is not flexible enough to accommodate these goals, but if you're willing to use longer service identities then there's no problem:

1. `{service_name}.{namespace}.any.acme-prod.com` can resolve to any instance of that service, regardless of location.

2. `{service_name}.{namespace}.dls1.acme-prod.com` can resolve to any instance of that service in the datacenter `dls1`.

3. `{pod_id}.{service_name}.{namespace}.dls1.acme-prod.com` resolves to a specific instance.

Now you have consistent identities and universal mTLS authentication, so you can write authorization policies that grant services specific permissions they need to do their work.

jmillikin··on A brief guide to Kubernetes networking
Thank you for the correction regarding Azure! I knew they had a CNI plugin, but did't realize it has the limitations you describe (I've only used GCP and AWS).

  > I've not had a problem with CoreDNS? Ours seems to cope fine at about
  > 500 requests / min. The bigger problem is stuff in the cluster being
  > really irresponsible about how many DNS queries they make…
My expectation for DNS is that name lookups should function at thousands of requests per second. Serving that kind of QPS implies caching, and thus some sort of update notification -- for example watching the EndpointSlices resource for changes, and pushing them directly into the DNS server's state.

(As you might notice from my article + employment history, my expectations around cluster sizes differs from the community norm.)

jmillikin··on A brief guide to Kubernetes networking
For folks interested in the Kubernetes networking model, I recommend looking elsewhere. This post appears to be content marketing wrapped in a thin shell of introductory tutorial for an approach that isn't used by non-trivial deployments.

First, if you want to do a networking tutorial, start with something simple -- an HTTP server and curl, for example. You want to be able to tcpdump the traffic to understand what's going on under the hood. The first half of the blog post is some massively complex magic -- CRDs? An operator? Just why? -- and there's no reason to run an Elasticsearch instance just to test out packet routing.

Second, you almost certainly don't want to use kube-proxy. It uses (abuses?) iptables/nftables in a way that will make your sysadmins cry tears of blood. For small deployments, every major cloud provider (AWS, Azure, GCP, etc) has a CNI plugin that lets you allocate pod IPs out of a dedicated NAT prefix. For larger or bare-metal deployments, either use IPv6 natively (if available) or 6to4 (if on an IPv4-only network). I wrote a tutorial on the 6to4 approach[0], but honestly if you have someone on staff who is familiar with the Linux kernel network configs they'll probably have a better idea of how to set it up to work with your system.

Third, you probably want to avoid getting super-magical with your DNS. Approaches like that described in the article (coredns configured to directly resolve non-namespaced Kubernetes service names) have poor performance once you get beyond toy-sized clusters, and having to hunt down all the places your code does a single-name lookup is not fun. Instead, configure a "normal" DNS server (or equivalent non-DNS address resolver) to read Kubernetes-announced endpoints in bulk (with caching, etc), and use hostnames like `myservice.mynamespace.mycluster.yourproddomain.com`, which lets you (1) figure out where your packets are getting routed to, and (2) provision mTLS certificates to pods that let them authenticate themselves as a given service identity. Yes, it's longer, but your future self (or future underlings) will thank you.

[0] https://john-millikin.com/stateless-kubernetes-overlay-netwo...

jmillikin··on Scaling Kubernetes to 7,500 nodes (2021)
You'd generally want each of those 7500 machines be a full-sized server. No point running Kubernetes on tiny VMs, since its purpose is to provide bin-packed scheduling in a datacenter.
jmillikin··on Scaling Kubernetes to 7,500 nodes (2021)
Kubernetes is based on Omega, which was an attempt to replace Borg using a more distributed architecture. Borg uses a single master with elections, with the replicas serving as hot spares.

Source: I was on Borg-SRE at Google for ~six years, then worked with Kubernetes at Stripe for ~five.

jmillikin··on Stripe announces new round of funding and plan to provide employee liquidity

  > It is a capital raise. It's right there in the announcement. They are
  > raising new capital, issuing shares, then buying back different shares.
If the transaction completes with the company having the same amount of shares as before, and the same amount of money as before, then it's not a capital raise.

  > But the series I investors are getting a little over 10% of the company
  > for their 6.5B, instead of about 5% for their 6.5B (if the valuation was
  > 120B as suggested).
The valuation is independent of the ownership percentage. If 5% of the company is held by employees (with locked-up RSUs) and 5% held by institutions, then allowing employees to sell their shares to institutions does not change how many shares exist.

Even if the employees decide to sell all of their shares (leaving the external investors with 10%), whether this happens at a 50B or 95B or 120B valuation doesn't affect the percentages. It only affects how much money the employees receive for selling their ownership interest.

  > The existing equity holders as a group are in a position which is about
  > 5% different economically than it could have been
You're still confused about valuation, price, and share count. Issuing new shares directly affects the fair market value of a company, and can be additive (if sold above fair value) or dilutive (if sold below). In contrast, temporary price fluctuations (due to bubbles, panic, etc) do not affect the value of the shares except in very unusual circumstances.
jmillikin··on Stripe announces new round of funding and plan to provide employee liquidity
Their point is that this isn't a capital raise and Stripe isn't issuing new shares.

Also, if it were to be a capital raise: issuing new shares comprising 5% of a company is not the same as having the price of existing shares reduce by 5% (in many ways).

Your post is completely off-track.

jmillikin··on My startup banking story
The author is used to thinking of their bank account as an anonymous number, with operations handled impassively by whoever happens to be staffing the bank's reception desk that day. They probably never considered 35 million dollars to be an amount worth fussing over ("this is not a lot of money to a big bank, they'll barely notice").

The bank thinks of multi-million dollar business accounts as a deeply personal relationship between the CEO and account manager, where the AM takes the CEO to fancy dinners and golf outings and remembers the names of the CEO's children. HashiCorp's account might have been the largest that had ever been opened at that particular branch.

This article is an amusing(?) story about those two worldviews making contact.

jmillikin··on My startup banking story

  > Perhaps we're just supposed to know that, but this is never
  > directly explained in the story and I found it a little upsetting.
This might be an age / generational thing, but all of the issues described in the article are immediately obvious to people who grew up before internet banking.

The default for business accounts in "traditional" banks is that they're handled by a specific named account manager, who is given the credit (or blame) for that account's growth over the years. So walking into some remote branch and opening an account will tie that account directly to the political success of whoever was sitting across the desk from you when you signed the paperwork.

It's a failure mode that doesn't need an explanation to people who are the expected audience of the article. In software terms, imagine an article about launching an MVP that starts off with "so I copy-pasted some PHP from StackOveflow and ran it with default permissions in the same database that stores customer credit card details ...". There doesn't need to be a long digression about why that's bad.

jmillikin··on Credit Suisse finds ‘material weakness’ in reporting, scraps exec bonuses
It's a bit gutsy to ask someone who casually drops a CDS valuation formula into a conversation to justify themselves against an Investopedia article, as if a retail-oriented content farm is the definitive resource on finance.

To answer your question, "idiosyncratic risk" in this context means that an individual company might have problems that don't broadly apply to the rest of its industry. For example Credit Suisse might have management that is bad at running a bank, leading to repeated investment losses and regulatory actions well beyond what's normal for big multinational banks.

Look at the CS balance sheets over the past few years, or its stock price and PE ratio, or hell just Google "Credit Suisse books loss" and look at how many times they tried to stick their fingers in the wrong cookie jar. They got hit by the Hwang thing, they got hit by Greensill, and apparently they can't even accurately report how much money they're making (losing).

I wouldn't want to have any position on their equity either.

jmillikin··on Silicon Valley Bank failure could wipe out 'a whole generation of startups'
Salaries, rent, debt payments, and AWS bills are charged in dollars. They don't accept T-bills, even though in accounting terms those are "cash-equivalent".

Notice how so much of the fallout is secondary to failed payroll transfers.

Big companies do the same thing, except their cash is kept in bigger banks with more attentive risk-management departments. Apple will never miss payroll due to someone at JP Morgan doing a YOLO on 10-year bonds.

jmillikin··on Ask HN: Am I Crazy?
This page looks like you can't decide whether to create a lightly interesting tech demo[0], an avant-garde art project[1], or an introductory article on s-expression markup templating.

Pick one. If you want to do all three, do so separately.

Regarding the project itself, as someone with quite a bit of functional programming experience, I don't see much appeal over standard HTML (at the low end) and DOM-aware templating (e.g. JSX). The Lisp language family has never delivered on the promised benefits of using an AST as the textual encoding, and to the extent nested expressions are useful in markup I think HTML/XML gets 90% of the value with 5% of the frustration.

Regarding your HN post's question, one part that did activate my "crazy sidewalk preacher" detector was the text on beauty of mathematical constants and symmetry. The following quote is exactly the sort of thing I would expect to read from a young adult who is losing their connection to consensus reality:

  the strangest of transcendental numbers, π = 3.141592653589793,
  whose decimals are totally randomly distributed, (high
  entropy), is captured in a beautiful, perfectly determined
  series, (high neguentropy). Our (decimal) representation of
  numbers is decidedly poor and I imagine that in the noosphere
  minds use continued fractions to manipulate them.

  It's endless.
I don't mean to be insulting when I say this, but that sounds like a Terry Davis quote. If you find yourself thinking in that mode frequently, it might be worth asking a mental health professional for a screening.

[0] As a sort of personal knowledge base, I find the lambdatalk format interesting. Horizontal lists of narrow columns is a classic of printed text but rarely encountered on a screen, and I like how the figures (formulae, code blocks, etc) are integrated into the prose.

[1] The angled subsection, bizarre choices of example content, and long lists of links to subpqges(?) come across poorly in terms of information, but are aesthetically pleasing in a sort of surreal way.

← PreviousPage 6 of 34Next →