adtac

You can use Subtrace entirely locally without talking to subtrace.dev using the -devtools flag. For example:

    subtrace run -devtools=/subtrace -- python3 -m http.server

Then go to http://localhost:8000/subtrace and you'll see the Chrome DevTools interface running in your browser like a regular app. Any request sent to http://localhost:8000 should appear there in realtime. Note that the -devtools flag is kinda best effort because Subtrace is intended to be used in production.

We have fully on-prem installations for enterprises, of course (e.g. if you're in a highly regulated industry like healthcare).

Thanks!

re the Wireshark analogy: the reason I used that was because: (1) Subtrace operates at roughly the same level in the operating system stack, (2) has similar capabilities, (3) has an overlap in use-cases, and (4) has been the most effective at communicating what Subtrace is in my experience so far. I can see why the analogy is not a perfect 1:1 mapping (obligatory xkcd: https://xkcd.com/624), but naming things is hard and taglines are just names in idea space :)

You can tag each request with arbitrary key-value maps and search over these later. For example, if you add a `x-subtrace-tags: user=foo, project=bar` header on the response, you can apply a `tags.user == "foo"` filter in the dashboard to see all requests across your entire infra from that user and only that user. Each request is pre-populated by default with tags like hostname, pod name, AWS/GCP location, etc.

It's like Honeycomb's wide events but even better because: (1) you can see whole request including the payload alongside the event fields, and (2) it's fully automatic and requires no code changes out of the box (you can incrementally add these tags when you find a need for each one instead of the huge upfront cost from instrumenting the hell out of your codebase).

> use subtrace in an ephemeral pod for debugging purposes

That's a great suggestion. It'd be like kubectl exec-ing into a shell inside the pod, but for network activity. I think I'm going to prototype this tonight :)

> pod filtering such as namespace and label selectors

Yep, Subtrace already tags each request with a bunch of metadata about the place where it originated so that you can filter on those in the dashboard :) Things like the hostname, pod, cluster, AWS/GCP location are automatically populated, but you can also set custom tags in the config [1].

[1] https://docs.subtrace.dev/tags

httptap is really cool! Their technique is different (they do a filesystem mount instead of intercepting syscalls like Subtrace does) but both tools effectively reach the same goal using different routes.

That's totally fair, I can see why Wireshark wouldn't be the most accurate description to someone working on those kinds of problems. And fwiw, I wish my problems (before Subtrace) were cool enough to need to whip out Wireshark for packet-level inspection :)

Subtrace already works great on Kubernetes (https://docs.subtrace.dev/kubernetes)! Add a single line to your image's Dockerfile and that's it.

I'm working on an even simpler way where you can just `kubectl apply` a DaemonSet or a Helm chart to get automatic tracing for all pods in your cluster instantly without any code-level changes. If anyone is interested in beta testing this, email me at adtac@subtrace.dev, I'd love to understand your usecase!

Yep, certificate pinning is the one scenario Subtrace can't handle in my experience, but thankfully, it's fairly rare like you said. And IMO there is no general solution to the problem [1], but it's one of those very interesting problems to daydream thinking about when you're stuck in traffic or whatever :)

We still try our best by handling as much of the long tail of environments with some library/framework specific workaround (e.g. Deno bundles all TLS certs in its binary so we set the DENO_CERT env var when applicable).

[1] https://news.ycombinator.com/item?id=42923998

For now, yes :)

Since we operate at the TCP level, we can actually handle pretty much any protocol. I have an implementation of a postgres handler in my git stash that intercepts and shows the SQL queries executed + the resulting rows alongside the HTTP request that triggered it (I still need to do some robustness and correctness testing before it's ready to merge). With a handful of other protocols like MySQL, Mongo, Redis, Kafka, or even FTP lol, I think Subtrace can cover most practical dev workloads.

Btw Subtrace can already record .pcap files today since it's just a simple TCP stream proxy, but raw network packet captures are mostly only useful when you're implementing new protocols, which 99% of the people using Docker containers today aren't doing. It's also a solved problem because you can just run `apt-get install tcpdump` inside the container.

Automatic tracing for app-level protocols that is easy to setup, works everywhere, lightweight for prod, fast to search, and can show the data in a clean interface is still insanely difficult today. That's the problem Subtrace is trying to solve.

Subtrace proxies the program's connection using a regular TLS connection to the upstream server. For example, if you do `subtrace run -- curl https://example.com`, curl thinks it's talking to example.com over TLS, but it's really talking to Subtrace locally. Since we injected the ephemeral root CA into the system store, curl will trust the valid TLS certificate that Subtrace presents for example.com. From within the same server, Subtrace will handle the actual TLS connection to upstream example.com. That upstream connection is undecipherable to outsiders.

Everything is exactly as secure as before Subtrace. In other words, using Subtrace doesn't make the NSA's job any easier ;)

standards ftw!

More than 0.01 ms, less than 0.10 ms. Noticeable when you're a high frequency trading firm, probably not otherwise :)

In the hot path, Subtrace is just a dumb proxy that copies bytes. All of the processing + indexing happens offline in a different machine (in Clickhouse).

Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :)

We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent). If so, we append the ephemeral root CA to the list of actual CA certificates; if not, we let the kernel handle the file open like usual. This way, none of the other programs are affected, so only the program you start with `subtrace run` sees and trusts the ephemeral root CA.

After we get the program to trust the ephemeral root CA, we can proxy outgoing TLS connections through Subtrace transparently but also read the cleartext bytes.

All of this is fully automatic, of course.

If I do, whether or not I remember my comment, I'll certainly remember yours :(

I know that the US government is more complex than twitter lol. I just think it's stupid to automatically invalidate an idea because it was tried in a less complex system.

All key personnel were immigrants? Why did the others come back? Also, since the H1B were key to operations and therefore smart, couldn't at least some of them have found a new employer? 61% of the H1B population switched jobs that year, so why did all key H1B come back?

I think key personnel came back because they wanted to. It's the simplest explanation.

Yes, so it's a good thing the first step is:

> [Step 1] Question every requirement. Each should come with the name of the person who made it. You should never accept that a requirement came from a department, such as from "the legal department" or "the safety department." You need to know the name of the real person who made that requirement. Then you should question it, no matter how smart that person is. Requirements from smart people are the most dangerous, because people are less likely to question them. Always do so, even if the requirement came from me. Then make the requirements less dumb.

> Had they decided to move on

Why didn't they?

People should read Elon Musk by Walter Issacson. Here's an excerpt from the chapter on his "algorithm":

> [Step 2] Delete any part or process you can. You may have to add them back later. In fact, if you do not end up adding back at least 10% of them, then you didn't delete enough.

He thinks this is a feature, not a bug. Is he wrong? I don't think so.

>from untrusted sources

if you can't trust your own backend, step one is reevaluating your life choices

IMO there's no general solution to the HTTPS part that will work for all kinds of programs and the long tail of certificate pinning implementations.

As a proof by counterexample, imagine malware that uses TLS for communication and goes to great lengths to obfuscate its compiled code. It could be a program that bundles a fixed set of CA certificates into its binary and never open any files on the filesystem. It can still create valid, secure TLS connections (at least for ~10 years or so, until most root CA certificates expire). TLS is all userspace and there's no guarantee that it uses OpenSSL (or any other common library), so you can't rely on hooking into specific OpenSSL functions either. If the server uses a self-signed certificate and the client accepts it for whatever reason, it's worse.

With that said, it's definitely possible to handle 99% of the cases reliably with some work. That's better than nothing.

How would hooking on write(2) solve TLS? You'll be able to read and modify the ciphertext, but the process will never call write(2) with the plaintext bytes, so you can't actually read the HTTP request. You'll just see the encrypted bytes that go on the wire, but so does the NSA :)

You need the kind of CA certificate trick that httptap uses. It comes with its own set of caveats (e.g. certificate pinning), but it can be made to work reliably in most practical scenarios.

I've spent an unjustifiable amount of time thinking about this specific problem building Subtrace [1], so I'm genuinely very interested in a simpler / more elegant approach.

[1] https://github.com/subtrace/subtrace

Thanks! Subtrace uses BPF, not eBPF :) I think eBPF could be made to work with the same approach, but there's a few differences:

- eBPF requires root privileges or at least CAP_BPF. Subtrace uses seccomp_unotify [1], so it works even in unprivileged environments.

- eBPF requires using eBPF maps as the data channel + weird restrictions in the code because of the eBPF verifier. IMO these two things make it way harder to work with for the kind of networking logic that both httptap and Subtrace have in userspace. Everything is perfectly possible, just harder to reason about and debug.

>half-finished attempt of the same thing for the firefox network tab

Hahahah this is incredible. Something something great minds.

[1] https://man.archlinux.org/man/seccomp_unotify.2.en

Using a TUN device for this is a really cool idea! And the "How it was made" section is one of the best things I've read in a Github README.

I'm building something called Subtrace [1] but it can intercept both incoming and outgoing requests automatically. Looks like we converged on the same interface for starting the program too lol [2]. Subtrace's purpose is kinda different from httptap's though (more observability / monitoring for cloud backend services, hence the emphasis on both incoming and outgoing). Also, it uses a different approach -- using Seccomp BPF to intercept the socket, connect, listen, accept, and ~10 other syscalls, all TCP connections get proxied through Subtrace. We then parse the HTTP requests out of the TCP stream and then show it to the user in the Chrome DevTools Network tab, which we repurposed to work in the browser like a regular webapp.

Any fun stories there from running programs under httptap? Who phones home the most?

[1] https://github.com/subtrace/subtrace

[2] https://docs.subtrace.dev/quickstart

It thinks hard about it

> It is strictly less likely than Earth-bound abiogenesis

> all panspermia does is layer on an additional series of long-shot coincidences

Disagree. There is a lot more of not-Earth than Earth. I agree that the probability density (per unit volume of the universe) of life originating on Earth is much higher than anywhere else, but there's just so much more room out there for life to originate that the probabilistic cost of traveling to Earth is tiny in comparison.

All you need is a proto-life that's stable in an inert environment with sufficient radiation shielding. It could've originated billions of light years away and still have had enough time to arrive on Earth 4.2 billion years ago. That's a mind-boggling number of Earth-like environments.

In fact, proto-life doesn't even need to look like Earth life, so even environments that are hostile to current Earth life today could've been the cradle of origin (as a wise man once said, "life, uh, finds a way"). Additionally, environments that used to be Earth-like but eroded away are candidates too since all we need is for life to have escaped before the erosion.

In my opinion, panspermia is strictly more likely than Earth-bound abiogenesis.

The question above was not preceded by anything; that was the whole question. The facts are at hand in the sense that you have the internet and you're allowed to use it. The hard part is knowing what to search and recognising the answer when you see it. This is much harder than any 7th grade comprehension test I've done :)

The paper evaluates itself on the GAIA benchmark and it was my first time hearing about it, so I tried to evaluate myself as a human.

Here's a level 3 question from the GAIA paper (level 3 = hardest):

>In NASA’s Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute? Exclude any astronauts who did not spend any time in space. Give the last name of the astronaut, separated from the number of minutes by a semicolon. Use commas as thousands separators in the number of minutes.

I timed myself solving the problem. It took me 9 minutes, 5 Google searches, 14 web pages, multiple Ctrl+F in these pages and 1 calculator use to figure out the answer.

DynaSaur seems to have a 10% to 20% success rate at this level.

Try for yourself. This is one of the few empirically grounded reference levels for how far we are from AGI.

A much more useful trick I learnt from Tyler Cowen's podcast is to ask what they think is the most underrated / overrated thing in the category. Everyone understands that the answer is going to be subjective, so there's no pressure to be diplomatic. And in my experience, the answers are also high variance, which leads to more interesting conversations (most people agree that Messi is the greatest of all time, but everyone has a different opinion on who is the most underrated / overrated).

In your opinion, what fraction of the small restaurants in a typical city are engaging in money laundering at this scale? And if weighted by volume, what fraction?