HNHacker News
TopNewBestAskShowJobs

javierhonduco

250 karma · joined August 30, 2012

https://hondu.co/
submissionscomments
javierhonduco··on Notes on the Cost of Go Finalizers
Curious what the overhead of dealing with files in Go would be if finalisers were cheaper / not in use.

Sorry there’s no link to the source, AFK right now, but files opened with os.Open will be automatically closed once all its references have been collected.

Found out about this behaviour some months ago while debugging some code at work.

javierhonduco··on Introduction to Sentry Symbolicator
Thanks for sharing! I work in this space too (profilers, debuggers, etc) and have seen a good amount of interesting edge cases.

Are there any interesting bugs and edge cases that you could share? Always good to hear good war stories from production :)

javierhonduco··on File Expiration Using BPF
ptrace is not used in this project, is it possible you meant some other system call or project?
javierhonduco··on File Expiration Using BPF
Exactly this. I should have mentioned it in the post. Might amend it.

Would be interesting to compare the chances of race conditions with inotify vs BPF for this contrived use-case.

javierhonduco··on File Expiration Using BPF
Author here! As some comments mention, there are other approaches that could make more sense.

The aim of this proof-of-concept was to showcase how BPF enables the kernel to be programmable. Many behaviours can be retrofitted without having to submit changes upstream. In some cases the overhead may be lower than using other methods (due to avoiding context switches, etc). Even if you decide that the changes can be useful for the broader community, the new feature can be first implemented in BPF, where it’s faster (and safer!) to evolve it.

javierhonduco··on Blind: Meta's layoff rumor March 14-16, cut ~1200 D1-D2-M1, M2 and most M0
They are levels in the management track.

M: Manager D: Director

The rough matching of levels is: M0, E5 or senior engineer, M1 to E6 or staff engineer, M2 for manager of managers, etc.

Director is the level above M2.

javierhonduco··on Frame pointers vs. DWARF – my verdict
This is a very good point. Thanks for the context on how ORC does it, I was not familiar with how it evolved.

In our case this is not an issue yet. We split up the unwind tables in chunks of up to ~1MB so we typically have in very few L2 cache misses.

This has to be done once per frame so in my 4 year old i7 processor. There are roughly `2 L2 cache misses (+ few other misses from reading ancillary data structures) * frames`.

We have more work to do regarding benchmarks but I collected some numbers the other day and 90 frames can be walked in less than 500ns (slide 50 https://fosdem.org/2023/schedule/event/walking_stack_without...)

There are more optimisations we have in the works to make our unwinder more efficient, mostly related to fitting more data in the CPU cache and reducing cache misses.

javierhonduco··on Frame pointers vs. DWARF – my verdict
As surprising as it seems, it does! My colleague and I spoke about this in our project in FOSDEM https://fosdem.org/2023/schedule/event/walking_stack_without.... Let us know if you have any questions or feedback :)
javierhonduco··on Frame pointers vs. DWARF – my verdict
This would be ideal. There's some great work by folks at Oracle in this space: SFrame (https://www.phoronix.com/news/GNU-Binutils-SFrame) née ctf_frame that I hope will be integrated in the kernel.

As this will take few years, in the meantime I've developed a DWARF-based unwinder in BPF [0]. Some perf maintainers showed interest in this, so thanks for bringing up the GSoC project idea, didn't occur to me!

[0]: https://news.ycombinator.com/item?id=33788794

javierhonduco··on Frame pointers vs. DWARF – my verdict
The pervasive lack of frame pointers is the reason why we've developed a custom format derived from DWARF unwind information thanks to some insights: DWARF unwind information is incredibly flexible, it supports many architecture and allows restoring any arbitrary register. But we only need 3: the frame pointer, the stack pointer, and in non-x86 the return address.

While DWARF unwind info doesn't use that many bytes, but unfortunately reading and parsing that information is quite expensive.

For that reason I've developed a new unwinder that uses custom unwind information derived from DWARF (https://www.polarsignals.com/blog/posts/2022/11/29/profiling..., previously discussed in https://news.ycombinator.com/item?id=33788794) that runs in BPF. This new compact representation can be binary searched easily and each unwind row has a size of 16 bytes. I am currently working on reducing it down to ~10 bytes.

All the code is fully OSS (Apache 2.0 for userspace and GPL for BPF), and part of the Parca project (https://github.com/parca-dev/parca-agent). We've also given a talk this year in FOSDEM going deeper into how we made it scale for many big processes.

javierhonduco··on Most FOSDEM 2023 videos have been uploaded
Ah! Glad you liked my talk :)
javierhonduco··on Most FOSDEM 2023 videos have been uploaded
7 years of cgroup v2: the future of Linux resource control (https://fosdem.org/2023/schedule/event/container_cgroup_v2/) was particularly enjoyable
javierhonduco··on Fedora now has frame pointers
That would be great! There’s a precedent in Ubuntu, in case it could help to push this forwards `libc6-prof`.
javierhonduco··on Fedora now has frame pointers
I’m looking forward to this change. Ensuring fast, reliable stack traces when debugging under pressure or tracing complex systems is invaluable. Once you try it it’s hard to go back.

It’ll take some years until frame pointers become more mainstream. Still, some people might decide to never enable them in libraries they use or their own code.

That’s why I’ve been working on a BPF DWARF-based unwinder to fill the gap. It’s fully OSS (Apache 2.0 for user space and GPL for BPF) and should get out of beta in ~1month.

Previous HN discussion [0]. Happy to help anybody that wants to integrate it in their tools / systems.

- [0]: https://news.ycombinator.com/item?id=33788794

javierhonduco··on Finding people's use of /usr/bin/Python with the Linux audit framework
I’m not sure there’s any public write-up on this, sorry
javierhonduco··on Show HN: A version control system based on rsync
Yes! This is something they Facebook’s version of Mercurial implements. It’s called Commit Cloud [0]

- [0] https://engineering.fb.com/2022/11/15/open-source/sapling-so...

javierhonduco··on Finding people's use of /usr/bin/Python with the Linux audit framework
The script was running on a subset of the production machines, which still accounted for a rather large number. The message bus eventually transported bpftrace's output to the database.

The roll out was done in phases to ensure that the overhead would not be prohibitive as well as to make sure that it wouldn't cause egregious priority inversions.

Scribe [0] + LogDevice [1] + Scuba [2]. I believe Scribe was open sourced a very long time ago, and LogDevice more recently, but they are both archived now.

- [0]: https://engineering.fb.com/2019/10/07/data-infrastructure/sc...

- [1]: https://logdevice.io/

- [2]: https://research.facebook.com/publications/scuba-diving-into...

javierhonduco··on Finding people's use of /usr/bin/Python with the Linux audit framework
At my previous job, I wrote a simple bpftrace script [1] to track Python2 invocations. The output of this program was sent to a message bus and eventually stored in a columnar database for exploration.

While Python2 got removed from the base operating system image, we wanted to uncover weird edge cases of engineers running special builds. We found more than one :)

I think BPF is underused for safe deprecations. It's super useful to prove that an executable is not loaded, or that a method is not called while being traced :)

[1]: https://github.com/javierhonduco/bpf-playground/blob/master/...

javierhonduco··on DWARF-Based Stack Walking Using eBPF
That's right! There's no "traditional" loops as programs have to be proved to terminate at some point.

That being said, very recently support for bounded loops landed [0]. It's very exciting and useful, and I've seen it reduce verification times significantly, but we can't use this yet as it requires kernel 5.3 or greater, and we would like to support as many users as possible!

[0]: https://lwn.net/Articles/877062/

[1]: https://github.com/iovisor/bcc/commit/38304256c49a02aecbf78f...

javierhonduco··on DWARF-Based Stack Walking Using eBPF
Thanks so much!

You are right! Hopefully these formats will become standard and implemented in more toolchains. My worry is that there seem to be several "competing" formats for a more compact DWARF debug (as opposed to unwind) information. I hope this doesn't lead to more fragmentation of the ecosystem.

Definitely! I assume you work at Google, so most of your binaries probably have some crash handlers set up already, right? I find this very useful, and wish it were more commonplace. I really like Folly's implementation of this idea [0] that just requires a single line of code: `folly::symbolizer::installFatalSignalHandler()`

javierhonduco··on DWARF-Based Stack Walking Using eBPF
That's very curious! We definitely see way more DWARF opcodes in the wild, including `DW_CFA_remember_state`, `DW_CFA_restore_state`, `DW_CFA_advance_*`, among others.

Statistically speaking, a profiler has more chances of bumping into program counters (PCs) that fall in ranges defined by a larger set of DWARF opcodes. We run at 100Hz, which can quickly yield a large number of different program counters, while I assume that most debugging sessions won't hit that many different PCs.

I'm using Fedora 36 which ships `gcc (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)`

> I've been writing a debugger from scratch for the past few months

Sounds fun! Are you planning on writing about this? I am sure there would be many people interested in reading about this topic!

Either way feel free to reach out at javier @ my company's domain if you want to discuss the DWARF opcodes thing further!

javierhonduco··on DWARF-Based Stack Walking Using eBPF
Thanks!

As @brancz mentioned, Delve uses DWARF unwind information to produce backtraces (they are stored in the .debug_frame section for Go).

You are right, Go enabled frame pointers for all architectures as of 1.17 [0]. This is enabled to allow profilers to work well, without having to use to techniques such as the one we describe in our post.

When it gets funny is that there's `gopclntab`, a 3rd option in Go to unwind stacks, used by `panic` and I believe other parts of the runtime. If you are interested in more details, Felix Geisendörfer's repo contains way more details [1]

[0]: https://go.dev/doc/go1.17

[1]: https://github.com/DataDog/go-profiler-notes/blob/main/stack...

javierhonduco··on DWARF-Based Stack Walking Using eBPF
Let me check! I don't our approach to be significantly more compact than DWARF's. DWARF is very compact despite being so much more expressive.

Funny that you mentioned perfect hashing, I thought about this, but didn't go for this approach as it would have too many drawbacks for our use-case:

- It would further increase the complexity of the unwinder, which is already not trivial. Ideally, we would like to reduce the surface area of things that can go wrong.

- Implementing perfect hashing in BPF might be tricky for several reasons. First it might take quite a bit of precious instructions, but also, it would force us to ship a compiler to generate the BPF code in the fly rather than shipping it pre-compiled. We really want to avoid this.

- Last, it would force us to generate entries for every program counter, while thanks to using binary search we can omit redundant entries. DWARF expressions would suffer from a similar issue.

javierhonduco··on DWARF-Based Stack Walking Using eBPF
I find this surprising! Was this for off the shelf applications or some custom binaries?

We see DWARF expressions such as `DW_CFA_def_cfa_expression` on the regular. See the "Test Plan" section and commit messages of the PR that introduced support for this particular opcode, as well as the simple executable that shows this opcode [0] and its printed unwind table [1]

[0]: https://github.com/parca-dev/parca-agent/pull/1058

[1]: https://github.com/parca-dev/testdata/blob/7f9206570a743206f...

javierhonduco··on DWARF-Based Stack Walking Using eBPF
> I'd love for every binary/library to embed enough information to cheaply unwind during profiling

Definitely! We hope to be able to change this. There's the .ctf_frame format [0] that could be used too. While its design goals are very much aligned to ours, implementing a reader for this format in BPF will be complicated.

> and symbolize shortly afterward

Definitely, I think that C Type Format (CTF) [1] could be a candidate here. Another possibility would be to get BTF (BPF Type Format) [2] working in userspace. While it wouldn't cover everything that DWARF deals with, it might be a reasonable tradeoff for symbolization.

FWIW both in Polar Signals and Parca (our OSS offering) we resolve to async symbolization done in the server, as it's very expensive to do and we want to reduce load where the Agent runs.

[0]: https://gcc.gnu.org/wiki/cauldron2022talks

[1]: https://lwn.net/Articles/795384/#:~:text=The%20Compact%20C%2....

[2]: https://docs.kernel.org/bpf/btf.html

javierhonduco··on DWARF-Based Stack Walking Using eBPF
(this talk is great!)

I think this would be cool. Have you considered asking Alexei directly?

javierhonduco··on DWARF-Based Stack Walking Using eBPF
100% this. Disabling frame pointers is not worth it in my opinion. Hindering debugging and profiling is a massive regression. It's one of these historical (due to x86 having fewer registers) and short-term thinking changes that have unfortunately stayed with us. I'm optimistic that this can change but it will take a while.

BTW, I'm a big fan of your work, thanks for Flask and Jinja!

javierhonduco··on DWARF-Based Stack Walking Using eBPF
Author here!

This project looks interesting and as you mention it seems conceptually similar! I would agree with that quote, generally speaking, but I think it really depends on a binary to binary basis.

Some binaries outsource most of the work to dynamic libraries. Unfortunately, DWARF expressions are typically emitted for these program counter ranges, so it's desirable to at least implement a subset of expressions [0].

Even if that's not the case, we want to produce profiles that are as accurate as possible :)

You are totally right, they were discussing kernel stacks where stakes are higher as it needs to work perfectly, otherwise kernel live patching would not reliably work, among others. The kernel has now some unwind table format, that can be used in x86_64, called ORC [1].

That being said the parser for DWARF would still have to live in the kernel, and I am not sure if kernel devs would like to accept such a patch.

Ideally we would transition to an unwind-specific format for user-space (something like ours, for example) and perhaps have a suitable unwinder in the kernel, rather than having to implement it in a BPF program. This is something we are considering for the future, but it's not free from problems (increased executable size, redundant unwind information, etc). But this is exactly why we wanted to have a conversation with the communities interested in this work!

[0]: https://github.com/parca-dev/parca-agent/pull/1058/commits

[1]: https://lwn.net/Articles/728339/

javierhonduco··on DWARF-Based Stack Walking Using eBPF
Thanks! :)

Good question! We are building atop the incredible job that the BPF community does. BPF is a restricted environment [0], among others, the stack size is very small, programs have to be proved to finish, and there's no access to arbitrary kernel function.

We've done most of the work to loosen the minimum required kernel, sorry if that wasn't clear. The minimum supported kernel is 4.18.

More context, for those interested:

In order to provide an API to interact with the rest of the kernel, we use BPF helpers, which are like a library (sorta "syscalls") functions that BPF programs can make.

Not every BPF helper is available in a given kernel. The BCC project has a comprehensive and up-to-date list of the different helpers and other features and their introductory commit [1].

The minimum kernel that we can support is then decided by the most modern helper or feature we use. In our case, the most modern features we require are:

- BPF tail calls: since 4.2

- BPF Type Format (BTF): since 4.18

[0]: https://www.kernel.org/doc/html/latest/bpf/bpf_design_QA.htm...

[1]: https://github.com/iovisor/bcc/blob/master/docs/kernel-versi...

javierhonduco··on DWARF-Based Stack Walking Using eBPF
We don't know yet, but we are planning to start this work soon. In theory the unwind tables should be very similar thanks to still following the DWARF spec.

We would have to modify the BPF code to handle saved return addresses in arm64, which as far as I am aware are not defined by the ABI, so they might be at any offset from the previous frame's stack pointer (what DWARF calls CFA).

There's (at least) these other things that we might want to look into, such as:

- split stacks

- pointer authentication

- unwind tables stored in a different ELF section

← PreviousPage 2 of 3Next →