Debugging a rustc segfault on Illumos
sunshowers.io
sunshowers.io
> [...] and why heterogeneous platforms are important
This prompts me to wonder vaguely whether there's any untapped juice in fuzzing approaches that might be relevant here. As in, how much of the platform (including configuration and heuristics, and so on) could be fuzzed as program input?
Regarding fuzzing... maybe? I've wondered that a couple of times myself but in reality there are really just a finite number of platforms, and so much of this is determined at compile time by library call availability. But I'm probably not thinking as deeply about this as someone could be, and I'd be interested to hear other folks' thoughts.
RFD 26 talked about the context around this choice: https://rfd.shared.oxide.computer/rfd/0026
There's a lot more to it than just this one thing I mentioned :)
This point bothers me, but I can't say with confidence that it's completely wrong. I know there are occasional rifts within the open source world, but I wish I knew two things:
1) How much overhead (in totality) is there when contributing to a project you don't control?
2) How different is the end result of collaboration between distinct groups or individuals versus doing things separately?
My comment wasn't so much about the overhead of collaboration, but of the chance of there being significant differences in opinion, leading to a place where we'd basically have to fork anyway. Remember, in this specific context, we're talking about an operating system and hypervisor that are core to our product, and we're building our own hardware.
You can't get one single answer for these questions for the entirety of the open source community. Even cross-language norms can be different. These things are inherently tradeoffs.
Edit: also worth pointing out (again) that the bug actually exists everywhere -- it was just being masked on the other platforms.
That said, for CI, cross building is much easier to scale than tracking down every permutation. For something like Illumos, that's not too bad. But for Solaris/SPARC? Heh.
Since cranelift-codegen was an optional component that can be disabled,
the x.py tooling could notice if a build failed in such a component
Edit: Just took a quick look, and it sure looks like cranelift isn't built by default (at least that's what config.example.toml says and the none of the defaults available via 'x setup' seem to override that).
# This is an array of the codegen backends that will be compiled for the rustc
# that's being compiled. The default is to only build the LLVM codegen backend,
# and currently the only standard options supported are `"llvm"`, `"cranelift"`
# and `"gcc"`. The first backend in this list will be used as default by rustc
# when no explicit backend is specified.
#codegen-backends = ["llvm"]
The defaults for the Config struct are set in src/bootstrap/src/core/config/config.rs and codegen-backends is indeed just "llvm" (line 1167).
Nothing in src/bootstrap/src/core/build_steps/compile.rs appears to override that list.
So that's all very curious (to me). Did the gcc backend get built as well?
Tangentially: a year on and the Github interface is still nasty to use – and one of the big motivating factor for me backing off of hacking on the cross build stuff. Every day seems to bring a new WTF moment. If I could get one thing for my birthday it would be for Rust to wean itself off of Github.
Back in the day I was eyeing one of those Toshibas that used to come with Solaris, but they were hard to get in Europe.
> [the core dump is supposed to be in the CWD, and named core, but isn't; what gives?]
Followed by,
$ find / -name core -type f
/-- requires -- evidence
| ^
v |
answer -- requires -/
> But there are also other bits of evidence that this theory doesn’t explain, or even cuts against. (This is what makes post-mortem debugging exciting! There are often contradictory-seeming pieces of information that need to be explained.)
I wish more people appreciated this; too many people are apt to sweet such discrepancies under the rug. This post does a good job on not just following through on them, but also showing how figuring some of them out ("why is our stack weird?") leads to the key insights: "oh we're using stacker and … $the_bug".
I do wonder how the author managed to notice that line in a 1.5k line stack trace, though. The "abrupt" address change would have probably gone unnoticed by me. (The only saving grace being a.) it's close to the bottom b.) most of the rest is repetitive, an artifact of a recursive descent parser recursing, and if we just consider that repetition "one chunk", it gets a lot smaller. I still dunno if I'd've seen it, though.)
Agree about the power of brute force solutions! When you're struggling to get a foothold, those kinds of approaches are extremely helpful. For the cwd thing specifically, it was clear in hindsight, and I should have known about it (having written nextest which has to handle crate cwds carefully). But I don't beat myself up too much over it, and now I know where to look next time this happens.
quick check of the docs
On Solaris or Illumos I'd eye coreadm(8) and maybe chuck something like this in my shell login script:
coreadm -p $HOME/coredumps/core.%f.%p
This post suggests that actually growing the stack automatically may just be very difficult with standard POSIX APIs: https://news.ycombinator.com/item?id=36020073
The Rust runtime installs a SIGSEGV handler but it's purely informative, and it can be overridden without affecting correctness.
To "grow" a stack at runtime, if the virtual address space to just add mappings in the "right" place is unavailable (already used for something else, say), it means a stack switch. That isn't entirely impossible. But if done anywhere else but in the "base thread initialisation" (as seen in that example) it becomes complicated and hazardous. It's more like "coroutines" then, your current function "spawns" a func on a new callstack and on return, undoes that and cleans up. One can imagine things like thread-local signal handlers to "simulate" this transparently but that would become more of a piece of programming performance art. If there ever were an "unbelievable unix underbelly programming contest" (UUUPC) stuff like that might well make it.
Stack size is the one area where Windows does have something like "overcommit" by default, as you can separately configure the reserved and commit sizes of the stack: https://learn.microsoft.com/en-us/windows/win32/procthread/t...
Definitely a fun read. Debugging crashes has, in the last decade or so, become something a bit like a "lost art". Noone looks at coredumps in the cloud ...
I don't want to outdo you on Solaris debugging (plenty of old-time Solaris folks at Oxide who are totally capable to show how to get things like open files and their contents from a coredump, or how to configure the system to include those should it not be there ... etc ... etc ... Solaris has the best coredumps for all that's worth ...).
A note on the fix side of things though, while adding pthread_get_attr_np() for stack location/size gives Solaris the Linux interface, it already has its own for those - pthread_attr_getstack{size,addt}(), see https://docs.oracle.com/cd/E19455-01/806-5257/6je9h032l/inde... - I happen to remember this because I used this decades ago somewhere in the Solaris name lookup code to choose at runtime between using alloca() and malloc() ... don't ask. Those were different times.
I remember debugging a gnarly c++ crash in some vendor code we had extended on solaris circa 1998 and I ended up with a coredump[1] which, when I loaded it into the sun workshop debugger, caused the debugger to dump core. That's one of those moments you go and get a coffee while figuring out what to do next.
[1] From a runaway recursive process
But stacker already has an implementation that uses the np method, and it was easier to co-opt that.
One minor meta point if the author is (still) around: there is something strange with the styling of the hexadecimal literals in the code. Instead of having the prefix "0x", they look like "0×" even though they seem to be normal x:es in the source.
Edit: Firefox 128.0.3 on Linux, btw.
Sounds like a horrible default. That's a security risk (working directory might be readable by untrusted users), and pollutes a random directory with a file that could cause problems for other applications processing files in that directory.
A fixed location inside the user's home directory feels like a much better choice to me.
Security-wise I wouldn't worry too much about the Solaris/Illumos defaults. There, dumps can be created in up to three contexts: system-wide "global", zone-wide "global", and local. All are created with mode 600 and global dumps are created with owner of uid 0. Local core dumps are owned by user that owns the process unless its uid/gid has changed (e.g. setuid/setgid), then the owner is the superuser like the global dumps.
Otherwise yeah I'm not a huge fan of leaving core dumps in the current directory. What if you're doing something on a read-only filesystem?
(dtrace userland is a library; you can, if you so choose, write your own tooling there. The command is "just a wrapper", a lot of similarities to bpftrace on Linux)
(can I be forgiven since I haven't really touched anything Solaris-ish for near 15 years ?)
... mdb sure has a full-on "oldskool" CLI. I don't think that's a good thing, from a perspective of tool accessibility to developers...
This isn't some passive aggressive gotcha, I'm actually curious what people prefer about the Solaris distros nowadays. I know Zones and ZFS are cool, but FreeBSD supports Jails and ZFS out of the box, but maybe there are cool features I'm not aware of.