tanelpoder

I guess one point of the article is that eBPF-injected programs are another place where malware can hide. But then again, it's easy to list all currently loaded eBPF programs (well, unless there's a way to somehow hide/disconnect them, but keep them running).

This would help against basic ps commands but if your security tool reads the task's PF_ flags (from /proc/PID/stat or kernel memory), you'd still see that your [task] is not an actual kernel task... (the PF_KTHREAD flag is 0x200000):

  $ ln -s /bin/sleep '[kernelstuff]'
  $ ./\[kernelstuff\] 999 &
  [2] 29499
   
  $ awk '{ printf("%x\n", $9) }' /proc/29499/stat
  40400000
   
  $ pgrep kthreadd
  2
   
  $ awk '{ printf("%x\n", $9) }' /proc/2/stat
  208040

Edit: as you said, it's superficial disguising (that may even work in some cases)

Didn't at all expect this to be downvoted, but point taken (pun intended)

... beat Apple at running Windows

System call internal numbers are meaningless, are different across platforms and can change across kernel compiles/upgrades. There are also gaps in the internal numbering, so the max value seen in define _NR_syscall doesn't show the actual number of used syscall numbers in your current kernel...

Edit: this answer has some relevant details:

https://stackoverflow.com/questions/63713056/why-is-their-a-...

Good to know, I suspected that this might be the case, but never got to confirm this. I guess one could set up a test comparing the syscalls listed in syscall_64.tbl (or syscall table read from kernel memory) with the syscalls listed under /sys/kernel/debug/tracing/events/syscalls

The question was about syscall numbers. The man pages don't show the numbers.

The internal syscall numbers are defined in syscall_64.tbl in kernel source and you can parse the (libc-used) syscall numbers from various /usr/include/asm/syscall.h files, but these files don't necessarily contain all the available syscalls in your currently running kernel.

You can read your kernel's/platform's currently available syscalls from the relevant /sys/kernel/debug/tracing tracepoint files (I posted a link to my script in another comment here). That way you'll see all currently available system calls and their arguments, but tracefs doesn't show the syscall internal numbers (syscall table slot numbers), but rather the new syscall ID.

The internal syscall numbering can change when you switch/build a new kernel and syscalls have different numbers across platforms. Syscall 0 is "read" on x86_64, but is "io_setup" on aarch64, for example. The new syscall ID aims to provide stable numbering with no conflicts and overlaps across platforms and kernel versions, as I understand.

Do you mean arguments and the internal syscall number used for a syscall on your given platform?

I recently had enough of parsing the various syscall.h files on different architectures and wrote a debugfs syscall info reader instead. That way you can see all tracepoint-instrumented syscalls and arguments available exactly on your currently running kernel on your platform:

https://tanelpoder.com/posts/list-linux-system-call-argument...

Edit: changed "all" to "all tracepoint-instrumented" based on a comment below - some added syscalls don't (immediately) get instrumented with a tracepoint so tracefs wouldn't show them (until someone instruments them in a later kernel version as seems to be the case). The tracefs approach has been good enough for me, but the only 100% guaranteed way to see all currently available syscalls would be to read the syscall table from kernel memory and see which syscall handler kernel functions they call (as the syscall name itself is meaningless inside the kernel).

I'm curious why this post is still not the 1st (but 2nd after an ebook reader announcement), despite all the upvotes.

Probably not, but maybe they rely on some Active Directory server that is running it?

But if the Azure outage is due to Windows machines crashing because of the currently ongoing CrowdStrike crash/reboot loop issue, then such servers might end up being down in all regions. Looks like there might be some advanced lessons to be learned about blast radius here...

For some NoSQL champions it may still be eye-opening that with SQL you can push down resultset correctness and processing logic to the cabinets, instead of implementing all this (again) in the application layer...

The other alternative is LTTng, released in 2005 (requires loading a kernel module):

https://lttng.org/

Lots of the low level exception/trap/fault handling functions are blacklisted, probably to avoid lockups and unwanted recursion mayhem:

  $ sudo wc -l /sys/kernel/debug/kprobes/blacklist 
  783 /sys/kernel/debug/kprobes/blacklist

Edit: Perhaps an alternative approach would be to attach probes to relevant (precise) PMU events. There's also this prototype of adding breakpoint/watchpoint support to eBPF [1]. But actually doing stuff within this context may get complicated very fast, so would need to be severely limited, if feasible at all.

[1] https://ebpf.io/summit-2020-slides/eBPF_Summit_2020-Lightnin...

GitHub & docs: https://github.com/apache/arrow-nanoarrow

Yep and if there's a limit, it should be documented and going past that should gracefully fail with an error, not a surprise crash...

There are also plenty of multitenant apps out there that have the same set of tables (with their own data) duplicated across all schemas.

Good list!

Also, the launch happens fast - if you plan to record it with your phone, set it up on a tripod or something before and not muck with it at all, you’ll miss the first seconds of the launch fast and by then the rocket is already pretty far up.

For example I used something like "pid:module:funcname:entry" probe for userspace things (not pid$123 or pid$target, just pid to catch all PIDs using the module/funcname of interest). And back when I tested, it didn't automatically catch any new PIDs so these probes were not fired for them unless I restarted my DTrace script (but it was probably year <2010 when I last tested it).

Execname is a variable in DTrace and not a probe (?), so how would it help with automatically attaching to new PIDs? Now that I recall more details, there was no issue with statically defined kernel "fbt" probes nor "profile", but the userspace pid one was where I hit this limitation.

Perhaps familiarity with the syntax of DTrace if coming from Solaris-heavy enterprise background. But then again, too many years have passed since Solaris was a major mainstream platform. Oracle ships and supports DTrace on (Oracle) Linux by the way, but DTrace 2.0 on Linux is a scripting frontend that gets compiled to eBPF under the hood.

Back when I tried to build xcapture with DTrace, I could launch the script and use something like /pid$oracle::func:entry/ but IIRC the probe was attached only to the processes that already existed and not any new ones that were started after loading the DTrace probes. Maybe I should have used some lower level APIs or something - but eBPF on Linux automatically handles both existing and new processes.

I sent it back before the return deadline, would have considered keeping it if it had supported showing more than one display of my MacBook. I know by now they've released some sort of a "very wide display" in a VisionOS update, but back then it didn't really make sense. I thought, ok, I'll just try using one Mac desktop and all the other apps, messages & browser would be the Vision OS native ones open side by side. Then it turned out I had to connect my bluetooth mouse/keyboard to the Vision device instead (if I wanted to type something into the browser/Messages) and it was too much friction.

I did like the brief period of working (or just browsing stuff) while lying on the couch, but I knew from the beginning that I didn't need an even lazier position for staring at a screen all day.

Thanks, yes I was more or less aware of that (I'd been using DTrace since Solaris 10 beta in 2004 or 2003?)... By rollout I really meant "getting the word out there"... that's half the battle in my experience (that's why this post here! :-)

What I loved about DTrace was that once it was out, even in beta, it was pretty complete and worked - all the DTrace ports that I've tried, including on Windows (!) a few years ago were very limited or had some showstopper issues. I guess eBPF was like that too some years ago, but by now it's pretty sweet even for more regular consumer who don't keep track of its development.

Edit: Oh, wasn't aware of the timeline, I may have some dates (years) wrong in my memory

Yeah, I already see limitations, the last one was yesterday when I installed earlier Ubuntu versions to see how far back this can go - and even Ubuntu 22.04 didn't work out of the box, ended up with some BCC/kernel header mismatch issue [1] although the kernel itself supported it. A workaround was to download & compile the latest BCC yourself, but I don't want to go there as the customers/systems I work on wouldn't go there anyway.

But libbpf with CO-RE will solve these issues as I understand, so as long as the kernel supports what you need, the CO-RE binary will work.

This raises another issue for me though, it's not easy, but easier, for enterprises to download and run a single python + single C source file (with <500 code lines to review) than a compiled CO-RE binary, but my long term plan/hope is that I (we) get the RedHats and AWSes of this world to just provide the eventual mature release as a standard package.

[1] https://github.com/iovisor/bcc/issues/3993

The on-cpu sample profiling is not a big deal for my use cases as I don't need the "perf" sampling to happen at 10kHz or anything (more like 10-1Hz, but always on).

But the sched_switch tracepoint is the hottest event, without stack sampling it's 200-500ns per event (on my Xeon 63xx CPUs), depending on what data is collected. I use #ifdefs to compile in only the fields that are actually used (smaller thread_state struct, fewer branches and instructions to decode & cache). Surprisingly when I collect kernel stack, the overhead jumps higher up compared to user stack (kstack goes from say 400ns to 3200ns, while ustack jumps to 2800ns per event or so).

I have done almost zero optimizations (and I figure using libbpf/BTF/CO-RE will help too). But I'm ok with these numbers for most of my workloads of interest, and since eBPF programs are not cast in stone, can do further reductions, like actually sampling stacks in the sched_switch probe on every 10th occurrence or something.

So in worst case, this full-visibility approach might not be usable as always-on instrumentation for some workloads (like some redis/memcached/mysql lookups doing 10M context switches/s on a big server), but even with such workloads, a temporary increase in instrumentation overhead might be ok, when there are known recurring problems to troubleshoot.

Same here, until now. I built the earlier xcapture v1 (also in the repo) about 5 years ago and it just samples various /proc/PID/task/TID pseudofiles regularly, it also allows you get pretty far with the thread-level activity measurement approach, especially when combined with always-on low frequency on-CPU sampling with perf.

Yep (the 0x.tools author here). If you look into my code, you'll see that I'm not a good developer :-) But I have a decent understanding of Linux kernel flow and kernel/app interaction dynamics, thanks to many years of troubleshooting large (Oracle) database workloads. So I knew exactly what I wanted to measure and how, just had to learn the eBPF parts. That's why I picked BCC instead of libbpf as I was somewhat familiar with it already, but fully dynamic and "self-updating" libbpf loading approach is the goal for v3 (help appreciated!)

Brendan Gregg's site (and book) is probably the best starting point (he was involved in DTrace work & rollout 20 years ago when at Sun) and was/is instrumental in pushing eBPF in Linux even further than DTrace ever went:

https://brendangregg.com/ebpf.html

A related blog entry:

- https://blogs.oracle.com/linux/post/bpftune-using-reinforcem...

I had the same experience/question at some point when I noticed (and investigated), why didn't a new Ubuntu Linux install break out interrupt usage in vmstat anymore. [1]

Perf+flamegraphs were able to catch in-interrupt-handling samples as I was on a bare metal machine with PMU counters accessible. But the various NVMe I/O completion interrupts happened all over the place, even when the CPUs were in random userspace code sections. Neither the bottom-up & top-down FlameGraph approach made it visually clear in this case.

But since interrupt time did show up and since the FlameGraph JS tool had a text search box, I searched for "interrupt" or "irq" in the search box - and the JS code highlighted all matching sections with purple color (completely distinct from the classic color range of flamegraphs). And seeing that purple color all over the place made me smile and gave me a strong visual "a-ha" moment. Probably with some extra JS tinkering, these kinds of "events scattered all over the place" scenarios could be made even clearer.

[1] https://news.ycombinator.com/item?id=26139611 (2021)

Great idea! And if for some reason you feel like your filenames should stay as they are (without a comma), you could just add symlinks to all executable files in your bin directory:

  $ cd ~/bin
  $ for x in $(find . -type f -perm /a=x -exec basename {} \;) ; do echo $x ; done
  temps

  $ for x in $(find . -type f -perm /a=x -exec basename {} \;) ; do ln -s $x ,$x ; done

  $ ls -l
  total 4
  lrwxrwxrwx 1 tanel tanel   5 Jun 23 16:38 ,temps -> temps
  -rwxr--r-- 1 tanel tanel 251 May 30 23:26 temps

But if the number of platters, rotation speed and bit-density is exactly the same as in regular drives, the number of “bits” moving past the read/write heads per rotation would still stay the same? Would be interesting to know how much time typically is spent on actually reading the bits vs the tiny seeks & track calibration when reading, say, 10 GB sequentially.

Edit: other comments below explain the missing part, in a regular drive, apparently just one head can read/write (to one platter) at a time due to calibration reasons, even if your disk physically has 8 platters. So having two actuators allows two platters out of 8 to do IO concurrently.