The script was running on a subset of the production machines, which still accounted for a rather large number. The message bus eventually transported bpftrace's output to the database.

The roll out was done in phases to ensure that the overhead would not be prohibitive as well as to make sure that it wouldn't cause egregious priority inversions.

Scribe [0] + LogDevice [1] + Scuba [2]. I believe Scribe was open sourced a very long time ago, and LogDevice more recently, but they are both archived now.

- [0]: https://engineering.fb.com/2019/10/07/data-infrastructure/sc...

- [1]: https://logdevice.io/

- [2]: https://research.facebook.com/publications/scuba-diving-into...