Cloud-native service meshes provide topology-aware tracing, causal attribution, and controlled debugging, but their sidecar mechanism does not transfer to tightly-coupled Message Passing Interface (MPI) jobs. Slurm accounting and Prometheusstyle dashboards report what resources did, not which rank caused a slowdown or why. We present MAD-MPI (Mesh-Aware Debugging for MPI), the MPI counterpart of our mesh-aware service debugging that carries the same observability principles without a sidecar. MAD-MPI combines MPI profiling interface (PMPI) spans with host-context signals. In the evaluated prototype, off-CPU classification uses span-derived thread CPU time, while eBPF/bpftrace probes prototype the scheduler and flow signals for production integration. Because collective synchronisation makes per-rank wall time nearly uniform, MAD-MPI attributes slowdowns using per-rank compute excess, then uses off-CPU time to distinguish compute stragglers from descheduled contention victims. On a containerised Open MPI testbed with controlled faults, MAD-MPI localises compute and intermittent stragglers to the correct rank in all ten controlled runs, localises CPU contention to the affected node and identifies its descheduled ranks, and correctly reports no single timing culprit for uniform network loss and load imbalance. At one rank per core, the hotpath cost is 40–80 ns per intercepted call with low single-digit wall-clock overhead, while contended and oversubscribed cases show higher overhead.
- Címlap
- Publikációk
- From Service Meshes to MPI: Sidecar-Free Observability with PMPI and eBPF