Fewer alerts, faster answers, a rotation people will hold

Arcline builds condition monitoring hardware for industrial machinery and sells the data platform behind it as a subscription. Roughly 3,400 machines across 140 customer sites push telemetry into their ingestion pipeline. When ingestion stalled, customers noticed gaps in their dashboards before Arcline did, and the first sign was frequently a support ticket.

There was no shortage of monitoring. Three tools had accumulated over four years, each with its own alerts, and the on-call channel received around 214 alerts per week. Almost none corresponded to anything a customer would notice. The rotation had two willing participants out of a team of nine, and the engineering lead had been paged eleven times in the previous month for a disk threshold on a queue node that auto-recovered every time.

Median time to restore across severity one and two incidents fell from four hours ten minutes to twenty-two minutes in the six months after handover. The largest single contributor was tracing: investigations that previously meant correlating timestamps across three tools now start from a trace that shows exactly which stage of the pipeline stalled.

Alert volume fell from 214 per week to 47, with 11 of those severe enough to page a human. The on-call rotation now has seven volunteers out of nine engineers, which the engineering lead describes as the most useful outcome of the engagement. Telemetry costs also fell, because retention tiers and sampling replaced the previous approach of keeping everything at full fidelity for ninety days.