A vendor case study exists to sell, and there’s nothing wrong with that on its own. Every vendor publishes its best results. The problem is a headline percentage on a one-page PDF is usually the least informative part of the document, stripped of everything that would let you judge whether it applies to a plant that looks anything like yours. Reading a case study well means asking for the parts that got left out, and knowing which missing pieces are disqualifying versus merely annoying.
The percentage alone tells you almost nothing
“Reduced downtime by 22%” sounds precise, and precision is exactly what makes it easy to mistake for evidence. A percentage with no basis attached could describe one machine or an entire plant, a single good month or a sustained year, a comparison against a real baseline or against nothing measured at all. Ask what the 22% is a percentage of. Downtime hours compared to what period. On how many machines. Compared to what baseline, and how was that baseline itself measured. A vendor with a real result behind the number can answer all of this immediately, because the answer already exists somewhere in their own data. A vendor without one gets vague, or pivots to a different, equally unsupported number.
Sample size decides whether the number generalizes
A result from one machine, in one favorable month, is a data point, not a pattern. A result across fourteen machines over six months is something closer to evidence. Ask how many machines, lines, or plants the case study’s number is actually drawn from, and whether that’s the full customer or a hand-picked subset. A vendor that cites results from three machines out of a customer’s forty deployed ones is telling you something, whether they intend to or not: the other thirty seven either didn’t move the number or moved it in the wrong direction.
The same question applies at the customer level for any claim that spans multiple accounts, like “customers see an average 18% improvement.” Averaged across how many customers, weighted how, over what window each. An average across five customers where four improved 5% and one improved 70% tells a very different story than an average across fifty customers who each landed close to that same 18%, and the headline number alone can’t distinguish between them.
The measurement window matters as much as the number
Ask specifically what period the result covers, and what it’s being compared against. A single strong month right after a system went live is the least trustworthy version of a result, because early enthusiasm, a temporary process change, or plain seasonal variation can produce a good month on their own, with or without the monitoring system’s contribution. A comparison spanning six months or more, ideally against the same months a year earlier to control for seasonal effects, is a more solid basis. Ask, too, whether the before period was measured the same way as the after period. A case study comparing an automated after-number against a hand-tracked clipboard before-number is comparing two different instruments, not measuring an improvement.
Ask what changed at the same time
A case study rarely mentions the other things happening at a plant during its measurement window, a new hire, a different shift structure, a process change unrelated to the monitoring system. That’s not necessarily deceptive. Most case studies aren’t built to surface it at all. It’s still worth asking directly: was anything else changing at the plant during this window that could explain some of the result. A vendor willing to name the confounding factors and explain why the result still holds is showing real confidence in the number. A vendor who hasn’t considered the question at all hasn’t done the basic legwork that separates a measured result from a good story.
A short checklist before citing a case study
Ask for the machine or line count the number is drawn from, beyond the customer name attached to it. Ask for the measurement window, both before and after, and whether they’re the same length. Ask what the before number was measured against, an automated system or a hand-tracked one. Ask whether the case study is the customer’s best month or a sustained average. Ask what else changed at that plant during the window. And ask, if the vendor is willing, to talk directly to the customer named in the case study, beyond reading their quote. A vendor confident in a result usually says yes to that last one without hesitation. A vendor who hedges on a reference call is telling you something the case study itself won’t.
A worked comparison
Picture two case studies landing on the same desk. The first: a one-page PDF with a logo, a quote from a plant manager, and “22% reduction in downtime” in large type. Nothing else. The second: the same headline number, plus a footnote stating it covers fourteen CNC machines over a six-month window, compared against the same six months the prior year measured the same way both times, with a note that the plant also added a second shift during that window and the vendor’s own estimate of how much of the improvement that shift change might account for. The second document is less flattering to read at a glance. It’s also the only one of the two a buyer could actually defend in a budget meeting if someone pushed back on the number.
That gap, between a number and a number with its basis attached, is the entire difference this guide is about. A vendor willing to publish the second kind of case study, or at least produce it on request, is telling a buyer something about how the whole relationship is likely to go: numbers that can be checked, not numbers that only sound good.
What this doesn’t mean
None of this means every case study without full methodology is worthless, or that a vendor withholding a customer’s exact numbers is hiding something bad. Plenty of customers won’t let a vendor publish machine counts or specific dollar figures for reasons that have nothing to do with the result, competitive sensitivity, internal policy, a legal review that strips detail before anything goes out the door. The test isn’t whether every number is public in the case study itself. It’s whether the vendor can answer these questions in a conversation, even if the polished document can’t carry the detail. A vendor who won’t or can’t answer in conversation either is the one worth walking away from, not the one whose PDF happens to be short.
Quick recap
- A bare percentage tells you almost nothing: ask what it’s a percentage of and against what baseline
- Ask how many machines, lines, or customers the number is actually drawn from, beyond whose name is on the case study
- Ask for the measurement window on both sides of the comparison, and whether they were measured the same way
- Ask whether the result is a best month or a sustained average over a longer period
- Ask what else changed at the plant during the measurement window that might explain part of the result
- Ask for a reference call with the actual customer. A vendor confident in the result rarely says no