A vendor demo is built to look good. Clean data, a machine that’s clearly running, a dashboard that updates while you watch. None of that tells you what happens on your floor, with your oldest machine, on the week your shift schedule falls apart because three people called in sick. The questions that actually matter are the ones that surface what the system does when conditions aren’t clean, and the best time to ask them is before you sign, not three months into a pilot when the gaps start showing up on their own.
How does it handle a shop with no formal shift schedule?
Plenty of job shops run shifts nobody’s ever entered into a system, an informal day and night split everyone just knows. Ask what happens to production data outside a defined shift window. Does it get dropped from shift comparisons without a flag? Guessed into whichever shift is nearest? Or does the system tell you plainly that a chunk of the window has no shift assigned, and let you fix that before trusting the comparison?
The wrong answer here is confident and vague, something like “it handles that fine.” The right answer is specific: here’s exactly what a shift comparison shows when part of the window is unassigned, and here’s the number that tells you how much of it is missing.
What happens to a machine it can’t measure yet?
Every floor has one, the twenty year old manual mill nobody’s gotten around to instrumenting, or a machine whose controller doesn’t expose the data a connector needs. Ask directly what the dashboard shows for that machine. A system that fills the gap with a made-up number, a default 100% because nothing’s reporting downtime, or a flat zero because nothing’s reporting output, misleads you in a way that’s easy to miss until a decision gets made on a number that was never real. The right answer is a visible gap. A dash, a “not yet measured” label, anything that admits the absence instead of papering over it. You want a system that tells you what it doesn’t know as clearly as what it does.
Does the trial need your IT department?
Before you commit a pilot start date, find out exactly what the connection requires. Does a machine’s controller need an inbound firewall rule opened. Does it need a VPN tunnel maintained for the length of the trial. Does something get installed directly on the control itself, and if so, does that require a maintenance window and a change ticket. Each of these is a dependency outside your control, and each one is a place a pilot can stall for weeks waiting on someone else’s calendar.
A vendor who can answer this with a specific list of what’s needed, an Ethernet drop, a static IP, a box that sits on the network reading the machine, is telling you the pilot can start on your schedule. A vendor who says “we’ll work with your IT team to figure it out” is telling you it can’t.
Read-only, or does it write to your controllers?
This is worth asking twice, because the answer changes what the vendor’s box actually is. A system that reads only, connecting as a client and pulling data without ever sending anything back, has a bounded worst case: if it can’t reach the machine, you get a gap in a chart. A system that writes to a PLC or CNC, adjusting parameters, pushing programs, controlling anything, has turned itself into a component of your machine. That’s not automatically wrong, some integrations actually need it, but it changes the risk conversation completely, and it deserves a specific answer, not a reassurance.
Ask exactly what gets written, and why, and what happens if that write is wrong. Spall’s own answer here, for what it’s worth as one data point among however many vendors you’re evaluating: the connection is read-only to a customer’s controllers, full stop, no code path writes to a PLC or CNC. That’s not a claim every vendor can make, which is exactly why it’s worth making them say it plainly instead of letting “we integrate deeply with your controls” pass as a selling point without anyone asking what “deeply” means.
Where does your data live, and can you get it out?
A dashboard is not a data strategy. Ask whether you can pull your own numbers as a CSV export, whether there’s an API, or whether the only way to see your own production data is logging into their interface and looking at it. If the answer is only the dashboard, you’ve bought a window, not ownership of what’s behind it. Ask what happens to your historical data if you ever leave, and get that answer while you still have negotiating power to ask it, not after you’re three years and a database’s worth of history into the relationship.
What happens when the vendor is wrong?
Every system gets something wrong eventually, a sensor drifts, a threshold gets set too aggressively, a mapping breaks after a controller firmware update. The question isn’t whether that happens, it’s what happens next. Ask how corrections work when you flag a number that doesn’t look right. Is there a support path with an actual response time, or does a wrong number just sit there until someone happens to notice it again. A vendor that’s thought about this has a clear answer. A vendor that hasn’t will change the subject back to the dashboard.
A bad number nobody’s watching for is worse than no number at all, because a missing number gets treated with appropriate suspicion and a wrong number that looks confident gets trusted right up until it costs you a real decision. Ask, too, whether a correction is retroactive. If a threshold was set wrong for two weeks before anyone caught it, does the historical report stay wrong forever without anyone flagging it, or does the vendor tell you plainly which numbers in your history were affected.
The answer to watch for
Across all of these, the pattern worth noticing is the same. A vendor who’s actually built for accurate measurement answers with specifics, here’s exactly what the chart shows, here’s exactly what gets written and what doesn’t, here’s exactly how an export works. A vendor who hasn’t thought hard about this answers with reassurance, “don’t worry about that,” “we handle edge cases,” “our customers love it.” Reassurance is cheap. A specific answer to the read-only question, the unassigned-shift question, or the missing-data question is the tell that separates a system built to survive contact with a real floor from one that was only ever tested on a clean demo.
Quick recap
- Ask what happens to production data with no shift schedule assigned to it
- Ask what a machine it can’t measure yet actually shows, a gap or a made-up number
- Ask what the trial needs from IT before you commit a start date
- Ask read-only or write access, and get specific about what gets written and why
- Ask whether you can export your own data or only view it in their dashboard
- Ask what the correction process looks like when a number turns out to be wrong