ML Predictions Are Random Variables, Not Truth

Machine learning models usually enter business systems as if they are facts. A model predicts demand, conversion, cancellation, lead time, fraud risk, or customer response. Then another system takes that number and acts as if the future has been revealed.

That is the wrong mental model.

An ML prediction is not truth. It is a statistical estimate produced by a model trained on historical data under assumptions that may or may not hold tomorrow. It has error, bias, drift, uncertainty, and blind spots. When that prediction feeds an optimization model or decision policy, the error does not disappear. It gets transformed into business consequences.

The prediction is an input, not the objective

The business does not get paid for predicting demand. It gets paid for making better decisions with demand uncertainty. The same is true for lead times, prices, cancellations, returns, and customer behavior. The prediction matters because it supports an action.

That means the decision system should not simply consume the point prediction. It should ask: what uncertainty remains, how costly is the error, and what action should we take given the distribution of possible outcomes?

A point prediction can be useful, but it is rarely enough. If two products both have predicted demand of 100 units, but one has tight uncertainty and the other has extreme volatility, those are not the same decision. If underpredicting one product is much more expensive than overpredicting it, the mean is not the right target. If lead time uncertainty is skewed, a symmetric error metric can hide the operational risk.

Error propagates through optimization

When ML predictions feed optimization models, prediction error becomes decision error. This is where many systems fail quietly.

A forecast model may improve average accuracy, but the optimization model may be sensitive to errors in only a few places. A demand error on a constrained item can change allocations across an entire network. A lead-time error can trigger stockouts, expediting, or excess safety stock. A cancellation prediction can change staffing, capacity, and service. The downstream decision determines which prediction errors matter most.

This is why ML models should be evaluated through the decisions they support. The right question is not only “how accurate was the model?” The right question is “did the model improve the action?”

Treat predictions as distributions when possible

A better approach is to represent ML outputs as uncertain quantities. Instead of passing one number downstream, pass a distribution, scenarios, quantiles, or calibrated uncertainty intervals. This lets the decision layer reason about risk.

For example, an inventory system can use demand scenarios rather than a single forecast. A staffing model can use probabilistic arrival rates. A pricing system can model uncertain conversion response. A routing system can model travel-time distributions. The optimizer or policy can then choose actions that reflect the economic asymmetry of being wrong.

This does not mean every system needs a complicated probabilistic model. It means the decision layer should not pretend uncertainty vanished just because the prediction has a decimal point.

The decision metric should drive model evaluation

If the decision is asymmetric, the model evaluation should reflect that asymmetry. If stockouts are more expensive than overstock, then underprediction may deserve heavier penalty. If capacity violations are catastrophic, then tail risk may matter more than average error. If the decision only changes when the prediction crosses a threshold, calibration near that threshold may matter more than global accuracy.

The model should be judged by marginal decision value. Does it change the action in useful ways? Does it reduce regret? Does it improve expected profit, service, utilization, or risk? Does it make the policy more robust?

Those are decision questions, not just ML leaderboard questions.

Close the loop

A mature decision system closes the loop. Predictions feed actions. Actions create outcomes. Outcomes update models, policies, and assumptions. The system learns where prediction errors matter, which uncertainties drive cost, and which decisions are sensitive to model drift.

Without that loop, the ML model becomes a fragile oracle. With the loop, ML becomes one component in a decision engine.

The practitioner takeaway

Do not put ML predictions on a pedestal. Use them, but use them honestly.

A prediction is a random variable with business consequences. Treat it as an uncertain input. Evaluate it through the decision it supports. Pass uncertainty forward when it matters. Measure whether the resulting action improves economics under realistic scenarios.

The goal is not better predictions in isolation. The goal is better decisions.