Forecast Bias Is Not Symmetric

A forecast that is 20 units too high and a forecast that is 20 units too low have the same absolute error.

They usually do not have the same business consequence.

That sounds obvious, but a surprising amount of forecasting work still evaluates models as if the error itself were the final product. Teams compare MAE, RMSE, MAPE, WAPE, bias, and a dozen variations. The model with the cleaner scorecard wins. Then the forecast gets handed to an ordering system where an error becomes inventory, a stockout, an expedite, a missed MOQ, or a purchase that cannot be undone.

The forecast is not the decision. Its value depends on what the decision system does with it.

This is where forecast bias becomes more interesting than a statistical diagnostic.

Start With the Decision

Suppose every Monday you decide how much to order from a supplier.

Let:

(q_t) be the quantity ordered in week (t),
(I_t) be inventory available at the start of the week,
(D_t) be uncertain demand,
(L_t) be uncertain supplier lead time,
(p) be unit selling margin,
(c) be purchase cost,
(h) be inventory holding cost,
(s) be the economic consequence of an unmet unit of demand.

A basic inventory transition might look like:

[ I_{t+1} = I_t + A_t - S_t ]

where (A_t) is the quantity arriving from previous orders and (S_t) is sales or fulfilled demand.

The forecast affects (q_t), but the cost of forecast error appears later through the state trajectory.

An overforecast may create inventory that sits for six months. An underforecast may create no economic loss at all if there was already enough inventory in the pipeline. The same underforecast may be catastrophic for another item with a 20-week lead time and no substitute.

So the useful question is not:

How wrong was the forecast?

It is:

What decision did the forecast cause, and what did that decision cost us?

Statistical Symmetry Is Not Economic Symmetry

Consider a product with expected demand of 100 units.

Forecast A says 120. Forecast B says 80. Actual demand is 100.

Both forecasts miss by 20 units.

From an MAE perspective, they are identical.

Now suppose the product has a 40% gross margin, a 16-week replenishment lead time, and customers disappear when the product is unavailable. Underforecasting may cause several weeks of lost contribution margin before replacement inventory can arrive.

Overforecasting may instead create 20 extra units that remain saleable and eventually clear at full price.

The economic loss functions are completely different.

For another product, reverse the situation. Imagine seasonal merchandise that becomes nearly worthless after December. Overforecasting creates markdowns and disposal. Underforecasting merely means a few missed low-margin sales.

There is no universal direction in which bias is bad.

There is only a relationship between forecast error and the downstream decision economics.

Bias Should Be Conditional

A single portfolio-level bias number hides most of what matters.

Suppose your forecasting dashboard says the model has +1.5% bias. That may sound harmless. But perhaps the model systematically overforecasts slow-moving products and underforecasts high-margin products. The errors cancel statistically while reinforcing economically.

At minimum, inspect bias conditional on variables that change the cost of error:

lead-time bucket,
margin or contribution value,
lifecycle stage,
inventory position,
demand velocity,
intermittency,
substitutability,
perishability or obsolescence risk,
supplier flexibility,
order cadence,
MOQ structure.

You are looking for regions where the forecasting system is systematically wrong in a direction that matters to the decision.

A 10% underforecast on an item with one week of supply risk is not the same event as a 10% underforecast on an item whose next purchase opportunity is four months away.

Forecast Error Must Be Evaluated Against State

This is one of the easiest mistakes to make in supply-chain analytics.

Teams join forecast and actual demand, calculate error, and assume the error tells them how the business was affected.

It does not.

The impact depends on the state when the forecast was consumed.

If an item already had 12 weeks of inventory, an underforecast may not have changed the order at all. If the supplier MOQ forced the same order quantity under either forecast, the error had zero immediate decision impact. If capacity was binding, the forecast may have changed which SKU received scarce supply rather than changing total purchasing.

This suggests a better diagnostic.

For each historical planning run, reconstruct:

the state visible at decision time,
the forecast vintage visible at decision time,
the action actually taken,
the feasible alternative actions,
what happened afterward.

Then ask whether a different forecast would actually have produced a different action.

Forecast errors that do not change decisions are often much less important than dashboards imply.

The Decision Boundary Matters

Imagine an order policy:

[ q_t = \max(0, S - IP_t) ]

where (S) is a target position and (IP_t) is inventory position.

A small forecast change far away from the ordering threshold may do nothing. The exact same change near the threshold may trigger an entire purchase order.

Now add an MOQ:

[ q_t \in {0} \cup [M, \infty) ]

where (M) is the minimum order quantity.

A forecast increase of one unit can theoretically move the recommendation from zero to an MOQ of 1,000 units.

The forecast moved by one. The decision moved by 1,000.

This is why prediction metrics alone are especially weak for discrete decision systems. The mapping from prediction error to economic consequence can be nonlinear, discontinuous, and coupled across products.

Coupling Makes Bias More Dangerous

Real supply chains rarely optimize one SKU independently.

Products share:

supplier MOQs,
containers,
warehouse capacity,
cash budgets,
production lines,
transportation lanes,
shelf space,
labor,
purchasing calendars.

Suppose ten products compete for a constrained container. If one product is systematically overforecast, it may steal capacity from nine correctly forecast products.

The direct inventory cost of the biased forecast is only part of the damage. The opportunity cost imposed on the other decisions may be larger.

This is a strong reason to evaluate forecasts through the actual optimization or policy layer rather than scoring each forecast series independently.

Uncertainty Matters More Than Point Bias

Bias is only one feature of a forecast distribution.

Two forecasting systems can have the same mean prediction and completely different uncertainty.

Suppose both predict 100 units of demand.

Model A implies demand is usually between 95 and 105.

Model B implies demand could plausibly range from 30 to 220.

A planning system should not necessarily make the same decision under both.

If your optimizer consumes only the point forecast, it has no way to distinguish them.

This is one reason probabilistic forecasting is useful in decision systems. Instead of passing a single number downstream, represent a distribution or a collection of plausible demand paths:

[ D^{(1)}, D^{(2)}, \ldots, D^{(N)} ]

Then evaluate candidate actions across those futures.

The objective might be expected economic value:

[ \max_q \frac{1}{N}\sum_{n=1}^{N} \Pi(q, D^{(n)}, L^{(n)}) ]

where (\Pi) includes margin, purchase cost, holding cost, stockout consequences, expedites, markdowns, and other relevant economics.

Now forecast uncertainty is being translated into a decision rather than compressed into a safety factor after the fact.

Do Not Manufacture Asymmetric Forecasts to Fix the Optimizer

A common response to asymmetric costs is to intentionally bias the forecast.

The logic sounds reasonable: stockouts are expensive, so increase the forecast by 10%.

This mixes two different jobs.

The forecast should describe what you believe demand will be. The decision model should describe what you should do given those beliefs and the economics.

If shortage is expensive, encode shortage economics in the decision model. If the business wants a conservative tail-risk posture, encode risk aversion. If an SLA is contractual, represent it as a constraint.

Artificially inflating the forecast can work as a heuristic, but it makes the system harder to reason about. When conditions change, nobody knows whether the 10% buffer represents forecast calibration, risk preference, service policy, or an old workaround that somebody added three years ago.

Keep beliefs and preferences separate whenever you can.

A Better Evaluation Experiment

If you have a simulator or historical replay environment, compare forecasting models through the decisions they produce.

For each historical decision date:

reconstruct only the information available on that date,
generate forecasts from each candidate model,
feed each forecast into the same decision policy,
simulate or replay the resulting decisions,
measure the economic outcome.

This gives you something closer to decision value.

Useful metrics might include:

contribution margin,
lost sales value,
holding cost,
markdown cost,
expedites,
cancellation cost,
average inventory,
inventory age,
fill rate where operationally relevant,
cash consumed,
constraint violations,
decision volatility.

Keep MAE and bias. They are useful diagnostics. Just do not confuse them with the objective.

Measure the Value of Forecast Improvements at the Margin

Suppose a new forecasting model improves WAPE from 24% to 21%.

Is that good?

Statistically, yes.

Operationally, you still do not know.

Perhaps nearly all of the improvement occurred on items where inventory was abundant and orders would have been unchanged. Meanwhile, the new model got slightly worse on the handful of constrained, high-margin products where forecasts actually change purchasing decisions.

A useful experiment is to estimate the marginal decision value of forecast improvements.

Take the baseline forecast and selectively replace portions with the improved forecast. Replay the decision system. Measure where economic value changes.

This helps answer questions like:

Which products benefit from better forecasts?
Which horizons matter most?
Does improving tail calibration matter more than improving the mean?
Does lead-time uncertainty dominate demand uncertainty?
Are we spending modeling effort where the decision is insensitive anyway?

Forecasting resources should follow decision sensitivity, not merely statistical difficulty.

Common Failure Modes

Optimizing a forecasting metric in isolation

A model wins the forecasting benchmark but produces worse inventory decisions because the errors it introduces are economically expensive.

Looking only at average bias

Positive and negative bias across products cancel. The portfolio looks calibrated while important segments are systematically wrong.

Ignoring forecast vintage

Historical evaluation uses forecasts regenerated with information unavailable at the original decision date. The backtest becomes unrealistically clean.

Treating all units equally

A one-unit error on a $2 commodity is counted the same as a one-unit error on a constrained high-margin item.

Using forecast buffers as business policy

Teams inflate forecasts to achieve service goals instead of representing shortage economics or constraints in the decision layer.

Evaluating SKUs independently when decisions are coupled

The forecast looks harmless at the item level but changes allocation of shared capacity, budget, or MOQ commitments.

Ignoring decision thresholds

Small errors near discrete order boundaries can cause large operational changes.

What to Build in Practice

A mature forecasting evaluation stack should connect four layers.

Forecast layer. Store point forecasts, distributions or quantiles, model version, forecast vintage, and calibration diagnostics.

State layer. Preserve the inventory, pipeline, supplier status, constraints, prices, costs, and other information visible when the decision was made.

Decision layer. Record the policy or optimization model, parameters, feasible alternatives, recommended action, and final executed action.

Outcome layer. Measure what happened economically and operationally after the decision.

This structure lets you move beyond “the forecast was wrong” toward a much more useful diagnosis:

The forecast was low in a region where the ordering policy was sensitive, which delayed an MOQ-triggering purchase by one cycle and created three weeks of lost high-margin demand.

That is actionable.

Questions I Would Ask Before Tuning the Forecast

Before spending another quarter squeezing a few points out of forecast accuracy, ask:

What decision consumes this forecast?
At what horizon does the decision become committed?
Which direction of error is economically more expensive?
Does that asymmetry change by product or state?
How often does forecast error actually change the action?
Are there discrete thresholds such as MOQs or pack sizes?
Are products coupled through shared constraints?
Is lead-time uncertainty larger than demand uncertainty?
Are shortage and overage costs explicitly represented?
Are we evaluating historical forecast vintages correctly?
What economic metric would tell us whether a forecast improvement mattered?

If the team cannot answer those questions, another forecasting model is probably premature.

The Practical Takeaway

Forecast accuracy matters. Forecast bias matters. Calibration matters.

But none of them matter independently of the decision.

The same forecast error can be irrelevant in one state and enormously expensive in another. The same statistical bias can help one product and hurt another. A tiny prediction change can trigger a massive discrete order, while a large prediction error can leave the recommended action untouched.

So evaluate forecasts where they actually create value: inside the decision process.

Build the forecast to represent uncertainty as faithfully as practical. Build the decision model to translate that uncertainty into economics. Then judge the system by the quality of the decisions it produces.

That is the level where forecasting becomes useful.