Forecast Vintage Leakage

One of the easiest ways to accidentally build a bad planning system is to give it information from the future.

Most teams do not do this intentionally. The problem usually appears during model development, backtesting, simulation, or policy evaluation. A data scientist builds a forecasting model. An optimization model consumes the forecast. A simulator evaluates the resulting decisions. The numbers look fantastic. Everyone gets excited.

Then the system goes live and the results collapse.

In many cases, the root cause is not the optimization model, the forecast model, or the simulator. The root cause is that the evaluation used information that would not have existed when the decision was actually made.

This is forecast vintage leakage.

The Core Question

For every decision in your system, ask a simple question:

What information was available when the decision was made?

Not what information is available today.

Not what information exists in the data warehouse.

Not what information can be reconstructed after the fact.

What information existed at the exact moment the planner, buyer, scheduler, or system made the decision?

If your evaluation uses anything beyond that information set, your results are inflated.

A Supply Chain Example

Suppose a buyer places purchase orders every Monday.

The lead time is eight weeks.

You want to evaluate a new ordering policy.

Many teams perform the following experiment:

Train a forecast on historical data.
Generate forecasts for every week in history.
Run an inventory simulation.
Compare service and inventory results.

This sounds reasonable.

The problem is that the forecast used in the simulation may not match the forecast that would have existed at the time.

Maybe the forecast model was retrained using future observations.

Maybe promotional information was cleaned after the fact.

Maybe missing demand records were corrected months later.

Maybe product hierarchy mappings changed.

The simulation is now consuming a forecast that never existed operationally.

The policy is effectively seeing the future.

Forecasts Have Versions

A forecast is not a number.

A forecast is a dated artifact.

For a decision made on January 1st, the relevant forecast is the forecast generated on January 1st.

Not the forecast generated on March 1st.

Not the forecast regenerated last week.

Not the forecast produced after two years of model improvements.

Every forecast has a vintage.

If you are evaluating historical decisions, you should know which forecast vintage was available at that time.

The Policy Is What Matters

Most organizations spend enormous effort measuring forecast accuracy.

The business does not operate on forecast accuracy.

The business operates on decisions.

The relevant question is not:

“How accurate was the forecast?”

The relevant question is:

“Would this decision have been available using only information known at decision time?”

A mediocre forecast evaluated correctly is more useful than a perfect forecast evaluated incorrectly.

Questions To Ask

When reviewing a planning system, ask:

What information was available at decision time?
Which forecast version was used?
When was the forecast generated?
When was the decision generated?
When was the order released?
When was demand observed?
When were data corrections applied?
When were master data updates applied?
Are we replaying history or reconstructing it?

These questions often reveal hidden leakage.

Decision Variables

In many supply chain systems the decision variables are straightforward:

Purchase quantities
Transfer quantities
Production quantities
Replenishment triggers
Order timing
Inventory targets

The challenge is rarely the decision variables.

The challenge is ensuring that every decision is generated using the information set that existed at the time.

Simulation Failure Modes

Several common mistakes appear repeatedly.

Rebuilt Forecasts

Teams regenerate historical forecasts using the latest forecasting model.

The resulting forecast quality is far better than what planners actually had.

Corrected Demand

Historical demand data often gets cleaned months later.

Using corrected demand during replay introduces future information.

Updated Lead Times

Lead time estimates frequently improve over time.

Using modern lead time estimates to evaluate historical decisions creates bias.

Revised Product Mappings

Product hierarchies, substitutions, and assortment mappings evolve.

The current mapping may not match the historical reality.

Metrics That Matter

Instead of only measuring forecast quality, track:

Profit
Inventory investment
Service outcomes
Stockout cost
Obsolescence cost
Decision stability
Capital efficiency
Forecast vintage coverage

A policy should succeed using information that would have been available operationally.

Practical Implementation

The cleanest approach is maintaining historical forecast snapshots.

Store:

Forecast creation timestamp
Forecast horizon
Forecast values
Model version
Feature version

When running a replay simulation, retrieve the forecast snapshot that would have existed at the decision date.

Treat forecasts as first-class operational records rather than temporary model outputs.

This requires more storage.

It also produces far more credible evaluations.

What To Do In Practice

If your simulator, optimizer, or policy evaluation looks surprisingly good, assume nothing.

Verify the information timeline.

Draw the sequence on a whiteboard.

Write down exactly when each forecast was created, when each decision was made, when each order was placed, and when each observation became available.

Most decision systems fail because of information mistakes rather than mathematical mistakes.

A replay that respects information availability is usually less impressive.

It is also far closer to reality.