Scenario Count Is Not Model Quality

A planning team runs 50 demand scenarios and nobody trusts the answer. So they run 300. Then 1,000. Then 10,000. The runtime explodes, the batch jobs get fragile, and the business still asks why the recommendation changed from yesterday.

More scenarios can reduce sampling noise. They do not automatically create a better decision.

The number of scenarios is one of the easiest technical knobs to understand and one of the easiest to abuse. It feels scientific because it is measurable. It gives everyone a comforting story: if the result is noisy, increase the sample count. But in real supply chain optimization, decision quality depends just as much on scenario design, dependency structure, policy representation, objective economics, and variance control.

A bad simulator with 10,000 scenarios is still a bad simulator. A poorly framed decision with perfect Monte Carlo precision is still the wrong decision.

Ask what the scenarios are for

Before arguing about sample count, ask what role scenarios play in the system.

Are scenarios being used to evaluate one fixed policy? Compare a small set of candidate policies? Optimize directly over decisions? Tune policy parameters? Estimate tail risk? Stress-test constraints? Explain outcomes to planners? Each use case has a different compute requirement.

If the goal is to compare two policies that differ by millions of dollars in expected value, a small scenario set may be enough. If the goal is to choose between two policies that are economically close, even thousands of scenarios may not resolve the ranking unless the evaluation is designed carefully.

The decision question matters more than the sample count. A team should be able to say: “we need enough simulation accuracy to reliably choose between policies that differ by at least this much economic value.” Without that threshold, scenario count becomes a superstition.

Demand paths are not just random numbers

A scenario is not merely a row of random draws. It is a coherent possible future.

For inventory, that future may include demand by week, lead time, supplier reliability, returns, substitutions, price changes, promotions, capacity, transportation delays, and receipt timing. The joint behavior matters. Weekly demand draws that are independent may have the right marginal distribution and the wrong business behavior. Lead times sampled independently of vendor disruption may miss the exact risk the model is supposed to manage.

A demand path should preserve the dependence structure that affects the decision. If high demand this week makes high demand next week more likely, the scenario generator should reflect that. If port congestion affects multiple vendors at the same time, the scenario generator should not pretend each lane fails independently. If a promotion lifts demand across related products, the scenario should move those products together.

A larger number of incoherent scenarios does not fix incoherence. It only makes the wrong world more precisely wrong.

The policy determines the needed precision

A one-shot order decision and a recourse policy require different simulation design.

If the decision is simply “buy this quantity today,” scenarios evaluate how that quantity performs across possible futures. If the decision is a policy, such as “reorder when projected inventory falls below a threshold,” the simulation must include future state transitions and future policy actions. The same demand realization may trigger different future decisions depending on inventory, open orders, capacity, and lead time.

This distinction matters because many teams evaluate a policy as if it were a static plan. They simulate demand against a frozen order plan and conclude the plan performs poorly. Or they allow future replanning in the simulator but compare it to a deterministic optimization that did not have the same recourse. Either way, the scenario count is not the problem. The evaluation structure is.

Before adding scenarios, define the policy structure:

[ A_t = \pi(S_t, I_t) ]

where (S_t) is the state of the system, (I_t) is the information available at time (t), and (A_t) is the action taken by the policy. A scenario should drive the state forward so the policy can be evaluated as it would actually operate.

Common random numbers beat brute force

When comparing candidate policies, use common random numbers whenever possible. That means each policy is evaluated against the same underlying scenario futures.

This sounds like a technical detail, but it is often the difference between useful and useless simulation results. If policy A and policy B are evaluated on different random futures, the measured difference includes both policy performance and scenario noise. If they are evaluated on the same futures, much of the noise cancels out, and the comparison becomes more stable.

This is especially important in supply chain systems where policy differences are small relative to demand uncertainty. You do not need to estimate each policy’s absolute expected value with extreme precision. You need to estimate the difference between policies well enough to choose.

That is the practical goal: reduce decision noise, not decorate the simulation with a heroic sample count.

Adaptive sampling

Not every candidate policy deserves the same number of scenarios.

A weak candidate can often be eliminated quickly. A strong candidate deserves more evaluation. Two near-tied candidates may need additional scenarios, stress tests, or decision-specific diagnostics. Spending the same simulation budget on every candidate is usually wasteful.

A practical workflow is staged:

Evaluate many candidates with a small shared scenario set.
Eliminate policies that are clearly dominated.
Re-evaluate the remaining policies with a larger scenario set.
Use common random numbers for pairwise comparisons.
Stress-test the finalists on tail scenarios and operational edge cases.

The point is not to chase theoretical purity. The point is to spend compute where it can change the decision.

Constraints under uncertainty

Scenario count also interacts with constraints. A capacity constraint may be deterministic. A service promise may be probabilistic. A cash constraint may be enforced on the planned purchase order but evaluated under uncertain receipts, cancellations, returns, and demand. A truckload constraint may be hard in the execution system but soft in a planning simulator.

You need to decide which constraints must hold in every scenario, which constraints are expected-value constraints, which constraints are chance constraints, and which constraints are penalties in the objective.

For example, “never exceed warehouse capacity” is different from “keep the probability of exceeding warehouse capacity below 5%” and different again from “penalize overflow units at an estimated cost.” The scenario count required depends on which statement the business actually means.

If the constraint logic is vague, more scenarios will not help. You will simply compute a vague policy more slowly.

Metrics to track

A serious simulation system should track more than average objective value.

Track standard error of policy value, confidence intervals for policy differences, ranking stability, probability of regret, tail outcomes, constraint violation rates, time to first good incumbent, scenario generation time, evaluation time per candidate, and sensitivity to random seed. Track how often the chosen policy changes when the scenario set changes.

That last metric is extremely practical. If the selected policy changes every time the seed changes, the decision is not robust. The answer may still be acceptable if all candidates are economically close, but then the system should say that. It should not present one policy as mathematically superior when the difference is simulation noise.

Failure modes

The most common failure mode is using more scenarios to hide a bad scenario generator. If the generator ignores dependence, censoring, seasonality, lead-time correlation, or shared disruptions, sample count becomes a distraction.

Another failure mode is optimizing on a small scenario set and validating on nothing else. The model can overfit simulation noise just like a machine learning model can overfit training data.

A third failure mode is reporting expected value without uncertainty bands. This invites executives to compare numbers that are not meaningfully different.

A fourth failure mode is evaluating policies on different scenario sets. This turns policy comparison into a random-number contest.

A fifth failure mode is using one scenario count for every decision. A daily replenishment adjustment, a network redesign, and a long-horizon inventory bet do not deserve the same stochastic budget.

What to do in practice

Start by defining the decision tolerance. How much expected value must separate two policies before the business cares? Then design the simulation to distinguish differences at that scale.

Use common random numbers. Preserve the dependencies that matter. Separate policy search from policy validation. Use adaptive sampling. Report uncertainty around policy value, not just point estimates. Stress-test final candidates against scenarios the average-case simulator may underweight.

Scenario count is a tool. It is not a certificate of quality. The real standard is whether the stochastic system helps the business choose better actions under uncertainty.