How Many Scenarios Do You Actually Need? | Learn

One of the easiest ways to make a stochastic optimization system look sophisticated is to give it a giant scenario count.

Ten thousand demand paths. Fifty thousand lead-time samples. A million Monte Carlo replications.

That can be useful. It can also be a very expensive way to avoid asking the important question:

Did the additional scenarios change the decision?

If 300 scenarios tell you to order 4,800 units and 10,000 scenarios tell you to order 4,800 units, the extra computation did not improve the decision. It estimated the same answer more precisely.

Those are not the same thing.

This distinction matters in real supply chain systems because scenario generation is rarely free. Scenarios feed simulators, stochastic programs, policy searches, parameter tuning loops, and sometimes large MILPs. Multiplying the scenario count can multiply runtime, memory, model-build time, and cloud cost without creating any measurable operational value.

The goal is not to use as many scenarios as possible.

The goal is to represent enough uncertainty to make a good decision.

Start with the decision

Suppose an importer must choose an order quantity today for a product with a six-month lead time. Demand is uncertain. The decision might be

[ q \in {0, 100, 200, \ldots} ]

because the supplier ships in case packs of 100.

For scenario (s), let demand over the relevant horizon be (D_s). A simple scenario objective might be

[ \max_q \frac{1}{N}\sum_{s=1}^{N} \left[ p\min(q,D_s)

c q
h(q-D_s)^+
b(D_s-q)^+ \right]. ]

Here (N) is the number of scenarios.

It is tempting to treat (N) as a generic accuracy parameter. Bigger must be better.

But the business does not consume (N). It consumes (q).

If the estimated objective moves slightly while the selected order quantity stays at 4,800 units, nothing operational changed. If the estimated objective moves enough that the optimizer switches from 4,800 to 4,900, then the sampling error crossed a decision boundary.

That is the thing we should care about.

Objective convergence is not decision convergence

A common validation exercise is to increase the number of scenarios and plot the estimated expected cost. Eventually the line looks flat, and we declare convergence.

That is useful, but incomplete.

Consider two candidate actions:

order 4,800 units: estimated expected profit $121,040;
order 4,900 units: estimated expected profit $121,020.

The difference is $20.

If Monte Carlo noise around those estimates is $500, the apparent winner is not remotely stable even though the total expected-profit estimate might look very stable on a chart whose y-axis is around $120,000.

Now consider another problem where the best action beats the second-best by $18,000. You may need surprisingly few scenarios to identify the correct action reliably.

The scenario requirement therefore depends on the gap between competing decisions, not just the variance of the outcome.

This is one reason generic rules such as “always use 1,000 scenarios” are weak engineering advice.

Ask these questions before increasing the sample

Before adding scenarios, I want answers to a few practical questions.

What action is being selected? An order quantity? A set of suppliers? A transportation mode? A production schedule? A policy parameter?

How discrete is the action space? If orders move only in pallets of 500 units, tiny improvements in probability estimation may never change the executable decision.

How close are the best alternatives? A nearly tied decision requires more evidence than an obvious one.

Which tail events matter economically? A rare stockout during a critical launch may matter more than hundreds of ordinary demand realizations.

Are scenarios expensive because of sampling or because of evaluation? Generating 100,000 random numbers is cheap. Running 100,000 hour-long simulations is not.

Does the policy react after uncertainty unfolds? A static one-time order and a multi-period recourse policy can require very different scenario structures.

These questions tell you far more than a universal scenario-count recommendation.

Scenario quality matters before scenario quantity

A large sample from the wrong distribution is still wrong.

This sounds obvious, but it is a common failure mode. Teams spend enormous effort reducing Monte Carlo error while ignoring model error.

Imagine that historical supplier lead time has a long right tail because congestion occasionally adds six weeks. Your model replaces it with a narrow normal distribution centered at the historical average. Running one million samples from that distribution does not recover the missing tail.

Likewise, suppose demand and lead time become positively correlated during peak season. Sampling them independently can systematically understate exposure exactly when inventory is most vulnerable.

Before worrying about whether you need 500 or 5,000 scenarios, check whether the scenarios preserve the things that can change the decision:

marginal demand distributions;
correlations across products and locations;
temporal dependence;
promotions and regime changes;
lead-time tails;
supplier outages;
returns or yield uncertainty;
shared shocks such as weather, port congestion, or macro events.

Five hundred structurally credible scenarios can be much more useful than fifty thousand sanitized ones.

Use decision stability as a diagnostic

A simple experiment is extremely informative.

Take several scenario counts, for example

[ N \in {50,100,250,500,1000,2500}. ]

For each (N), generate multiple independent training samples. Solve the decision problem on each sample. Record the selected action and then evaluate every selected action on one large, independent evaluation set.

The output should look something like this:

Training scenarios	Most common order	% choosing same order	Out-of-sample profit gap
50	4,700	42%	$1,840
100	4,800	61%	$710
250	4,800	84%	$160
500	4,800	96%	$35
1,000	4,800	99%	$8
2,500	4,800	100%	$0

If this were my system, I would have a hard time justifying 2,500 training scenarios for every solve. Somewhere around 500 may already produce essentially the same operational decision.

The exact threshold depends on the cost of being wrong and the compute budget. The point is that we measured the thing we actually deploy.

Separate training scenarios from evaluation scenarios

This is one of the most useful practical distinctions.

You may need only 300 scenarios to choose a policy but 50,000 scenarios to measure its expected performance precisely.

There is no contradiction.

Optimization and evaluation are different jobs.

The optimizer needs enough information to rank actions correctly. The evaluator may need a much larger sample to estimate a small economic difference with a useful confidence interval.

This suggests a production architecture like:

generate a moderate scenario set for optimization;
solve or tune the candidate policy;
evaluate the resulting policy on a much larger independent scenario set;
compare it against the incumbent using the same evaluation scenarios;
increase the optimization sample only if the selected policy remains unstable or materially underperforms.

This architecture is usually cheaper than stuffing the full evaluation sample into every optimization run.

Common random numbers are your friend

Suppose policy A and policy B are evaluated on different random demand paths. Some of the measured difference will come from the policies. Some will come from luck.

Instead, evaluate both policies on the same scenarios.

For each scenario (s), calculate

[ \Delta_s = J(A,s)-J(B,s). ]

Then estimate the expected improvement from the distribution of (\Delta_s).

This is a standard variance-reduction idea, but it is especially practical in decision systems. If both policies experience the same demand spike, supplier delay, and capacity outage, the comparison focuses on how the policies responded rather than which one received easier random draws.

When comparing many parameter settings in simulation optimization, keeping scenario seeds aligned can dramatically reduce the noise in the ranking.

Do not optimize on your test set

There is another trap.

You start with 1,000 scenarios. Policy A wins. You tweak the parameters. Policy B wins. You tweak again. Policy C wins. After 200 iterations, you deploy policy Z because it performs beautifully on those same 1,000 scenarios.

Congratulations: you can overfit a simulator too.

The scenarios became a training set even if nobody called them that.

For policy tuning, I prefer a clean separation:

training scenarios for searching parameters;
validation scenarios for model and parameter choices;
final evaluation scenarios that remain untouched until the policy is frozen.

In historical replay, the same idea applies to time. Tune on earlier decision dates and evaluate on later decision dates using only information that would have been available at each decision point.

A stochastic optimizer does not magically remove leakage.

Tail risk changes the sampling problem

Expected value is not always the whole objective.

Suppose the objective includes CVaR, a service penalty triggered only during severe shortages, or a contractual failure cost that occurs in 0.5% of cases.

A random sample of 100 scenarios may contain zero examples of the event you care about.

Increasing the raw scenario count is one solution, but not always the best one. Depending on the application, you may use stratified sampling, importance sampling, explicit stress scenarios, or a mixture of ordinary and tail-focused scenarios.

The important requirement is to keep the probability accounting honest. A hand-picked disaster scenario should not silently receive the same probability weight as an ordinary week unless that is intentional.

For practitioner work, I often want two views:

probability-weighted performance, which tells me how the policy performs under the modeled distribution, and stress performance, which tells me what happens under named adverse conditions.

Do not mash those into one number unless the economics justify it.

Shared constraints can increase scenario sensitivity

Single-item examples are useful, but real supply chains couple decisions.

Suppose a vendor has 500 products sharing a $2 million purchase budget and a container-capacity limit. Each product has uncertain demand. The optimizer is not independently choosing 500 newsvendor quantities. It is allocating scarce capital and logistics capacity across a portfolio.

Now a change in one product’s scenario distribution can alter the opportunity cost of capacity for many other products.

A simplified model might include

[ \sum_i c_i q_i \le B ]

and

[ \sum_i v_i q_i \le C, ]

where (B) is budget and (C) is container capacity.

In this setting, scenario stability should be measured at multiple levels:

total economic value;
selected order quantities;
which constraints bind;
allocation of scarce capacity;
high-value products whose decisions flip frequently;
downstream service and inventory consequences.

A stable total objective can hide unstable individual decisions. That may be acceptable if the alternatives are economically equivalent. It may be unacceptable if operational teams cannot tolerate constant churn.

More scenarios can make the wrong bottleneck worse

Suppose the business requires a decision in two minutes.

Your current pipeline spends:

10 seconds loading data;
35 seconds constructing scenario-dependent features;
55 seconds building the optimization model;
15 seconds solving;
5 seconds writing results.

The solver is not your problem.

Doubling scenarios may increase model construction to 100 seconds and push the system over its SLA even if solve time barely changes.

This is why production profiling matters. Measure scenario generation, preprocessing, model construction, presolve, optimization, simulation, serialization, and downstream writes separately.

If scenario-dependent structures are reusable, cache them. If scenarios can be evaluated independently, parallelize them. If a policy search repeatedly evaluates the same random worlds, generate those worlds once rather than rebuilding them inside every trial.

Optimization runtime is an end-to-end systems problem.

Scenario reduction should preserve decisions, not pictures

Scenario reduction is often framed as finding a smaller set whose statistical distribution resembles the original sample.

That can be useful, but the ultimate test is downstream.

If two scenario sets look statistically similar yet produce different decisions, they are not interchangeable for your application.

Conversely, two samples can look different on generic distribution metrics and still induce the same decision because the differences occur far away from relevant decision boundaries.

A practical reduction procedure should therefore be validated by repeatedly solving the actual decision problem and measuring policy regret out of sample.

The decision model is part of the definition of a good scenario set.

Metrics I would log in production

For each decision run, I would keep enough information to answer whether uncertainty is actually affecting the policy.

At minimum:

scenario-set identifier and random seed;
scenario count;
model/version identifiers;
uncertainty-model version;
selected actions;
objective value;
best-bound and optimality gap where relevant;
runtime broken into generation, build, solve, and evaluation;
binding constraints or important resource utilization;
expected profit or cost on an independent evaluation set;
tail metrics such as shortage probability or CVaR when relevant;
difference from the incumbent policy;
difference from the previous run’s decision.

Without this, teams tend to debate scenario counts philosophically. With it, you can run an experiment.

Failure modes I see repeatedly

Picking a round number and calling it science

“One thousand scenarios is enough” is not a result. It is a configuration value.

Show that the decision stabilizes.

Using more samples to compensate for a bad uncertainty model

Monte Carlo error decreases with more samples. Structural model error does not.

Evaluating candidates on different random worlds

This adds noise to comparisons for no benefit. Reuse scenarios when comparing policies unless independence is specifically required.

Reusing the same scenarios forever

Common random numbers help comparison, but a fixed scenario bank can eventually become a target that the tuning process overfits. Maintain fresh holdout evaluation sets.

Measuring only expected objective

A policy can have similar expected cost but materially different stockout exposure, cash usage, decision churn, or tail loss.

Ignoring discrete decisions

If the executable action moves in MOQ-sized jumps, estimate stability at those jumps. Precision far inside one decision region may have no value.

Sending every scenario into a giant MILP

Sometimes that is appropriate. Sometimes a smaller optimization sample plus large out-of-sample simulation is faster and just as good. Test it.

What I would do in practice

For a new stochastic supply chain decision system, I would not begin by arguing about whether the correct number is 300, 1,000, or 10,000.

I would build a scenario-count experiment.

Choose a representative set of decision states: normal inventory, low inventory, excess inventory, tight capacity, loose capacity, promotion periods, high uncertainty, low uncertainty, and ugly edge cases.

For each state, solve the problem repeatedly across increasing scenario counts and independent seeds. Save the selected actions. Evaluate all candidate actions on a large common holdout sample.

Then answer four questions:

At what sample size do the important decisions stop moving materially?
At what sample size does out-of-sample regret become economically negligible?
Which states require more scenarios than others?
Where is the runtime actually being spent?

You may discover that 250 scenarios are enough for most decisions and 2,000 are useful near a handful of high-value boundaries. Great. That suggests adaptive computation instead of one giant global setting.

You may discover that scenario count barely matters because the MOQ is enormous. Great. Spend the engineering time elsewhere.

You may discover that decisions never stabilize because your scenario model has unstable tails or the top alternatives are genuinely near-tied. Also great. Now you know what problem you actually have.

The point

Scenario count is not a badge of sophistication.

A stochastic model exists to improve a decision under uncertainty. Once the decision is stable and its out-of-sample economic performance is good enough for the application, additional sampling has diminishing value.

Use enough scenarios to resolve the decisions that matter. Use larger independent samples to measure performance when you need precision. Preserve the uncertainty structure that drives the economics. Profile the entire pipeline. And stop spending compute just because a bigger number looks more serious.

The question is not how many scenarios your model can handle.

The question is how many scenarios it takes to make the right decision.