Separate Search From Evaluation | Learn

A surprisingly common optimization workflow looks like this:

Generate 500 demand scenarios.
Search thousands of parameter combinations against those scenarios.
Pick the combination with the highest simulated profit.
Report that simulated profit as evidence that the policy is good.

That is not a clean evaluation.

You used the same simulated futures to choose the policy and to judge it.

The optimizer has effectively been allowed to study the exam.

This problem is easy to miss because there may be no machine-learning model anywhere in the system. You might be tuning reorder points, planning horizons, safety factors, allocation weights, expedite thresholds, or parameters inside a cost function approximation. The search algorithm might be differential evolution, Bayesian optimization, a grid search, a hill climb, or a person changing cells in Excel.

It does not matter.

If you repeatedly search against a finite set of uncertain outcomes, you can overfit those outcomes.

For practitioners building simulation-based supply chain policies, separating search from evaluation is one of the cheapest ways to avoid fooling yourself.

Start With the Decision, Not the Optimizer

Before discussing train/test splits or Monte Carlo sample sizes, define what the system actually controls.

Suppose every Monday you decide how much to order from a supplier.

A simple policy might be

[ q_t = \pi(S_t; \theta) ]

where:

(S_t) is the state observed at decision time,
(q_t) is the executable order quantity,
(\pi) is the ordering policy,
(\theta) is a vector of tunable policy parameters.

The state might include:

on-hand inventory,
open purchase orders,
expected arrival dates,
current demand information,
supplier availability,
remaining budget,
storage capacity,
MOQ status,
product lifecycle state.

The policy parameters might include:

a reorder threshold,
a target coverage horizon,
an underage penalty,
an overage penalty,
an expedite trigger,
a risk-aversion coefficient,
a transfer threshold.

The first question is therefore not:

Which optimization algorithm should we use?

It is:

What decision rule are we tuning, what information will it have when it acts, and what economic outcome are we trying to improve?

That framing prevents a lot of sophisticated nonsense later.

The Basic Simulation Problem

Assume a candidate policy (\theta) is evaluated across scenarios (\omega_1,\ldots,\omega_N).

Its estimated value is

[ \hat{J}(\theta) = \frac{1}{N}\sum_{s=1}^{N} J(\theta,\omega_s) ]

where (J) could represent profit after purchase cost, holding cost, lost sales, expediting, disposal, transportation, and other relevant economics.

You then solve approximately

[ \theta^* = \arg\max_{\theta \in \Theta} \hat{J}(\theta). ]

There is nothing wrong with this as a search procedure.

The problem begins when you say:

(\hat{J}(\theta^*)) is our estimate of how well the policy will perform.

It is usually optimistic.

Why? Because (\theta^*) was selected precisely because it looked unusually good on those scenarios.

The Winner’s Curse Exists in Simulation Too

Imagine two policies are economically almost identical in the real world.

On a finite Monte Carlo sample, one may look better simply because the sampled demand paths happened to favor it.

Now compare 10 policies.

Then 1,000.

Then let Bayesian optimization intelligently search 20,000 possible parameter combinations.

The more aggressively you search, the more opportunities you create to find a policy that exploits random peculiarities in the simulation sample.

This is the simulation equivalent of multiple comparisons.

A powerful optimizer can make the problem worse, not better, because it is better at finding whatever signal exists in the objective—including noise.

This is why I do not trust a statement like:

Bayesian optimization found a policy worth $4.2 million more.

My next question is:

Worth $4.2 million more on which scenarios?

Use Three Different Jobs for Scenarios

A practical architecture separates scenarios into three roles.

1. Search scenarios

These scenarios exist to help the optimization algorithm navigate the policy space.

They can be relatively cheap.

You may use 200, 500, or 1,000 depending on simulation cost and noise.

The exact number is an engineering choice. The important point is that these scenarios are allowed to influence the selected policy.

2. Validation scenarios

These scenarios help you make modeling and tuning decisions during development.

For example, you might use them to choose:

search bounds,
optimizer settings,
policy structure,
regularization or stability penalties,
number of search scenarios,
whether another optimization round is worthwhile.

Once you repeatedly inspect validation results and change the system, the validation set is no longer truly untouched. That is fine. Its job is development.

3. Confirmation scenarios

These should be generated independently and used only after the policy is selected.

This is where you estimate the policy’s actual out-of-sample economic performance.

If simulation is cheap, confirmation might use 5,000 or 20,000 scenarios even if search used only 500.

Search needs enough signal to locate good decisions.

Confirmation needs enough precision to tell the business whether the improvement is real.

Those are different jobs.

Search Cheap, Confirm Expensively

This asymmetry is useful in real systems.

Suppose one policy evaluation with 10,000 scenarios takes 30 minutes.

Running 2,000 candidate policies that way would be absurd.

Instead:

search with 300 scenarios,
retain the best few policies,
validate them with 1,000 independent scenarios,
confirm the finalist with 10,000 independent scenarios.

You are spending compute where it changes the decision.

This is usually much better than giving every bad candidate an expensive high-precision estimate.

Common Random Numbers Are Still Useful

There is an important nuance.

During search, I often want candidate policies evaluated on the same scenarios.

That is intentional.

Suppose policy A and policy B are evaluated on unrelated Monte Carlo draws. Some of the measured difference may simply come from A receiving easier futures.

Instead, evaluate both policies on the same scenario seeds:

[ \Delta_s = J(\theta_A,\omega_s)-J(\theta_B,\omega_s). ]

Then estimate

[ \hat{\Delta} = \frac{1}{N}\sum_s \Delta_s. ]

This common-random-number design often reduces the variance of the comparison dramatically.

But do not confuse two ideas:

reuse scenarios across candidates inside search to make comparisons cleaner;
reuse search scenarios for final evaluation, which contaminates the evaluation.

The first is good experimental design.

The second is the problem this article is about.

The Policy Must Be Evaluated as a Policy

Another common mistake is to evaluate a decision with information that would not have existed when the decision was made.

Suppose the policy orders every week.

At week (t), it should see the state and information available at week (t), choose (q_t), experience the next realization, update the state, and decide again.

That means the simulation should look roughly like:

for scenario in scenarios:
    initialize state

    for t in planning_periods:
        information = information_available_at(t)
        action = policy(state, information, theta)
        enforce_or_repair_executable_constraints(action)
        realization = scenario.realization_at(t)
        state = transition(state, action, realization)
        economics += score_period(state, action, realization)

Do not hand the policy the realized demand for the next 26 weeks and then call the result a forecast-driven ordering policy.

That is hindsight optimization.

It can be useful as an upper bound, but it is not a production benchmark.

Respect Forecast Vintage

This matters especially when historical replay is used instead of synthetic scenarios.

If you are evaluating what the policy would have done on March 1, 2025, the policy should receive the forecast that actually existed on March 1, 2025—not today’s reconstructed forecast for that historical period.

Otherwise you leak future information backward through the forecasting system.

Store forecast vintages.

At minimum, your evaluation data should identify:

forecast creation timestamp,
decision timestamp,
target period,
source data cutoff,
model version,
scenario-generation version.

A beautiful simulator with bad temporal discipline is still a bad experiment.

Constraints Must Exist in Evaluation Too

Suppose your candidate policy recommends quantities (q_{i,t}), but real orders must satisfy:

[ q_{i,t} = p_i k_{i,t}, \qquad k_{i,t}\in\mathbb{Z}_+ ]

for case pack (p_i).

A vendor may also impose

[ \sum_i c_i q_{i,t} \ge M y_t ]

when an order is placed, where (M) is a minimum order value.

You may have shared capacity

[ \sum_i v_i q_{i,t} \le C_t, ]

budget

[ \sum_i c_i q_{i,t} \le B_t, ]

or container and truck constraints.

If search scores a candidate before these realities are applied, the optimizer will learn to exploit infeasible actions.

If production later rounds, clips, or repairs the recommendation, then you are deploying a different policy from the one you optimized.

The evaluation loop must score the executable action.

Measure Economics, Not Just Forecast Metrics

For an ordering policy, useful outcome metrics may include:

expected contribution margin,
expected lost-sales cost,
holding cost,
disposal or markdown cost,
expedite cost,
transportation cost,
working capital,
fill rate,
order frequency,
average order size,
capacity utilization,
tail loss,
decision stability.

I would usually make one economic measure the primary objective and treat the others as diagnostics or constraints.

A policy that improves WAPE while reducing profit is not a better ordering policy.

A policy that improves fill rate by buying unlimited inventory is not impressive either.

The point of the experiment is to evaluate decisions.

Compare Against the Policy You Actually Use

Every candidate should be evaluated against a realistic baseline under the same scenarios.

Let (\pi_0) be the current production policy and (\pi_1) the proposed policy.

For each confirmation scenario compute

[ R_s = J(\pi_1,\omega_s)-J(\pi_0,\omega_s). ]

Then report more than the average.

Look at:

mean incremental value,
median incremental value,
standard error,
probability the new policy wins,
lower-tail outcomes,
inventory and service changes,
operational side effects.

If the average gain is $100,000 but the Monte Carlo standard error is $90,000, you do not have a $100,000 result. You have an unresolved experiment.

Run more confirmation scenarios.

Decision Stability Is Another Signal

Objective estimates are not the only thing that should stabilize as scenario count increases.

The decision itself should be inspected.

Suppose the recommended supplier order is:

Search scenarios	Recommended order
100	4,800
200	6,000
400	4,800
800	4,800
1,600	4,800

That is useful evidence.

Now suppose it is:

Search scenarios	Recommended order
100	2,400
200	7,200
400	3,600
800	8,400
1,600	4,800

You have a decision-stability problem even if the estimated objective values look similar.

That might mean:

the policies are economically near-equivalent,
the scenario count is too small,
the parameterization is poorly identified,
the objective is noisy,
the decision sits near a discrete breakpoint.

Those cases require different responses.

Near-Equivalent Policies Should Not Be Overinterpreted

Suppose two policies produce confirmation estimates of:

Policy A: $18.41M expected profit
Policy B: $18.40M expected profit

but A orders much more erratically and is harder to explain operationally.

Do not declare A scientifically superior because your optimizer returned it first.

If the economic difference is inside simulation noise or operational irrelevance, treat the policies as near-equivalent and use secondary considerations such as:

stability,
simplicity,
lower transaction cost,
easier implementation,
lower downside risk.

Optimization should distinguish economically meaningful decisions, not manufacture precision.

Failure Modes I See in Practice

Tuning and reporting on the same scenarios

This is the core error. The reported gain is biased toward the futures used during search.

Regenerating scenarios for every candidate

Now optimization comparisons become unnecessarily noisy. Use common random numbers within a search experiment.

Keeping the test set but repeatedly looking at it

If you change the policy every time the test result disappoints you, it has become a validation set. Generate a fresh confirmation set later.

Using realized history as information

Historical demand is an outcome, not necessarily information the policy knew at decision time.

Ignoring execution repair

If production rounds, clips, or reallocates the action, include that logic in evaluation.

Comparing against a straw-man baseline

Beating a naive forecast-ordering rule is irrelevant if the actual operation already uses a sophisticated planner process.

Reporting only the winning mean

A mean without uncertainty around the estimated improvement hides whether the experiment can actually distinguish the policies.

Searching forever

At some point the search algorithm is spending thousands of simulations to find basis-point improvements that will disappear in confirmation.

Stop when further search is worth less than the compute and engineering effort.

What I Would Log

For every serious policy-tuning run, keep an experiment record with:

policy version,
parameter vector,
simulator version,
scenario generator version,
search seed set,
validation seed set,
confirmation seed set,
forecast vintage rules,
number of scenarios by stage,
objective decomposition,
constraint violations and repairs,
runtime,
incumbent trajectory,
baseline comparison,
final confidence interval or standard error.

This makes results reproducible and makes debugging much easier when somebody asks three months later why a parameter changed from 17 to 23.

Do not rely on screenshots from an optimization notebook.

A Practical Workflow

Here is a workflow I would actually use for a production supply-chain policy.

Step 1: Define the executable decision

Write down exactly what is committed, at what cadence, and at what granularity.

Step 2: Define the information state

Specify what the policy knows at decision time. Enforce point-in-time correctness.

Step 3: Define the economics

Include the costs and rewards that materially change the decision. Keep an objective decomposition so the result can be audited.

Step 4: Build the closed-loop simulator

The policy acts, uncertainty realizes, state transitions, and the policy acts again.

Step 5: Create fixed search scenarios

Use common random numbers so candidate comparisons have less Monte Carlo noise.

Step 6: Search economically meaningful policy parameters

Do not optimize 200 arbitrary knobs just because the code exposes them.

Step 7: Validate candidates independently

Check whether improvements survive new scenarios and whether decisions remain operationally sensible.

Step 8: Freeze the finalist

No more parameter changes.

Step 9: Run a large independent confirmation

Compare against the actual baseline policy using paired scenarios.

Step 10: Inspect the whole outcome distribution

Look at economics, risk, inventory, service, execution frequency, and decision stability.

Step 11: Shadow before rollout

Run the frozen policy beside production without executing it. Compare recommendations, explain disagreements, and verify data plumbing.

Step 12: Reopen tuning only with a new experiment

If the policy needs changes, fine. But acknowledge that the previous confirmation is no longer the final evaluation of the new policy.

The Bigger Point

Simulation-based optimization is powerful because it lets us evaluate decisions across futures that never happened.

That power creates a responsibility: we have to keep the search process from grading itself.

The optimizer’s job is to find promising decisions.

The evaluator’s job is to tell us whether those decisions survive uncertainty we did not use to choose them.

Keep those jobs separate.

You will usually need fewer arguments about whether a 0.3% improvement is real, fewer mysterious regressions after deployment, and much more confidence that the policy is learning the economics of the problem instead of the random seed.