Debug the Policy, Not Just the Model

A solver can return OPTIMAL and the decision can still be terrible.

That sounds obvious until you watch a production optimization system fail. The first instinct is usually to inspect the solver. Was the MIP gap too large? Did presolve remove something important? Did the model hit a time limit? Are the coefficients badly scaled?

Those are good questions. They are also only one layer of the problem.

A production optimizer is not a mathematical model sitting in isolation. It is a decision policy embedded in a system. Data becomes state. State and forecasts become model inputs. The model produces a decision. Business logic transforms that decision. Execution changes the physical system. New information arrives. Then you do it again.

If you only debug the optimization model, you can prove that the solver correctly optimized the wrong state, the wrong economics, or the wrong decision.

Start with the decision

Before opening a solver log, write down the actual decision the system is supposed to make.

For a replenishment problem, that might be how many units of each SKU to order, from which supplier, in which period, and using which transportation mode, subject to pack sizes, MOQs, capacity, cash, and receiving limits.

If the production system instead chooses a target inventory position and another service later converts that target into an order, then the optimizer does not actually control the order. That distinction matters. The downstream conversion logic is part of the policy and therefore part of the debugging surface.

A useful abstraction is:

[ S_t \rightarrow X_t \rightarrow W_{t+1} \rightarrow S_{t+1} ]

where (S_t) is the state you observe, (X_t) is the action you choose, and (W_{t+1}) represents information and uncertainty that arrive afterward.

When a decision looks wrong, debug that chain in order.

1. Is the state correct?

This is where many optimization failures actually begin.

Suppose the model sees 800 units of inventory. Is that on-hand inventory? Available-to-promise? Does it include damaged units? Inventory already allocated to customer orders? Inventory in transfer? Purchase orders that have been placed but may arrive late?

A perfectly formulated inventory balance cannot rescue a bad definition of inventory.

For every important state variable, ask what the field means economically, when it was observed, whether the optimizer could have known it at decision time, whether it is a stock, flow, forecast, or accounting artifact, and whether the same quantity is represented somewhere else.

The timestamp question is especially important in backtests. If today’s optimization uses a forecast that was regenerated next week, you have leaked future information into the policy. The backtest may look fantastic while the production policy is mediocre.

Keep point-in-time snapshots of the information that was genuinely available at each decision epoch.

2. Is the decision variable actually executable?

Mathematical convenience can quietly create fake decisions.

Imagine a model chooses continuous quantities (x_i), but the supplier only ships cases of 24 and imposes a 500-unit MOQ across a product family. The continuous solution may be economically sensible but operationally meaningless.

The executable structure might instead require

[ x_i = 24k_i, \qquad k_i \in \mathbb{Z}_+ ]

plus a group activation variable

[ \sum_{i \in G} x_i \ge 500 y_G. ]

Now the decision surface is discrete. A one-unit change in demand does not necessarily change the order. A small change near an MOQ breakpoint might change it dramatically.

This is why debugging should compare the raw model decision, the post-processed decision, and the executed decision. If those three are different, log all three.

3. Are the economics pointing in the right direction?

When a model produces a surprising decision, do not immediately add a constraint to stop it. First ask why the objective prefers it.

Decompose the objective by economic component:

[ \text{Value} = \text{Revenue} - \text{Purchase Cost} - \text{Holding Cost} - \text{Shortage Cost} - \text{Freight} - \text{Other Costs}. ]

For the suspicious decision, calculate each component separately.

This catches errors that a solver log never will: a holding cost applied once instead of every period, a stockout penalty with the wrong unit, a freight rate expressed per case but multiplied by units, a salvage value applied to all inventory rather than terminal inventory, or a cost coefficient with the wrong sign.

One of the best debugging tools is not sophisticated at all: manually calculate the economics for a tiny instance where you already know what should happen.

If ordering one additional case costs $240 and can generate at most $180 of incremental value, the optimizer should not buy it unless another constraint or future consequence makes the case worthwhile. If it does, trace the objective terms until you understand why.

4. Ask which constraint changed the decision

Constraints should represent genuine limits on feasible actions, not vague business preferences.

When a result surprises you, solve a sequence of controlled experiments. Remove or relax one family of constraints at a time and observe the decision delta.

If removing warehouse capacity changes nothing, capacity is not causing the behavior. If removing a vendor MOQ completely changes the order mix, you have found an important decision boundary.

For LPs, dual values can help identify scarce resources. For MILPs, be careful: shadow prices from the relaxation are useful diagnostics but are not universal marginal values for a discrete problem. An MOQ, setup binary, or truck activation can make the value of capacity highly non-smooth.

For discrete models, perturb-and-resolve experiments are often more honest.

5. Separate uncertainty from model bugs

A decision can look bad after the fact and still have been correct ex ante.

Suppose you order 1,000 units because the demand distribution implied substantial upside and limited downside. Realized demand turns out to be 300. That does not prove the optimizer over-ordered. You observed one realization from a distribution.

The correct debugging question is whether the policy performs well across plausible realizations.

Use Monte Carlo evaluation. Freeze the information available at decision time, generate many plausible future demand, lead-time, or supply paths, run the policy on those paths, measure realized economic outcomes, and compare alternatives using the same scenarios.

Common random numbers matter. If Policy A and Policy B are evaluated on different random futures, simulation noise can masquerade as policy improvement.

The metric should usually be economic: expected profit, expected cost, regret, downside risk, or another quantity tied to the actual decision. Forecast accuracy can be useful diagnostically, but it is not the objective of an ordering system.

6. Build a decision trace

Production optimization needs better logs than OPTIMAL.

For each run, I want enough information to reconstruct why a material decision happened. At minimum, log the run ID and model version, timestamp and data vintage, important state variables, forecast or scenario version, objective components, major constraint utilization, raw optimization decision, post-processing adjustments, executed decision if available, and solver status, runtime, gap, node count, and relevant warnings.

For high-value decisions, add a local explanation. What happens if the order is one pack lower? One pack higher? Which constraint becomes active? How much objective value changes?

That creates a decision neighborhood rather than a mysterious point estimate.

7. Reproduce the failure with the smallest possible instance

Large production instances are terrible debugging environments.

If a strange behavior occurs for one vendor with 4,000 SKUs, isolate the affected vendor, product family, or time window. Preserve the state and economics that created the behavior while removing unrelated dimensions.

Then make the instance smaller again.

A five-SKU, four-period model that reproduces the failure is far more valuable than a million-variable production dump. You can inspect every coefficient, enumerate alternatives if necessary, and test hypotheses quickly.

This also separates formulation bugs from data-pipeline bugs. If the reduced mathematical instance behaves correctly but production does not, the failure is probably outside the core model.

8. Test invariants, not just examples

Example-based tests are useful. Structural tests are better.

Create invariants that should hold across broad classes of inputs. Inventory cannot become negative unless backorders are explicitly modeled. An order cannot violate a case pack. Removing a binding capacity constraint should not make the optimal objective worse. Increasing a pure cost coefficient should not make an otherwise identical action more attractive. A zero-demand, zero-salvage SKU should not receive inventory without some coupling reason. Decisions must be reproducible when the random seed and inputs are fixed.

Property-based tests around these relationships catch errors that individual regression cases miss.

9. Measure policy quality, not just solve quality

Solver metrics and policy metrics answer different questions.

Solver metrics include runtime, MIP gap, nodes, iterations, numerical warnings, and memory. They tell you whether the mathematical problem is being solved effectively.

Policy metrics include expected economics, regret, stockouts, waste, expedite frequency, decision volatility, capacity utilization, and constraint violations after execution. They tell you whether the decision system is useful.

Track both.

A formulation change that reduces runtime from 90 seconds to 20 seconds is valuable if policy quality stays effectively unchanged. A parameter change that closes the MIP gap from 0.5% to 0.05% is not automatically valuable if it adds two minutes of runtime and changes no executable decision.

The business consumes decisions, not optimality certificates.

Common failure modes

Debugging only the solver log. The solver may be doing exactly what you asked. Trace the full policy.

Adding constraints to suppress surprising outputs. A surprising output is evidence. Understand the economics before encoding another permanent rule.

Using realized demand to judge one historical decision. Evaluate ex ante policy quality across uncertainty, not hindsight on one sample path.

Comparing policies on different simulations. Use identical scenarios whenever possible so the difference comes from the policy rather than Monte Carlo noise.

Ignoring post-processing. Rounding, MOQ repair, overrides, and downstream business logic can materially change the optimized action.

Testing only average SKUs. Production failures often live in tails: huge lead times, tiny volumes, enormous MOQs, sparse demand, or extreme costs.

Optimizing a proxy and reporting the proxy. If the business cares about dollars, evaluate dollars. Service level, forecast accuracy, and inventory turns are diagnostics unless they are genuinely the economic objective.

What I would do in practice

When a production optimizer makes a decision that looks wrong, I use a simple hierarchy.

First, freeze the exact information set and reproduce the run. If you cannot reproduce it, fix reproducibility before doing anything else.

Second, verify the state and the executable decision. Make sure the optimizer saw what you think it saw and controlled what you think it controlled.

Third, decompose the objective. Understand the marginal economics around the suspicious action.

Fourth, perturb constraints and decision quantities. Find the breakpoint that caused the action to change.

Fifth, reduce the instance until the behavior is obvious enough to inspect manually.

Sixth, evaluate the policy across common simulated futures. Determine whether you found a bug, a bad modeling assumption, or simply an unfavorable realization of uncertainty.

Finally, turn the failure into a permanent test and a better production log.

That last step matters. Debugging should make the next failure cheaper to understand.

A good optimization system is not one that never produces surprising decisions. Real economics under uncertainty will produce plenty of them. A good system is one where you can explain, reproduce, challenge, and improve those decisions without guessing.

That is the difference between having a mathematical model and having a decision system.