A Timeout Is Not a Model Benchmark

The Model Took Three Minutes. What Did You Actually Learn?

Suppose you run 100 supply chain MILPs with a 180-second time limit and a 0.1% MIP gap.

Ninety-five prove optimality in 70 to 100 seconds. Four hit the time limit with a remaining gap. One never finds a feasible solution.

The obvious conclusion is that you have a difficult model and should start tuning the solver.

Maybe.

Now export the same instances to MPS and run them in another solver. Ninety-nine solve to proven optimality in under one second. The last one takes 1.09 seconds.

That changes the diagnosis completely.

The original experiment did not measure the inherent difficulty of the business problem. It measured the performance of one model representation, in one solver, with one parameter configuration, through one software stack, on one machine, under one stopping rule.

Those are not the same thing.

This distinction matters because teams waste weeks tuning the wrong layer. They change branching parameters when the formulation is weak. They rewrite constraints when the API is spending most of its time building the model. They accept a three-minute runtime as unavoidable when the exact same mathematical instance solves in a second elsewhere.

A timeout is an observation. It is not a diagnosis.

Start With the Decision, Not the Solver

Before benchmarking anything, write down the actual operational decision.

For a supply chain model, that might be:

• which purchase orders to place,
• how much of each SKU to buy,
• which plant produces each item,
• how inventory moves across a network,
• which facilities or lanes activate,
• or how scarce capacity is allocated.

Then ask what the business actually needs from the optimization system.

Does the decision run once per quarter or every five minutes? Is a 30-second solution useful? Is a 0.1% gap economically meaningful? Does the plan need a proof of optimality, or merely a feasible solution that beats the current process? What happens if no incumbent is found? How much does the input change between runs? Can yesterday’s solution be reused?

A solver benchmark without the decision context can become a contest over milliseconds that have no operational value.

The opposite mistake is also common. A team declares that runtime does not matter because the model runs overnight. Then the business changes the process, asks for 200 scenarios, or wants planners to interact with the model. A 90-minute solve suddenly becomes an architectural constraint.

Benchmark against the decision cadence you actually need.

Define the Mathematical Instance

For a generic MILP, write the problem as:

min c^T x

subject to:

A x <= b

x_j integer for j in I

The solver does not see your procurement story, inventory story, or production story. It sees the algebra you give it.

That algebra includes far more than the business logic:

• variable bounds,
• coefficient scaling,
• redundant constraints,
• symmetry,
• Big-M values,
• indicator structure,
• implied bounds,
• objective degeneracy,
• and the exact integer representation.

Two models can represent the same business decision and behave very differently computationally.

Two solvers can also behave very differently on the same exact algebra.

You need experiments that separate those effects.

The Four Layers of Runtime

When someone says, “the optimization takes 100 seconds,” split that number into four layers.

1. Data and preprocessing

How long does it take to read data, join tables, build scenario sets, construct MOQ trees, calculate bounds, or create network structures?

This work can dominate the pipeline. Solver tuning will not fix a Python loop that spends 40 seconds generating coefficients.

2. Model construction

How long does the application spend creating variables and constraints through the solver API?

A model with millions of tiny API calls can be slow to build even when the mathematical problem solves instantly.

3. Solver time

How long does the solver spend after optimization begins?

This is the number most people mean when they discuss MIP performance, but it is only one component of end-to-end latency.

4. Postprocessing and validation

How long does it take to extract the solution, reconstruct eliminated variables, validate constraints, write outputs, and create the plan the business actually consumes?

Measure all four separately.

If you only record total wall-clock time, you can easily tune the wrong system.

Export the Model and Remove Your Application From the Experiment

One of the highest-value debugging steps is also one of the simplest: export the generated mathematical model.

Common formats include MPS and LP.

The exported model lets you ask a clean question:

How hard is this exact mathematical instance when the application code is removed?

Run the exported file directly from the solver command line or a minimal script. Record:

• read time,
• presolve time,
• root relaxation time,
• first incumbent time,
• best bound progression,
• node count,
• final gap,
• total solve time,
• and termination reason.

If the exported model solves quickly but the production system is slow, the bottleneck is probably outside the core optimization algorithm.

If the exported model is also slow, you now have a reproducible artifact for formulation analysis and solver comparison.

This is much better than tuning through a large application where data loading, model construction, logging, callbacks, and postprocessing are all mixed together.

Cross-Solver Testing Is a Diagnostic Tool

Testing a second solver is useful even if you have no intention of switching solvers.

The purpose is not to declare a universal winner. MILP performance is highly instance-dependent. One solver may dominate on one model family and struggle on another.

The useful question is:

Does the performance problem follow the mathematical instance, or does it follow the solver environment?

If two mature solvers both struggle in similar ways, look hard at the formulation and instance structure.

If one solver needs 100 seconds and another needs one second on the same MPS file, investigate the difference before rewriting the model.

Possible causes include:

• presolve reductions,
• cut generation,
• primal heuristics,
• symmetry detection,
• branching decisions,
• numerical handling,
• default parameter choices,
• or a solver-specific interaction with the model structure.

The correct response is not automatically “switch solvers.” The correct response is to learn what the experiment is telling you.

Maybe a particular formulation pattern is hostile to one solver. Maybe a parameter was changed years ago and forgotten. Maybe the model uses a feature that is translated poorly by one interface. Maybe one solver is simply much better on this family of instances.

You now have evidence.

Benchmark a Family, Not a Favorite Instance

A single instance proves almost nothing.

Real supply chain models vary with:

• number of active SKUs,
• vendors,
• locations,
• time periods,
• demand levels,
• capacity tightness,
• MOQ activation patterns,
• inventory state,
• and forecast scenarios.

The easy Monday instance may have abundant capacity and obvious decisions. The hard Friday instance may sit exactly at several capacity and MOQ thresholds.

Build a benchmark set that represents the actual operating distribution.

At minimum, include:

• normal instances,
• large instances,
• highly constrained instances,
• instances with weak or missing incumbents,
• and known pathological cases.

Do not report only average runtime. Averages hide the cases that break production.

Track the median, 90th percentile, 95th percentile, maximum, timeout rate, and no-incumbent rate.

A solver that is slightly slower on average but never fails may be better for an operational system than one that is extremely fast 95% of the time and useless on the remaining 5%.

Use the Same Stopping Rules

Solver comparisons become nonsense when the stopping conditions differ.

For each experiment, control:

• time limit,
• relative MIP gap,
• absolute MIP gap if relevant,
• thread count,
• random seed where possible,
• hardware,
• memory limits,
• warm starts,
• and whether callbacks or custom cuts are active.

Be especially careful with the word “optimal.”

One run may stop at a 0.1% relative gap. Another may continue until the gap is exactly zero. Another may report optimality after tolerances are applied.

Record the actual termination status and final bound. Do not reduce the result to “solved” or “failed.”

For business decisions, also translate the gap into money.

A 0.5% gap on a $200 million objective may sound large, but the objective may contain $199 million of fixed baseline economics that the decision cannot change. The meaningful opportunity range may only be $500,000.

The solver gap and the economic decision gap are not always the same thing.

Measure the Search, Not Just the Finish Time

Two runs can both finish in 60 seconds and have completely different behavior.

Run A finds an excellent feasible solution in 0.2 seconds and spends the rest of the time proving it.

Run B finds no feasible solution for 59 seconds and then suddenly finishes.

Operationally, these are different systems.

Track at least:

• time to first feasible solution,
• first incumbent objective,
• time to a business-acceptable solution,
• time to target gap,
• time to proven optimality,
• final best bound,
• and incumbent improvement over time.

For many planning systems, time to a good incumbent matters more than time to proof.

For other systems, proof matters because the model is used to certify a minimum cost, validate a decomposition, or compare formulations.

Choose metrics based on the role of the optimization system.

The One Instance With No Feasible Solution Matters Most

Suppose 99 instances solve and one finds no feasible solution before the time limit.

Do not bury that result in the average.

Ask whether the model is actually infeasible or whether the solver simply failed to find an incumbent.

Those are very different conditions.

A production system should distinguish:

• proven infeasible,
• no feasible solution found yet,
• feasible but not proven optimal,
• optimal within tolerance,
• numerical failure,
• and resource termination.

For supply chain applications, build a deliberate fallback path.

Possible fallbacks include:

• warm-starting from the previous plan,
• relaxing lower-priority constraints with explicit penalties,
• running a feasibility model first,
• using a repair heuristic,
• freezing stable decisions and re-solving a smaller problem,
• or returning the last validated plan.

“No incumbent” should be a designed operational state, not an exception nobody considered.

Before Tuning Parameters, Inspect the Formulation

Solver tuning is attractive because it feels easier than changing the model.

But parameters cannot rescue every formulation.

Check the basics first.

Are variable bounds tight? Are Big-M values derived from real operational limits? Is there symmetry among identical facilities, periods, or resources? Are continuous variables accidentally declared integer? Are there duplicate constraints? Is the objective scaled reasonably? Are logical relationships expressed directly or through weak aggregate constraints?

Look at the solver log.

If the root relaxation is weak, work on the formulation.

If the root bound is strong but no incumbent appears, investigate primal heuristics, warm starts, and feasibility structure.

If the incumbent is excellent immediately but the bound barely moves, the business may already have a usable answer and the remaining problem is proof.

If presolve removes nearly everything, the model may be much easier than the application runtime suggests.

If performance changes wildly across nearly identical instances, investigate numerical scaling, degeneracy, and threshold effects.

The log tells you what kind of problem you have.

Uncertainty Changes the Benchmark

Many supply chain systems do not solve one deterministic model. They solve a family of models across scenarios, forecast vintages, simulation replications, or candidate policies.

That changes what runtime means.

If a simulation-optimization system evaluates 500 candidates and each candidate requires 1,000 replications, shaving 100 milliseconds from one MILP may matter enormously.

If the model runs once per month, it may not matter at all.

Uncertainty also changes instance difficulty. One scenario may create loose capacity and easy choices. Another may activate multiple shared MOQs, shortages, and substitution decisions at once.

Benchmark across the uncertainty distribution you actually use.

For repeated experiments, use common random numbers where appropriate so that changes in model or solver configuration are not confused with changes in sampled demand paths.

A performance comparison should change one thing at a time.

A Practical Benchmark Table

For every instance, record a row with fields such as:

• instance ID,
• model version,
• solver and version,
• parameter profile,
• hardware,
• rows,
• columns,
• integer variables,
• nonzeros,
• build time,
• presolve time,
• root relaxation time,
• time to first incumbent,
• incumbent objective,
• best bound,
• final gap,
• node count,
• solve time,
• total wall-clock time,
• and termination status.

Then add business-facing fields:

• economic value versus baseline,
• constraint violations after postprocessing,
• decision churn,
• number of manual repairs,
• and whether the output was actually usable.

The fastest mathematical solve is not necessarily the best production result.

Common Failure Modes

Tuning one lucky instance

You improve the case sitting on your laptop and make the full distribution worse.

Use a representative benchmark set.

Comparing different models

One implementation silently drops constraints, changes tolerances, or rounds data differently.

Compare exported models and validate objective values and decisions.

Comparing different stopping rules

One solver is asked for 0.1% and another for exact proof.

Normalize the experiment.

Ignoring model-build time

The solver takes one second, but the application takes two minutes to create the model.

Profile end to end.

Treating a timeout as infeasibility

No incumbent was found, so the pipeline reports “no solution.”

Preserve solver status precisely.

Changing ten parameters at once

The runtime improves and nobody knows why.

Use controlled experiments and keep parameter profiles versioned.

Assuming the fastest solver today will always be fastest

The model evolves. New constraints, new data distributions, and new solver releases can change the ranking.

Re-run the benchmark suite when the system changes materially.

What to Do in Practice

Use a disciplined sequence.

First, define the decision cadence and what counts as a usable answer.

Second, profile the full pipeline so you know whether the time is in data preparation, model construction, solving, or postprocessing.

Third, export representative mathematical instances.

Fourth, build a benchmark set that includes normal, large, constrained, and pathological cases.

Fifth, run controlled baselines with fixed hardware and stopping rules.

Sixth, inspect solver logs and classify the problem: weak bound, poor feasibility search, slow proof, numerical trouble, or external overhead.

Seventh, test a second solver as a diagnostic if available.

Eighth, improve the formulation before launching a random parameter search.

Ninth, tune only against the full benchmark suite.

Tenth, measure business usefulness alongside computational performance.

The main lesson is simple.

Do not tune a runtime number you have not decomposed.

A model that takes three minutes may be hard. It may be badly formulated. It may be badly implemented. It may be poorly matched to one solver. It may spend almost no time solving at all.

Find out which problem you actually have before you optimize it.