Monte Carlo cost estimation: Why it’s the most defensible method and what AI changes

Sixth and final part of a six-part series on Monte Carlo simulation and AI-assisted risk analysis

Of the 20 NASA projects above $250 million that auditors examined from 2000 to 2009, 17 went over budget [1]. In 2009, the agency made joint cost and schedule confidence levels, usually built with Monte Carlo simulation, compulsory for projects of that size, and the first projects estimated that way did markedly better. NASA’s Inspector General*, the agency’s independent internal auditor, then audited the new process and found its weak points—and they weren’t in the simulation.

In this article, we’ll cover why Monte Carlo is the most defensible method for cost estimation, where NASA’s process went wrong, and how to build, verify, and defend an estimate with @RISK Agent, without letting AI’s speed replace the work on your inputs.

What happened at NASA

NASA has a long record of projects costing more than promised. When its Office of Inspector General looked back in 2015, it found average cost growth of 53% on those 2000–2009 projects [1].

Under the 2009 rule, projects above $250 million now had to produce a joint cost and schedule confidence level: a probabilistic analysis, usually Monte Carlo, of a cost-loaded schedule with the project's risks built in. Budgets were to be set at the 70% level [1].

By August 2015, ten projects estimated this way had launched. Four came in under budget, one on budget and five over. Only two of the overruns exceeded 10% [1].

That is a striking change, and the Inspector General was careful not to oversell it. The sample was small. Most of the early projects were smaller and less complex than the ones still in development. Three of the ten had their analysis done at a rebaseline rather than at the original estimate. One of the best performers credited its result to good project management rather than to the confidence-level process [1].

The caution proved justified. By 2020, the GAO** was reporting cost growth across NASA's major projects of about 31% over baseline, the third year in a row it had worsened after a period of decline [2].

So the method did not fix NASA. The audit is still worth reading, because it shows where the method broke.

Where it broke

The simulation engine was not the problem. The inputs were, and so was the way people used the output.

The Inspector General examined nine projects in depth and found significant weaknesses in scheduling, risk assessment or cost estimating in five of them [1]. On one satellite, the uncertainty ranges came from interviews with team members, few of whom understood what uncertainty meant in this kind of analysis. The same project declined to use historical overrun data from comparable missions that its review board offered [1]. On another, team members overestimated their own chances and underreported risks. An independent estimate built on experience from earlier projects added $27 million and eight months [1].

The finding I would put in front of every estimator is this one. Several projects treated the analysis as a way to validate their point estimate and set their confidence levels to line up with it [1]. The model was asked to agree. It did.

Most reliable, or most defensible?

It is tempting to call Monte Carlo the most reliable cost estimating method we have. The evidence does not quite support that.

Batselier and Vanhoucke[3] compared reference class forecasting, which predicts a project's outcome from the actual results of similar past projects, with forecasting methods based on Monte Carlo simulation and earned value management. On real project data, reference class forecasting performed best for both cost and time.

That does not make Monte Carlo the wrong tool. It shows what a model fed only by its own team's judgement is missing: the outside view. NASA's Inspector General reached the same point from the other side and recommended that projects use historical data for their uncertainty ranges, alongside expert opinion [1].

What Monte Carlo does better than any single-number method is make its assumptions open to challenge. Every range, every correlation and every risk event sits in a cell a reviewer can question. The GAO's cost guide treats risk and uncertainty analysis as a step a credible estimate cannot skip, names Monte Carlo simulation, or another modeling technique, as the way to do it, and warns that ignoring correlation gives a "false sense of confidence" [4].

That is why I would call Monte Carlo the most defensible method rather than the most reliable one. How reliable it is depends on what goes in.

Putting it to work: A Monte Carlo cost estimate with @RISK Agent

The case project below is constructed for this article. The figures are illustrative and do not come from a real estimate.

A regional water authority is preparing the estimate for a €180 million wastewater treatment upgrade. The estimating team has priced 35 line items and given each a low and high percentage. The risk register holds 14 events identified.

The workflow in three steps:

• Build: Have @RISK Agent turn the estimate and risk register into a working model.
• Verify: Test the ranges against what happened on similar projects.
• Defend: Run the simulation and report the contingency and the drivers.

Step 1: Build

Using Claude for Excel, the @RISK Agent handles the mechanical work in minutes:

Prompt: "Using the @RISK Agent MCP, convert each cost line in B6:B40 into a PERT distribution using the low and high percentages in columns D and E, add the 14 events in the Risk Register sheet as discrete risks with their probability and impact ranges, and set total cost in B42 as an output."

Step 2: Verify

Run the simulation in @RISK. The P80 comes out 8% above the point estimate. That looks comfortable, and it should make you suspicious: every range in the model came from the people who wrote the estimate.

AACE’s recommended practice limits line-item ranging to inherent uncertainty on well-defined scope [5]. So check the model against what actually happened on similar work. The question is simple: if this project overruns the way the authority’s past projects did, how unusual does the model think that is?

Prompt: “Using the @RISK Agent MCP, calculate the median ratio of actual to estimated cost for past projects in the Reference sheet, apply it to the point estimate in B5, and add a formula showing which percentile of total cost in B42 that figure falls at.”

The agent writes the formula, so you don’t need to know the @RISK function behind it. Run the simulation again in @RISK to get the result.

If the typical past overrun lands at the P97, the model is calling a normal result a near-worst case. The ranges are too narrow. How you fix that is the modeler's call: widen the ranges using the reference data, add the systemic risks the register missed (ground conditions, permitting, a tighter discharge standard mid-design), and correlate cost lines that share a driver.

Step 3: Defend

The agent does not run the simulation. That stays with @RISK, and it should: the engine produces the numbers, the agent reads them. Set iterations to 10,000 and start the simulation from the @RISK ribbon. When it finishes, ask the question the board actually asks:

Prompt: “Using the @RISK Agent MCP, read the simulation results for total cost in B42, report the P50 and P80, the contingency each implies over the point estimate in B5, and rank the inputs by their contribution to the spread.”

The P80 will be higher than on the first run. It is also a number you can stand behind, because each range now traces either to the authority’s own history or to a judgement someone can be asked about.

Where AI fits

The usual story about AI and analytical roles runs as a chain: automation takes the routine work, the time it frees goes into analysis, an the role moves closer to decisions. The weak link is the middle step—nothing in the technology decides where the freed time goes.

We have heard this before. ERP arrived with the same promise for management accountants. A 2002 field study of ten Finnish companies found it had changed management accounting relatively little, and that only in some cases had it left accountants more time for analysis [6].

Experimental evidence on generative AI points the same way. In a preregistered field experiment with BCG consultants, those using GPT-4 completed 12.2% more tasks, 25.1% faster and at higher quality, on tasks within the model's capability. On a task chosen to sit outside it, they were 19% less likely to reach the correct answer [7].

Map that onto estimating. Building the model, writing the syntax and assigning distributions sit inside the frontier. Judging whether a range is honest, whether the reference class fits, and whether two lines share a driver sit outside it.

@RISK Agent does lower the barrier to build Monte Carlo models. An estimator who never learned the function syntax can now build a working model, and that is good for the discipline. But it lowers the barrier to building, not to defending. NASA's experience says where the saved hours should go: into the inputs, the history, and the review.

Four things worth watching

1. A model built from the estimate’s own ranges will usually agree with the estimate. If the team that wrote the estimate also set every range, the simulation tends to hand their number back to them. A P50 close to the point estimate is only good news once the ranges have been checked against past projects, correlations are in, and risk events are modeled.
2. Line-item ranges are not a risk analysis. Ranging captures inherent uncertainty. The events that break budgets need to be modeled as risks, with their own probability and impact [5].
3. Use the outside view as an input. Reference class data is the best check you have on whether your ranges are wide enough [1][3].
4. A confidence level is a policy choice, not a fact. NASA budgets at 70% partly because a portfolio can pool contingency across projects. Its own Inspector General questioned whether that level suits single-project programs that cannot [1]. Say which level you chose and why.

Closing the series

This series started with a simple question: what changes when AI arrives in Monte Carlo work, and what doesn't? Across energy, infrastructure and manufacturing the same pattern kept returning. AI builds the model in minutes. It does not know which risks move together, which ranges are too narrow, or what happened on the last ten projects like this one. That knowledge belongs to the people who sign the estimate.

Catch up on the rest of the series:

• AI prompt engineering for risk managers: A practical field guide with @RISK Agent
• Monte Carlo simulation in the age of AI: What changes, what doesn’t, and what gets better
• AI and risk management in energy & utilities: How @RISK and AI are changing the game
• AI and risk management in infrastructure and civil engineering
• AI and risk management in chip manufacturing

Build with AI. Verify with @RISK. Defend with both.

Sources

[1] NASA Office of Inspector General, "Audit of NASA's Joint Cost and Schedule Confidence Level Process", IG-15-024, 29 September 2015. https://oig.nasa.gov/wp-content/uploads/2024/02/IG-15-024.pdf 

[2] US Government Accountability Office, "NASA: Assessments of Major Projects", GAO-20-405, 29 April 2020. https://www.gao.gov/products/gao-20-405 

[3] Jordy Batselier and Mario Vanhoucke, "Practical Application and Empirical Evaluation of Reference Class Forecasting for Project Management", Project Management Journal, vol. 47, no. 5, pp. 36–51, 2016. https://www.pmi.org/learning/library/practical-application-empirical-evaluation-rcf-10277 

[4] US Government Accountability Office, "Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Program Costs", GAO-20-195G, Step 9: Conduct a risk and uncertainty analysis. https://gaoinnovations.gov/cost-guide/survey-of-step-9.html 

[5] AACE International, Recommended Practice 118R-21, "Cost Risk Analysis and Contingency Determination Using Estimate Ranging for Inherent Risks with Monte Carlo Simulation", Rev. 25 October 2022 (public sample pages). https://web.aacei.org/docs/default-source/toc/toc_118r-21.pdf 

[6] Markus Granlund and Teemu Malmi, "Moderate impact of ERPS on management accounting: a lag or permanent outcome?", Management Accounting Research, vol. 13, no. 3, pp. 299–321, 2002. https://doi.org/10.1006/mare.2002.0189 

[7] Fabrizio Dell'Acqua et al., "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality", Harvard Business School Working Paper 24-013, September 2023. https://papers.ssrn.com/abstract=4573321 

** The Inspector General. Every large US federal agency has an Office of Inspector General. It sits inside the agency but works independently of its management, and it audits and investigates the agency’s own programmes. 

** US Government Accountability Office.