The Value of Geotechnical Information in Decision Analysis - Part 1: Perfect Information
Geotechnical engineering and geohazard risk management are exercises in managing uncertainty, so engineers and geoscientists recognize that information about subsurface conditions, loading conditions, infrastructure performance, and cost is valuable. With the benefit of judgment and analysis, geotechnical data can be converted to information that informs design decisions, construction monitoring, and maintenance and emergency response activities.
Baecher and Christian (2003)[1] provide a great overview of methods for treatment of geotechnical uncertainty, which is often dominated by our lack of knowledge but also by natural variability of geology and earth processes over space and time. How valuable is it to reduce this uncertainty and what types of geotechnical information are expected to provide the greatest value?
For geotechnical information to be of value in decision analysis, it needs to reduce uncertainty, the reduction in uncertainty needs to enable a better decision, and the better decision must be made in a timely manner.
For geotechnical information to be of value in decision analysis, it needs to reduce uncertainty, the reduction in uncertainty needs to enable a better decision, and the better decision must be made in a timely manner. Uncertainty is usually related to the estimated magnitude of construction cost or the likelihood and cost of inadequate performance (and often both). For collection of geotechnical information to be worthwhile, the benefit of acquiring the information must be greater than the cost.
Our general sense is that we often spend too much on some types of geotechnical information and not nearly enough on others. In part this may reflect standards of practice – we are used to collecting certain types of data and these are sometimes required to satisfy codes or regulations. Consultants’ concerns about liability and owners’ concerns about contractor claims may create disincentives to optimize site investigations and designs. But perhaps, in addition, we don’t always put enough effort into understanding and communicating the information value proposition.
In this article we provide what we hope is a gentle introduction to quantifying the value of geotechnical information. We begin in Part 1 with quantifying the expected value of perfect information that could theoretically eliminate a key source of uncertainty or guarantee the viability of a lower cost alternative. We then, in Part 2, extend the approach to quantify the expected value of imperfect information which we expect to reduce, but not eliminate uncertainty or cost. This, of course, is more representative of reality.
In each case, we use the approach outlined in Baecher and Christian which shows that the expected value of information (EVI) is the difference between the expected value of a decision made using the information, and that made without (Equation 1).
EVI = (Expected Cost of Decision w/out Information) – (Expected Cost of Decision w/ Information)
We make extensive reference to expected values. Expected values are probability-weighted values, calculated by multiplying the estimated probability of realizing an outcome and its cost or benefit.
Hypothetical Scenario
Let’s begin with the following scenario. A geotechnical design team has developed two design concepts: a more aggressive design (Option A) that could be built for an estimated cost of $1M, and a much more conservative design (Option B) that could be built for $2M. If the design fails to meet required performance objectives, the estimated cost of failure (damages, delay and reconstruction) is $15M. The team is confident that Option B would achieve performance objectives, but is less confident about the performance of Option A.
In the spirit of starting simple, let’s assume for now that there is a high degree of certainty around the construction costs, that Option A will either succeed or fail to meet performance objectives, and that failure is certain to result in exactly a $15M loss. Let’s also assume that the decision maker is risk-neutral, meaning (for example) they are indifferent to a certain $1.5M loss or a 10% chance of a $15M loss because the expected (probability-weighted) value of the losses is the same.[2]
From this information alone, we gain a few early insights to the potential value of additional geotechnical information. Firstly, if the only decision is to choose between Option A or B, the important source of uncertainty to be reduced is the likelihood that Option A will fail to achieve performance objectives. For the purposes of supporting decision analysis, there is little point in collecting additional geotechnical information unless it helps reduce this uncertainty. And secondly, the maximum cost that could be justified to reduce this uncertainty must be less than $1M – if it were to cost more, the decision maker would be better off simply proceeding with Option B because acceptable performance is almost certain and it only costs $1M more than Option A.
We can tease out one more valuable insight with the information provided: For what maximum probability of failure of Option A does it become more cost-effective to build Option B? The answer can be found by calculating the probability of failure that causes the expected cost of Option A to equal to the $2M cost of Option B. A simple decision tree will show that if the likelihood of Option A failing is greater than 6.7%, the expected cost will exceed $2M.
Expected Value of Perfect Information
To gain insight into the potential value of additional geotechnical information, we require some initial estimate of the likelihood that Option A will fail to meet performance objectives.
Let’s imagine the decision-maker retains a geotechnical specialist with expertise in risk assessment to review the design concepts and work with the design team to assign a subjective probability to the chance that Option A will fail. Based on this exercise, the team arrives at an estimate of about 5% (less than the 6.7% break-even). Based on the estimated 5% probability of failure, the expected cost for Option A is $1.75M. While it carries $0.75M in expected risk cost because of the remaining uncertainty about performance, it still costs less than Option B ($2M). For a “risk-neutral” decision maker, the decision with the lowest expected cost is the one they ought to proceed with when required to decide with the information in-hand.
How valuable would it be if we could eliminate uncertainty about the performance of Option A through some process of collecting new (perfect) information – if we could know for certain that either it would fail or it would not? Unfortunately, we can’t know in advance if such perfect information will turn out to be good news or bad, so we must sum up the expected cost of the decision assuming both outcomes are possible (we must account for some probability that you spend money and get bad news). However, based on the subjective probability of failure estimate that was elicited as described above, we expect there is a 95% chance of receiving good news (Option A will be successful, therefore build it at a cost of $1M) and a 5% chance it will be bad (Option A will fail, therefore build Option B at a cost of $2M). The expected cost of this “decision” with the benefit of the yet-to-be acquired perfect information is $1.05M.
With reference to Equation 1, the logical decision without information was to proceed with Option A, which had an expected cost of $1.75M. The expected cost of the decision to be made with perfect information is $1.05M, therefore the expected value of perfect information (EVPI) is the difference:
EVPI = $1.75M – $1.05M = $0.7M.
If there was an opportunity to gather information that would eliminate uncertainty about the performance of Option A, $0.7M is the most a decision maker might be justified spending to obtain it.
In this simple example, with risk neutrality and all uncertainty being in the performance, and not considering time, and the information itself being perfect, the better bet would be to spend close to the $700k, whether you must make this decision once, or frequently for a portfolio of projects. In consideration of the simplifying assumptions, some of which are explored further later, the reality is that you’d have less to work with, but still a substantial sum could be wisely allocated to reducing uncertainty.
The Value of Information that Permits Use of the Observational Method
We know it is likely not feasible or practical to eliminate uncertainty around the performance of Option A, so how could we go about reducing the amount of risk it carries? One option is to apply the Observational Method to reduce the expected consequences should Option A perform poorly.
The Observational Method can be used to manage uncertainty and risk if observations made during construction and operation can inform a decision to alter the design (during construction) or undertake remedial works (during operation, as part of an asset management program) before catastrophic “failure” occurs. This may not be practical if the critical failure mode involves rapid loading or brittle failure but, in many cases, early evidence of less favourable ground conditions or inadequate performance can be detected during construction or operation and remedial actions taken. For the Observational Method to be effective, an inspection and monitoring program must be designed to identify the things that matter (changed ground conditions, deteriorating performance, etc.), and practical measures need to be identified (including their feasibility and cost) so that corrective action could be taken, if needed.
Let’s imagine that for our hypothetical example, the designers believe that an inspection and monitoring program would be effective at identifying evidence of poor or deteriorating performance, such that remedial actions could be taken that would prevent the $15M failure from occurring. They determine (with confidence) the potential required remedial actions, if identified early, will cost $2M. In this scenario, we are exchanging a 5% subjective probability of realizing a $15M loss for a scenario where there is a certainty of additional expenditure on an inspection and monitoring program and a 5% chance of needing to spend an additional $2M on a design change mid-construction or remedial works at some point in the future. The expected cost of the design change or remedial works (ignoring the time value of money) is $100k (a 5% chance of spending an extra $2M).
We can’t know in advance if the monitoring and inspection program set up to facilitate the Observational Method will reveal good news or bad. But we can estimate there is a 95% chance it will be good news (Option A will be successful, therefore build it at a cost of $1M) and a 5% chance it will be bad (Option A will require remedial works, therefore build Option A at a cost of $1M and spend an additional $2M on remediation). The expected cost of this decision is $1.1M.
If we were unable to collect perfect information that eliminated uncertainty about the performance of Option A, the prior alternative with the lowest expected cost was to take a chance and construct Option A at an expected cost of $1.75M. The Observational Method allows the decision maker to more confidently proceed with a modified version of Option A which now has an expected cost of $1.1M plus the cost of conducting the necessary inspection and monitoring. The maximum expected value derived from this inspection and monitoring program (if it functions perfectly) is the $650k difference. If the inspection and monitoring program was certain to identify the need for a design change or remedial works in a timely manner, this type of perfect information would have a maximum value of $650k. This is a different way that geotechnical information can provide value by enabling a less-costly alternative.
In our experience, it is common for geotechnical engineers and owners to declare they will manage uncertainty using the Observational Method, but less common that an attempt is made to quantify the expected costs and benefits of implementing such a program. The process of thinking through the likelihood and expected cost of design modifications or remedial works could help guide the design and implementation of this method. We expect that in many cases, even simplified assumptions and calculations like those outlined above will lead to better, more informed decisions.
Closing Remarks
In Part 1 of this article, we explored the value of two types of perfect geotechnical information: information that will eliminate an important source of uncertainty; and information that will ensure the viability of a lower-cost option. There is merit in quantifying the expected value of perfect information because it is relatively easy to do so, and because it helps put bounds on the maximum one should spend acquiring imperfect information.
Quantifying reasonable estimates of the expected value of perfect information became possible once we obtained an estimate of the uncertainty surrounding performance of Option A. As we will explore in greater detail in Part 2, quantifying the value of geotechnical information requires us to declare our degree of belief in the state of nature, and then be willing to change our degree of belief when presented with new information.
Our good friend, Roy Mayfield, once said he was taught to always sketch out the shape of the critical slip surface and estimate the factor of safety before running a slope stability analysis. We extend his advice to preparing stick logs with expected stratum types and depths before completing boreholes, and estimating parameter values before conducting in-situ tests or submitting samples to the laboratory. Doing so will accelerate development of our engineering judgment. It is also the key to unlocking the expected value of (imperfect) geotechnical information. The mechanics of doing so are introduced in Part 2.
[1] Baecher, G.B. and Christian, J.T. 2003. Reliability and Statistics in Geotechnical Engineering. John Wiley & Sons.
[2] The calculus would change a bit if a $15M loss would jeopardize the viability of their business, or there were other cascading consequences to either alternative, but the concepts presented here would still apply.