The Value of Geotechnical Information in Decision Analysis - Part 2: Imperfect Information

In Part 2 of this article, we continue exploring methods of quantifying the expected value of geotechnical data that lead to (imperfect) information, such as obtained through subsurface investigation. This is generally valuable, though it is especially important in situations where it would be challenging to effectively mitigate risk through the Observational Method. We build on examples that were presented in Part 1. If you have not read that yet, we encourage you to do so before proceeding here.

Expert Opinions are Imperfect, but Invaluable

Before exploring the value of geotechnical data to reduce uncertainty, let’s first imagine a case where the degree of uncertainty about the performance of Option A is the maximum possible amount – a 50/50 chance that it will fail. The expected cost associated with building Option A is $1M plus 50% of the $15M consequence if it fails (a total of $8.5M) while the cost of building Option B is $2M. Clearly, with the information in-hand, building Option B would be the best decision.

Here, we take the opportunity to point out that obtaining an expert opinion is a form of measurement, and, while imperfect, can be invaluable, especially when the current degree of uncertainty is high. Earlier, we imagined that a subjective estimate found that the probability of failure of Option A was 5%. Based on this information, Option A became the best option, with an expected cost of $1.75M. It was not possible to know in advance what the outcome of the risk assessment would be, but in hindsight it was worth $250k. Furthermore, it allows for quantification of the expected value of collecting additional information.

The Value of Geotechnical Site Investigation Data

Now we will turn our attention to how we might gather geotechnical data to reduce uncertainty and risk, and how to anticipate and quantify the expected value of the imperfect information that can be derived from this data (information that may or may not reduce uncertainty). The process is the same as for calculating the value of perfect information, except now we will try to estimate the probability that the imperfect information will reveal good news or bad, and then calculate updated expected costs for our options given the test outcome. In our hypothetical example, good news will increase our confidence that geotechnical conditions are more favourable, and Option A has greater chance of success than we previously assumed. Bad news will increase our confidence that geotechnical conditions are unfavourable, and that Option A has greater chance of failure than we previously assumed.[1] We will use Bayes’ theorem to update our belief about the nature of the geotechnical conditions.

The estimation process follows the steps illustrated below (example numerical values are provided to make it a little easier to follow the logic with the example that follows):

• Step #1: Estimate the prior expected cost of the best option given existing information
• Step #2: Estimate the cost to gather new information
• Step #3: Estimate the probabilities that the new information will suggest good news or bad
• Step #4: Estimate the updated expected cost of the best options with the benefit of the new information (we illustrate the process in the example that follows)
• Step #5: Multiply and sum the scenario probabilities and expected costs to calculate a combined probability-weighted expected cost of the best options with the benefit of the new information. This is the expected cost of the decision with additional information.
• Step #6. Calculate the gross expected value of the new information by subtracting the value of Step #5 from the value of Step #1.
• Step #7. Calculate the net expected value of the new information by subtracting the cost of gathering the information from the result of Step #6.

Example: Potential for Liquefaction During Structure Design Life

In this scenario, imagine our structure is to be placed on a site where the soils are potentially liquefiable. Design Option A ($1M construction cost) is expected to fail to meet performance objectives (at a cost of $15M) if liquefaction occurs, while Option B ($2M construction cost) has been designed to accommodate the potential for liquefaction. For this simplified example, we assume there is a 10% chance that liquefaction will occur over the structure design life if liquefiable soils are present, and the design team has assumed the probability of such soils being present is about 50% yielding about a 5% chance the structure will fail to meet performance objectives. We will again ignore the potential timing of failure and the time value of money.

Here we build off an example illustrated by Christian (2004) that illustrates how an initial estimate of the probability of liquefiable soils being present can be updated using the results of subsurface investigation results.[2] Christian proposed the following scenario in which the imperfect nature of the information is evident:

• 0.30 = the probability of finding evidence of liquefiable soil in a single test, if it exists (a true positive; aka True Bad News) (P[F|E])
• 0.70 = the probability of not finding evidence of liquefiable soil if it exists (a false negative; aka False Good News) (P[notF|E])
• 0.05 = the probability of finding evidence of liquefiable soil if it does not exist (a false positive; aka False Bad News) (P[F|notE])
• 0.95 = the probability of not finding evidence of liquefiable soil if it does not exist (a true negative; aka True Good News) (P[notF|notE])
• 0.5 = the initial assumption that the probability of liquefiable soils is present (P₀[E])
• 0.5 = the initial assumption that the probability of liquefiable soils not being (P₀[notE])

The various probability estimates listed above are hypothetical and would need to be estimated based on experience and judgement for application to a real scenario.

Based on the above assumptions, the probability that the first test finds evidence of liquefiable soil (bad news) is 17.5% (=0.3*0.5+0.05*0.5) and the probability that evidence is not found (good news) is 82.5%. So, we wouldn’t be too surprised if the first test does not find evidence of liquefiable soils, and a common form of Bayes’ theorem (Equation 3) would suggest that our estimate of the probability of liquefiable soils being present should be reduced a little to 42.4%. This leads to the value shown for Step #4 with favourable information.

Conversely, we would be more surprised to find evidence of liquefiable soils in the first test, and if that occurred, Bayes’ theorem (Equation 2) would suggest that we increase our estimate of the presence of liquefiable soils to 85.7%. This large increase is a reflection of our surprise.

After each test is complete, we can use the same process outlined above to assign probabilities to the outcome of the subsequent test and calculate how the outcome should further modify our belief that a liquefiable zone is present. Some possible outcomes for six tests are shown below. For example, if two tests are conducted there are three possible outcomes with associated posterior probabilities that liquefiable soils are present: two negative tests (P₂[E|F]=35.2%); one positive and one negative test (P₂[E|F]=81.6%) or, two positive tests (P₂[E|F]=97.3%). Results highlighted in blue are scenarios where it continues to make sense to proceed with Option A (because the estimated PoF of Option A is less than 6.7%), while results highlighted in red are scenarios where we should switch to Option B.

In addition, we provide a summary below of the probabilities that no evidence of liquefiable soil is found in tests 1 through 6 (the second column), when the prior probabilities are updated after each test. In other words, each test made us more certain that evidence is not there. The third column of the table below provides probabilities that the next test finds evidence of liquefiable soil when none of the prior tests did.

What insight can we gain from the results presented above? Let’s start with the two possible outcomes for the first test.

If the first test finds evidence of liquefiable soils, the updated estimate of the presence of liquefiable soils becomes 85.7%, implying Option A has an 8.57% chance of failure. The updated expected cost of Option A becomes $2.29M. With this outcome, the most logical decision (in the absence of more testing) would be to proceed with Option B which has an expected cost of $2M. Before conducting the test, the prior probability of realizing this result was 17.5%, as calculated above.

If the first test finds no evidence of liquefiable soils, the updated estimate of the presence of liquefiable soils becomes 42.4%, implying Option A has an 4.24% chance of failure. The updated expected cost of Option A becomes $1.64M. With this outcome, the most logical decision (in the absence of more testing) would be to proceed with Option A which previously had an expected cost of $1.75M and was already less than Option B. The prior probability of realizing this result was 82.5%.

The predicted outcomes of the first test are shown in the graphic of our desired seven-step evaluation process presented earlier. We expect there is an 82.5% chance the test will indicate we should proceed with Option A at an updated expected cost of $1.64M, and a 17.5% chance that we should switch to Option B with an expected cost of $2M. When the probability-weighted values are combined, the expected cost of this decision is $1.70M. The calculations suggest the expected gross value of the first test is $50k because it reduces the expected cost of our decision from $1.75M to $1.70M.

Following similar logic to that outlined above, we can now calculate the expected value of the second, third, fourth, and fifth test. In the results presented below, we only consider cases where the prior tests find no evidence of liquefiable soils, and the probabilities of liquefiable soils being present are updated after each test prior to proceeding to the next.

A closer examination of the expected gross value of the information gathered from each test reveals an interesting insight: if an unfavourable test result has no chance of causing us to alter our decision, the expected gross value of the information from that test is zero.[3] In our example, this was the case for Test #5 (assuming all prior test results were favourable). Even if Test 5 found evidence of liquefiable soils, we would still proceed with Option A – although it would carry an expected cost that was higher than we previously realized.

After the third favourable test result, the expected net value of additional testing becomes negative when the cost of the test is factored in. In the confines of this very simplistic example, it would only make sense to proceed with additional tests if the decision maker is risk-adverse (and not risk neutral), or if there is some other value that could be extracted from the test result.

Based on the foregoing, it might be tempting to conclude that if, after four negative tests, there is no value in conducting a fifth, then there must also be no value in conducting a sixth. A closer examination, however, shows that if both the fifth and sixth test found evidence of liquefiable soils, the probability would increase to 91.4%, high enough that Option B would clearly become the best option. If, after four tests we committed to doing two more, the a priori chance that both tests find evidence of liquefiable soils would be a little more than 1%. If this occurred, the posterior expected cost of Option A would increase to $2.33M ($330k more than the expected cost of Option B). The gross value of these two tests (considering both the probability and implications of two bad outcomes) would be about $3,770 – perhaps not enough to justify their cost, but still greater than zero.

Closing Remarks

In Part 2 of this article, we built on the notion that the expected value of geotechnical information can only be quantified if we declare our prior degree of belief in ground conditions, system performance or cost, and are willing to update our degree of belief when presented with new information. Bayes’ theorem provides us with a tool to update our degree of belief in a rational way. The possibility that new information will cause us to alter our decision is what gives it value.

The foregoing examples have been kept very simple, but the underlying thought process and general conclusions will still apply when we add greater complexity. For example, application of Bayes’ theorem to update prior estimates of the expected state of nature can be extended to the updating of prior estimates of probability density functions for important geotechnical design parameters. In our example of searching for liquefiable soils, we used an assumption that the pattern of investigation was random when estimating the likelihood of finding or not finding evidence. Similar methods can be applied where investigations will be conducted on a regular grid pattern or informed by an interpretation of the site geology. In each case, the math is a little more complicated, but it allows us to extract more insight and value from the data.

The simple methods we presented here are expected to be of value to most geo professionals – they can help us appreciate what we know and don’t know, cause us to ask more intelligent questions, and put bounds on the expected outputs of more complex models. This is especially important as we start leveraging machine learning and artificial intelligence in more of our analyses.

We hope we have demonstrated that, in the context of decision analysis, collection of expert opinions and geotechnical information will often be of great value when a decision carries a large amount of uncertainty and a high consequence of being wrong. If a targeted program can be designed to reduce the key sources of uncertainty, and implemented in a timely manner, it will likely be worthwhile. If additional geotechnical information has no chance of altering a decision, it may be of little value.

We recognize, however, that the decisions to be made are always more nuanced than our simplistic examples. We might collect geotechnical information in the hopes that it will help us optimize a design, allow us to set aside appropriate construction budgets, or reduce the potential for contractor claims. Mobilization costs are often high, so once we are mobilized it may cost very little to collect additional information, even if we don’t have an immediate need for it. Or we may collect information because there is a regulatory requirement to do so, or to satisfy accepted standards of practice. But even in these situations, there are decisions to be made, each with different uncertainties, costs and benefits. Another challenge with geotechnical decisions is that we sometimes encounter unknown-unknowns: the methods here help us quantify the expected value of reducing uncertainty for known-unknowns, but if geotechnical information happens to shine a light on an unknown-unknown, that will also be extremely valuable. It is just difficult to quantify that value in advance of collecting the information.

Lastly, we note that geotechnical information has the greatest potential to add value if it is collected, organized, visualized, and shared in ways that maximize the potential for correct and timely interpretation by those who need it. Bayes’ theorem shows us that we can learn more from each piece of information when the potential for false negatives and false positives can be minimized – good data management practices and visualization tools can help us achieve this. And lastly, few things are more frustrating and costly than investing great sums to re-gather information that was lost or forgotten. If geotechnical information has value, let’s be sure to treat it that way.

–

[1] To complicate matters slightly, we will also refer to bad news (evidence of unfavourable conditions) as a “positive” test and vice versa.

[2] Christian, J.T. 2004. Geotechnical Engineering Reliability: How Well Do We Know What We Are Doing? Journal of Geotechnical and Geoenvironmental Engineering, Vol. 130, No. 10, October 1, 2004.

[3] We later provide some qualifiers to this provocative statement