Brouhle, Keith, Charles Griffith, and Ann Wolverton

Daniel J. Fiorino holds a PhD in political science from Johns Hopkins University and is currently director of the National Environmental Performance Track at the U.S. Environmental Protection Agency. He teaches environmental and public policy at American University and Johns Hopkins University. He has extensive experience in federal regulation and innovation programs. His latest book is The New Environmental Regulation (2006, MIT Press) and he has published articles in several policy, law, and social science journals.

E-Mail: fiorino.dan@epa.gov

he article by Linda Pilkey-Jarvis and Orrin Pilkey on "Useless Arithmetic" takes a critical look at the use of mathematical models in environmental and natural resources (ENR) decision making. It addresses the use of these models on a range of issues, including pollutant dispersion, sea-level change, long-term changes in climate, and shoreline erosion and retreat. The authors are, to say the least, no friends of mathematical modeling, although it is not clear if they simply want models to be used better, differently, or not at all. In the end, it appears that "not at all" might be their preferred option.

The authors object to the use of mathematical models on several grounds. One is that their apparent mathematical precision lends an air of confidence that is unwarranted. The more complex the models are, they argue, the greater the likelihood they could be inaccurate. The authors contend that "an uncritical acceptance of them by policymakers may actually have exacerbated society's ENR problems." They do not discuss specifically why this is the case. A second objection is that such models displace more standard observational science as a tool for establishing the factual basis for decisions. The greatest damage incurred by relying on these models, the authors argue, is "the harm done to robust science." Models are, in a sense, a shortcut that relieves policymakers of having to use the more painstaking (and presumably more expensive) methods of observational science to establish the factual bases for their decisions. Although less explicit, a third objection is that policymakers do not know how to either use these models responsibly or use them to support their own preconceived ends. The conclusion that is drawn from this third point is that maybe policymakers should not be using these models at all.

This article offers a number of important cautions about the use of mathematical models. Like any analytical tool, such models are subject to being misused or misinterpreted. In the wrong hands, these models may be used to mislead, mystify, or obscure. In the right hands, however, modeling offers a valuable and an important aid in making decisions when we cannot rely on observational science alone, because of long time frames or the large number of variables affecting the decision. Although I agree with most of the criticisms made in this discussion, I think that the authors' recommendations go too far and urge us to dismiss a tool that often has a great deal of value.

Some of their examples do not advance or even undermine their assertions. The story of decisions made when Hurricane Floyd threatened Charleston, South Carolina, in 1990 is one such case. Thousands of people decided to evacuate, then found themselves trapped on jammed highways when they tried to leave the city. The governor had overruled emergency management officials by deciding not to turn all lanes of the interstate leading in and out of the city in one direction. Although Floyd missed the city, the massive traffic jam did not, and there was a negative political reaction. The authors fault the models for not considering the unpredictable element of human behavior—in this case, the governor's decision. However, it seems odd to fault the model when the real issue was the governor's decision. There is no way that a model, quantitative or otherwise, could have accounted for this decision.

The authors also assert at one point that "models offer no guarantee to policymakers that the right actions will be set into policy." The same could be said of any other analytical tool. At some point we have to depend on the skill and intuition of actual people who make the decisions. A mathematical model is no more than a tool for establishing the "facts" on which policymakers rely when making choices. People make bad decisions even when they have perfectly accurate, timely, and relevant information. No analytical tool can guarantee that people will make the right decision. The most we can expect of such tools is that they provide a reasonably accurate factual basis for making decisions that are inherently clouded by uncertainty.

Cost-benefit analysis, to take an example, is a commonly used analytical tool for making environmental policy decisions. It has been incorporated into a series of executive orders issued by four presidents over the last 30 years to guide regulatory decision making. Conducting a cost-benefit analysis involves a number of assumptions, particularly in how to monetize benefits. One can reasonably argue for or against cost-benefit analysis on any number of grounds. In the end, however, what is important is how policymakers use the analysis and their awareness of its strengths and limitations. Similarly, health and safety regulatory agencies routinely use risk assessments to establish some kind of factual basis for their policy decisions. Again, this is a form of quantitative analysis that requires analysts to make any number of assumptions that influence the information that is presented to policymakers. Assuming that the risk assessment follows generally acceptable methods, the key issue in the end is how the risk assessment is used and with what awareness of its limitations and value.

Like the mathematical modeling discussed in the article, these forms of quantitative analysis serve a fundamental purpose in ENR decision making—to allow policymakers to make decisions on more than mere guesswork. It is hard to dispute the notion that some reasonably good information is better than none at all. With this in mind, I would like to offer four lessons on the use of mathematical models that could supplement those in "Useless Arithmetic":

1. Beware of false precision.

The temptation for analysts seeking to influence decisions, and for policymakers needing to establish a scientific basis for those decisions, is to attribute more certainty than is warranted to quantitative analytical tools. As the authors rightly point out, those who give in to these temptations risk making poor decisions. Any use of quantitative models should include a careful assessment of uncertainties and limitations. Candor and transparency are essential. Neither analysts nor policymakers should be allowed to attribute more precision than is reasonable.

2. Compare modeling to other analytical tools that are available.

Part of the argument made by Pilkey-Jarvis and Pilkey is that people turn too quickly to mathematical models when other tools may be more appropriate. Clearly, there are times when a rush to quantification is not the best path to a sound decision. In such circumstances, using the model becomes the end rather than the means. Other sources of information may be available and more reliable than formal models. Smart decision making involves using the right kinds of information, as well as making the best choices based on whatever information is available.

3. Insist upon transparency in assumptions, limitations, and conclusions.

Probably the greatest risk of relying too much on these models is providing an exaggerated sense of confidence in the face of considerable uncertainty. Policymakers need to insist upon absolute transparency in the assumptions that underlie the model and their effects on the results. They need to understand the data and methodological limitations in the analysis and to communicate them as part of their decisions. Likewise, the analysts conducting and reporting on the modeling have an ethical obligation to inform policymakers fully of the limitations in the models.

4. Test and recalibrate the model regularly against events and facts on the ground.

The authors make an excellent point about the value of an adaptive approach to ENR decisions. Predictive models should regularly be tested and recalibrated in response to new information. The model should be seen less as a source of truth or facts than as a best guess that is constantly subjected to reevaluation and refinement. Scientific observation may be an alternative to modeling, as the authors point out, but it also offers a valuable supplement to traditional scientific study and a way to test the reliability of the model at key points in the decision-making process. The models can provide a long-term picture that is regularly tested with data gained from experience.

The big ticket item in the use of mathematical, predictive modeling is—of course—to understand long-term changes in the global climate and their consequences. The authors give the United Nation's Intergovernmental Panel on Climate Change high marks for transparency, stating that "these modelers agonizingly, and in great detail, list and evaluate the assumptions and model simplifications, strong and weak, behind the predictive models." Still, they do not exempt climate analyses from their general characterization of predictive models as "useless arithmetic."

Would we be better off without quantitative models of climate change and its effects? It is difficult to see how we would. Any sense that climate change is an issue worth responding to would have to wait for direct, observational, scientific evidence that global temperatures were rising over the long term and had consequences worth worrying about. One could always argue that direct evidence of the effects of warming—sea-level rise, melting of polar icecaps, slow changes in patterns of disease, unexplained changes in ecosystems—were part of normal climate cycles, variable weather patterns, or due to other causes. Indeed, skeptics offer these arguments even now. Without the long-term predictive insights offered by mathematical models, we may have been responding to a series of apparently unconnected events and whatever action has been taken would have been delayed by decades, if not longer.

Despite the many persuasive points the authors make about an excessive reliance on quantitative models, readers are not sure at the end just where they stand. Most of the article offers sound advice on the limitations of such models. The authors make a strong case that, especially in the hands of untutored or self-serving policymakers, such models offer a seductive air of precision that is at best misleading and at worst possibly flat out wrong. This is good advice and should be drilled into anyone who is involved in making policy decisions. But they seem to go farther when asserting, as they do in the final paragraph, "their strong preference for relegating quantitative predictive mathematical models to the historical dustbin of failed ideas.…" Should we be relegating what may be a valuable, if often limited, analytical tool so readily to the dustbin?

My differences with the authors of this provocative article turn not so much on their criticisms, which for the most part make sense, but on the notion that we generally would make better ENR decisions without these models. Would we have been better off over the last 20 years without the predictions that have been made available from the climate models? It is difficult to see how we would have been. The label "useless arithmetic" takes us too far. It would be much better to say "often useful, but handle with care." Used responsibly, with an awareness of their strengths and limitations, by well-informed policymakers who test the models against events on the ground, mathematical models offer a useful tool in the analytical arsenal for ENR decision making.

NOTE: The views expressed in this commentary are those of the author and not necessarily those of the U.S. Environmental Protection Agency.