In “Useless Arithmetic,” Robert Durant asks us to consider his central question, why bother with messy fieldwork and expensive laboratory studies when mathematical models can provide a cheaper and easier way gain numerical judgments about natural process

As the title of their article suggests, Pilkey-Jarvis and Pilkey find little use for virtual-reality computer models and no use for them when the availability of real-world data is a viable alternative. They lament and offer evidence that quantitative modeling by ENR managers has not only become accepted by policymakers, but also has become their preferred form of information. Policymakers pressed to make high-stakes and often costly management decisions desire concrete numbers on which to base their decisions. In effect, policymakers desire to justify their environmental decisions based on the best available science, and they generally equate numbers with science (the more digits past a decimal point the better). The authors make the case that modelers are treated by policymakers as modern oracles, prophets of a new mathematical religion, a religion that promises access to scientific truths and a direct window to the future.

Aside from their efforts to demystify mathematical modeling for policymakers, Pilkey-Jarvis and Pilkey perform a great service by reminding us that there are no crystal balls for the natural world. Given the complexity of nature and the countless relationships within natural systems, computer models can never fully capture natural phenomena, and they are rarely equipped to make anything but the most general predictions about the future. The authors warn us that in the world of ENR modeling, complexity virtually guarantees inaccurate estimations. According to them, the overarching problem rests in the fact that the more complex the natural phenomenon, the less reliable a computer model is likely to be. And unfortunately, the more complex the model, the more it is likely to be accepted by policymakers who are unduly cowed by modelers, because policymakers believe that they lack the expertise to understand the model and the assumptions on which it is built. In other words, it is not just that complexity leads to inaccuracy, but complexity also leads to blind acceptance of inaccurate findings.

Most of the authors' ten lessons regarding computer modeling relate to this central problem. As noted, they usefully offer suggestions to policymakers wanting to penetrate this arcane world and to be less at the mercy of modelers. Their seventhlesson, however, goes beyond lessons and makes a serious charge. As such, this claim warrants a closer look. It suggests that computer models might not only be inaccurate, but under certain circumstances, they might intentionally and systematically bias information in favor of one side of a policy debate. It is one thing to say a computer model is a poor measure of the natural world and hence an inappropriate management tool. It is something altogether different, however, to suggest that computer modeling has become a tool of the political process.

To make their case, Pilkey-Jarvis and Pilkey refer to the collapse of Canada's once great fishery off the coast of Labrador and Newfoundland. They explain that "politicians used the optimistic models as fig leaves to hide behind and ignored catch reports (i.e., actual field observations of cod populations) indicating that a catastrophic collapse was on its way." Later in the same section, they explain that "scoundrels" such as the Army Corps of Engineers sometimes misuse computer models to justify pork-barrel projects that have produced few tangible results. They go on to illustrate how the paid consultants of the Bureau of Land Management created models that found "the truth according to their clients' needs."

This "lesson" is bold and truly irreverent on its face. But this is especially true for those who worship at the digital altar of computer models. In making the points in lesson seven, Pilkey-Jarvis and Pilkey identify and treat each of the following issues separately: computer models can offer 1) fig leaves for politicians, 2) refuge for bureaucratic scoundrels, and 3) manufactured truths for industry consultants. They suggest that bureaucratic agencies like the Bureau of Land Management and the Army Corps of Engineers have "perverse incentives" to misuse computer models.

I agree. But does this indictment really go far enough? I argue that Pilkey-Jarvis and Pilkey do not fully consider the perverse incentives that lead politicians to accept the perverse stories of bureaucratic scoundrels and industry consultants. A look at the misuse of computer modeling in the nationally prominent and important Chesapeake Bay restoration effort illustrates my point. Since its creation in 1983, the Chesapeake Bay Program has been tasked with implementing the various Chesapeake Bay Agreements—voluntary agreements that were designed to provide direction to the multi-state restoration effort. The Bay Program currently receives more than $20 million per year in funding from the federal government and has a professional staff of over 100 employees. What it lacks, however, are independent regulatory powers. The Bay Program remains a non-regulatory body that lacks the authority to limit pollution, to restrict harmful development, or to manage the Bay's living resources. Stated differently, the Bay Program is an environmental agency with no regulatory powers, created to enforce Bay Agreements which are not legally enforceable. It is the sticky sweet stuff of light green environmentalism, but certainly not a recipe for success.

For more than a decade and a half, Chesapeake Bay resource managers have been discussing and debating numbers when it comes to the Bay restoration effort. The most important of these numbers was the 40 percent goal of reducing harmful nitrogen and phosphorous pollution from the Bay, a goal the EPA's Chesapeake Bay Program claimed it narrowly missed in 2000. After the 2000 goal came and went and the Chesapeake Bay failed to show meaningful signs of improvement, more and more people began to ask questions. How, they asked, could the Chesapeake Bay Program claim it was moving closer and closer to reaching its pollution reduction goals, while the Chesapeake Bay itself showed no significant signs of improvement?

As Pilkey-Jarvis and Pilkey might predict, much of the disconnect between the numbers and the results can be explained by the fact that the Chesapeake Bay Program relied heavily on computer models to assess the Chesapeake Bay's water quality. As they argue, despite their complexity, the computer models underpinning the program's assessments are nothing more than a massive accounting system. Researchers divide the watershed—the land that drains into the Bay—into specific land-use types (e.g., forested, urban, pasture, cropland, cropland under best management practices [BMPs], among others). Researchers combine land-use data from across the Chesapeake Bay watershed with estimates of the likely amount of pollutants that typically come from each land-use type.

As Pilkey-Jarvis and Pilkey correctly argue about computer models generally, the biggest problem associated with them is that these models are only as accurate as the estimates that are fed into them. The Bay Program's models are based on thousands of assumptions regarding how the land is actually being used and the amount of pollution that is likely derived from particular land types. If the assumptions that are fed into the Chesapeake Bay model are incorrect, the model, regardless of its sophistication, will yield faulty results. And even if accurate, the more assumptions that are fed into the model, the more likely the model will produce inaccurate results. Thus, when layer upon layer of assumptions are entered into a model, as is necessary when modeling a watershed as large and complex as the 64,000-square-mile watershed that drains the Chesapeake Bay, it virtually guarantees that the results will provide only a rough estimate of what is actually taking place in the water.

While honest shortcomings in computer modeling can explain why computer models generally fail to reflect real-world conditions, they cannot explain why a government agency persistently erred in its own favor, regularly overestimating its pollution reduction success. Thus, it is not the case that the Chesapeake Bay Program's arithmetic was useless; it is rather that its models were quite useful for those who favored inaction. How so? The Bay Program overestimated the reductions in nitrogen and phosphorous from 1987 through the late 1990s when it insisted on only calculating what it then called "controllable" nutrients. It also overestimated the reductions in nitrogen and phosphorous in 1999 when it reported not only that it would achieve its 40 percent reduction goal for phosphorous, but also that it would come close to meeting its reduction goal for nitrogen. And more recently, it overestimated the environmental benefits from agricultural BMPs.

Had the errors overestimated success in some cases and underestimated it in others, one could attribute the "mistakes" to simple errors in the model. That the models regularly overestimated the success of the Bay Program suggests something much more troubling and consonant with the lessons drawn by Pilkey-Jarvis and Pilkey in their article. Behind each overestimate of progress are human judgments that are better explained as the outcome of political pressures than the outcome of sound scientific judgments.

Take one of the most recent "mistakes" as an example—the overestimation of gains derived from agricultural land under BMPs. While it was politically expedient to assume the BMPs were completely implemented and perfectly maintained, as the Chesapeake Bay Program's models assumed, it was not scientifically justifiable. The Bay Program chose an unrealistic best case scenario that painted their restoration efforts in the best possible light, and in doing so they provided elected officials with the positive results that they desired and that the public demanded. It is also wrong to give the Chesapeake Bay Program credit for identifying its problems and working busily to correct its errors. As far back as 1991, in Tom Horton's first edition of Turning the Tide, he noted that the Bay Program's "accounting procedures are almost certainly overstating progress in keeping agricultural nutrients out of the water" (54). In my more recent book, Chesapeake Bay Blues, the balance of evidence revealed that the Bay Program's "models overestimate nutrient reduction efforts" (2003, 66). On July 18, 2004, a front-page article in the Washington Post ran under the title, "Bay Pollution Progress Overstated: Government Program's Computer Model Proved Too Optimistic" (Whoriskey 2004). A formal investigation by the Government Accountability Office reported in October of 2005 that misuse of computer modeling failed to "provide effective and credible information on the current health status of the bay" (GAO 2006, 2). As recently as December of 2007, a team of researchers at the University of Maryland found that the Bay Program continued to inflate the impact of several of its key pollution reduction efforts, leading to this Washington Post headline, "Cleanup Estimate for Bay Lacking: EPA Program's Computer Formulas Called Optimistic" (Fahrenthold 2007).

Perhaps the models served as a refuge for bureaucratic "scoundrels," as Pilkey-Jarvis and Pilkey suggest is quite common, by justifying the continuation of an environmental program (the Chesapeake Bay Program) that had produced few accomplishments. But given the amount and persistence of evidence of the misdeeds, even the least curious of politicians could have seen through the facade were they interested in looking. The persistence of abuse of computer modeling at the Bay Program, and more generally as the authors maintain in this article, suggests that the agency produced results that were consistent with the desires of the elected officials to whom they reported. The Bay Program misused computer models, scholars and the media reported the misuse, and the Bay Program continued its practices with slight modifications.

The result has been pernicious and offers yet another cautionary tale to those offered by Pilkey-Jarvis and Pilkey for anyone interested in protecting precious ENR resources. The models have fostered a culture of inaction that allowed elected officials to put off difficult decisions dealing with such enduring problems as agricultural reforms, sewage upgrades, and stormwater upgrades. It is not that elected officials were duped by the models; they welcomed the results, put the best spin on the findings, took credit for the "successes," and rewarded the Bay Program with continued funding. All the while, the water quality of the Chesapeake Bay continued to decline. Thus, the Chesapeake Bay saga suggests that computer models also can provide a refuge for scoundrels in elected office. It is not that mathematical models are "useless," as the title of this essay suggests; it is that they can be used in ways posing a real threat to the wise management of natural resources.

References

Ernst, Howard R. 2003. Chesapeake Bay Blues: Science, Politics, and the Struggle to Save the Bay. Lanham, MD: Rowman & Littlefield.

Fahrenthold, David A. 2007. Cleanup Estimate for Bay Lacking: EPA Program's Computer Formulas Called Optimistic. Washington Post, December 24, B01.

Horton, Tom. 1991. Turning the Tide: Saving the Chesapeake Bay. Washington, DC: Island Press.

U.S. Government Accountability Office (GAO). 2006. Chesapeake Bay Program: Improved Strategies Needed to Better Guide Restoration Efforts. Washington, DC: U.S. Government Printing Office. GAO-06-614T. http://www.gao.gov/new.items/ d06614t.pdf.

Whoriskey, Peter. 2004. Cleanup Estimate for Bay Lacking: EPA Program's Computer Formulas Called Optimistic. Washington Post, July 18, A01.