Pilkey-Jarvis and Pilkey [“Useless Arithmetic: Ten Points to Ponder When Using Mathematical Models in Environmental Decision Making”] offer insight and prescriptive crispness on the public sector’s use of quantitative models for decisions

Alan Lyles is the Henry A. Rosenberg Professor of Public, Private, and Nonprofit Partnerships in the University of Baltimore (UB) School of Public Affairs’ Health Systems Management program. He has published and lectured extensively in the U.S. and abroad. He is a senior fellow at the Hoffberger Center for Professional Ethics at UB and the Center on Drugs and Public Policy at the University of Maryland School of Pharmacy, and a Docent at the University of Helsinki. Formerly, he was on the Department of Health Policy and Management’s faculty at the Johns Hopkins Bloomberg School of Public Health, and he was the Yale Gordon College Distinguished Chair in Research and Scholarship.

E-Mail: calyles@ubalt.edu

ilkey-Jarvis and Pilkey offer insight and prescriptive crispness on the public sector's use of quantitative decision models. Their observations are, in part, a caution against over-reliance on a prediction method or explanatory model. They note that the potential strength of quantitative models may be cancelled by uncritical, mechanistic application and acquiescence to their outputs. These are important caveats.

But the evidence they present also raises questions, I believe, for MPA and MPP education itself (henceforth, graduate programs). I will focus on only one important question: Should individual programs, or even the National Association of Schools of Public Affairs and Administration (NASPAA), require graduate programs to view the quantitative reasoning applied in computer modeling as an expected core competence of our program graduates? I argue that the answer to this question is a resounding "yes." From a series of executive orders issued over decades to a host of regulatory actions, the use of quantitative models has been affirmed and is unlikely to disappear.

Consequently, we do not serve graduates of our programs well by not preparing them to become intelligent consumers of this information. In preparation for careers in public administration and public policy, students typically must complete a graduate-level course in statistics and often an additional course in analytic techniques, which includes some modeling. Are these rites of passage or are they legitimate core skills?

Nostalgie de la boue has a long lineage but seldom reverses developments. Lamenting the change to more specific and developed research funding procedures, Curt Richter wrote, "In making an application for a grant before World War II, a few lines or at most a paragraph or two sufficed for the experimental design…." Analogous to Pilkey-Jarvis and Pilkey's arguments against quantitative models in favor of reality-based qualitative research, Richter argued against formal scientific funding proposals. As he put it, "Let us try to give the funds for research with as few strings attached as possible, without asking a man [sic] exactly what he is going to do and why" (1953, 91-93). Today, do we see this as the preferred approach to grant and contract management? Absolutely not. The standards for professional competence in research are not static—they evolve. So, too, must the definition of professional competence evolve in our graduate programs to meet the prominence of quantitative modeling across a broad spectrum of policy arenas.

While there are clearly some circumstances in which quantitative modeling is inappropriate, there are many in which it is essential. A well-rounded professional should be able to do a differential diagnosis and select the most appropriate method from multiple possible approaches. Pilkey-Jarvis and Pilkey identify two fundamental constraints on the appropriate use of quantitative models: 1) innumerate (Paulos 2001) decision makers, and 2) a blind focus on the model to the exclusion of the modeling process itself. If correct, these are strong indictments of professional education and of selection and promotion practices within the profession.

First, although the authors identify quantitative modeling as occurring in "the rarefied and often intimidating realm of mathematical complexity," this experience is not inevitable. Instead, what the authors describe is a breakdown in critical thinking by public administrators and policy analysts, coupled with low competence in understanding, assessing, and communicating risk. As an antidote, Pilkey-Jarvis and Pilkey argue for qualitative models that "eschew precision in favor of more general expectations grounded in experience," noting that "in the real world, we cannot predict the unpredictable anyway" and that "models are no guarantee to policy makers that the right actions will be set in policy." Of course, no method guarantees the "correct" result. However, graduate programs can help decision makers improve the utility of these models by developing skills in critical, systematic thinking and by helping them to learn how to actively manage these projects as they would other projects.

Second, our students should learn that it is an irresponsible and feckless decision maker who uses quantitative models without being actively involved in their development, at least at the conceptual level, from start to finish. Even, or especially, at the project's conclusion, communicating the results with clarity and fidelity are critical technical skills (Redelmeier et al. 1997), but these should not be beyond the reach of professional public administrators and policymakers. Surprisingly, Pilkey-Jarvis and Pilkey argue the contrary: "They [policymakers and public managers] should train the public to expect and pay attention to this [emphasis added] type of prediction…." But isn't training the public possible only if public managers themselves understand the strengths and limitations of the models and if their explanations are balanced and accurate?

Managing Projects

Does all this leave us in the position of decision-making nihilism? I don't think so. Project-management skills do not necessarily include the knowledge to do all aspects of a project oneself. Rather, basic managerial skills are required: clarify objectives, set expectations, monitor and intervene if necessary, and hold those who perform the work accountable. Beyond this, evolving de facto standards of practice for quantitative modeling, including the development of a consistent set of criteria for mandatory elements and how they are presented in final technical reports, should be clear to our graduates and to all who manage such projects or practice in the field. Moreover, and central to NASPAA's concern about ethics, managing potential or real conflicts of interest is increasingly part of this supervision.

In my judgment, the effective performance of these collective functions would address many of the pathologies Pilkey-Jarvis and Pilkey discussed in their article. For example, the authors identify problems with black-box products for which the internal algorithms are inaccessible. But as in other contract and bidding arrangements, our students and public managers more generally should understand that negotiating (or mandating) contractual terms and managing to specified deliverables routinely should include prior performance scores. If transparency is the keystone for quantitative models, then accountability for products is its mortar. Over time and effectively employed, prior performance weighting can discourage low-performing consultants.

Nor are Pilkey-Jarvis and Pilkey's recommendations to substitute field experience for mathematical modeling not without problems of their own. Sometimes substitution of models for "field and laboratory studies" can be a consequence of regulatory requirements, rather than merely being relatively easy and cheap. Nor does the law ask modelers to determine the optimal choice. For instance, FDA approval of new molecular entities (drugs) only requires demonstration of superiority to placebo, not to the best active therapeutic alternative. Post-marketing determination of the relative value of approved, therapeutically related pharmaceuticals is frequently constrained to modeling based on these indirect results. Time compression as discussed by Pilkey-Jarvis and Pilkey is also relevant to the dilemma of determining the safety and efficacy of pharmaceutical products whose studies may have short observation periods compared to the times over which the drugs may be intended to be used post-marketing—for some chronic illnesses this may be a lifetime. Indeed, numerous decisions about potent pharmaceutical products must be made, many of which can only be based on quantitative models. When comparing medicines, how is the pre-marketing efficacy determined under ideal conditions for a limited spectrum of patients to be translated to post-marketing effectiveness under the conditions of usual community practice? What are decision makers who influence the existence, extent, and specific content of health insurance program benefits to make of such quantitative translations? Under these circumstances, an appeal to experience is an invitation to unmanaged conflicts of interest and perhaps decisions based on less complete information than is actually available. And graduates of our masters programs should appreciate this.

Managing Interests

The authors identify the Army Corps of Engineers and the Bureau of Land Management's perverse incentives and deleterious results for the environment. Nor, at the core, are these any different from managing other situations in which there are real or perceived conflicts of interest. Or are the authors identifying quantitative models as yet another mechanism of regulatory capture?

They write accurately that "mathematical models are closely held in the files of consultants and environmental firms and are not revealed—even to paying customers." This is not a problem unique to environmental decision making. For example, questions about whether specific pharmaceutical products are covered as insured benefits and, if so, to what extent, are largely determined in the marketplace (excepting the federal supply schedule). Yet purchasers in the marketplace have asymmetric and often incomplete information regarding products with possibly comparable therapeutic uses. In the United States, however, there is no central governmental role as there is in other nations which require comprehensive drug dossiers to determine whether a product will be covered or subsidized, and possibly priced.

Yet another approach for dealing with opaqueness in quantitative modeling involves the transparency afforded by competing quantitative models. During the federal healthcare reform initiative in the early 1990s, for example, having a quantitative analysis to challenge others' positions was a requisite for policy combat among the stakeholders involved. That some of these competing models were developed for different stakeholders in the battle by the same consultants can perhaps prove Pilkey-Jarvis and Pilkey's point. But it can also rebut it because of the transparency of model assumptions afforded by "dueling" models. The precise questions posed, the boundaries on the questions, the methodology, time horizon, assumptions, sources, and estimates of uncertain quantities of different models became the grist of policy debates.

An example of a market-based approach to transparency-enhancement involves the issuing of guidelines by professional associations. For example, the Academy of Managed Care Pharmacy has promulgated a recommended format in the United States by which health plans and pharmacy benefit-management companies can make consistent, comprehensive requests for information to pharmaceutical manufacturers. Significantly for transparency, however, the model that is used in the dossier is part of a larger process in which information needs and model decisions can be discussed between provider and recipient. The accompanying technical report also requires an unlocked electronic version of the model that the manufacturer used in developing the dossier. Thus, the recipient can make an independent assessment of the validity of how the model was constructed and performed (Academy of Managed Care Pharmacy 2005). Somewhat surprisingly, this approach has had extensive acceptance and is used in health plans and by some states to establish information for these drug-coverage decisions.

Considering Alternatives

As Maslow noted, "I suppose it is tempting, if the only tool you have is a hammer to treat everything as if it were a nail" (1966, 15-16). An answer to the Pilkey-Jarvis and Pilkey question, "Is there an alternative to quantitative modeling?," is both "yes" and plural. One thinks immediately of adaptive management (as they suggest), scenario analysis, and prediction markets. Adaptive management as advocated by the authors meets some shortcomings of quantitative models. However, and ironically, adaptive management also presumes skills that would remedy many of the deficits the authors identify with the use of quantitative models. But the matter is more fundamentally a question of diagnosing the decision(s) to be made and selecting the most appropriate technique, whether it is quantitative, qualitative, or mixed. Scenario analysis, a rigorous qualitative approach, allows for the inclusion of personal insights and development of decisions which may be more robust under uncertain future conditions. Meanwhile, prediction markets offer a newer, less developed approach to organizing the collective judgment of individuals about some uncertain, perhaps future event. Consider, for example, how prediction markets have demonstrated surprisingly robust results for futures ranging from political, financial, sales, and even a Policy Analysis Market under DARPA (the Defense Advanced Research Projects Agency) (Wolfers and Zitzewitz 2004). Despite their somewhat black-box approach, using speculators who trade in contracts whose payoff depends on unknown future events, prediction markets are in some ways more transparent and amenable to constrained resources than are quantitative models.

Implications for Graduate Education

The authors' perspective in "Useless Arithmetic" is also math-averse: "Because the math behind the models is impervious to the general public and policy makers, and even to other scientists, models are easily distorted to provide inaccurate cost estimates or overly optimistic environmental impact estimates." Their thinking echoes Colonel G. O. Ashley's "Declaration of Independence from the Statistical Method":

1. Numbers are a small tool of thought…but…must not be allowed to become a master of thought;

2. Numbers stand as symbols for things, but they are never the things they are frequently set to stand for;

3. Skill at numbers is not a mental magic, nor is that skill necessarily transferable when other disciplines of thought are required;

4. Statistical methods have a jargon designed specifically and with studied calculation and forethought to keep the methods obscure to "outsiders"; and

5. Decisions based solely on numbers DO NOT SOLVE problems. Such decisions have only solved equations. (1964, 83-84)

Is the practice of economic reasoning, a related component of quantitative models, any better? Paul Ferraro and Laura Taylor (2005) suggest a near complete failure of even advanced economic education to prepare graduates to reason using the idea of opportunity cost—possibly one of the most important contributions and basic ideas in economics. In their survey, students who had taken an economics course chose the correct answer less often (7.4 percent) than did students who had not (17.2 percent). Even more disconcerting, correct answers by PhD economists (21.6 percent) did not differ from those by "undergraduates with no prior exposure to economics."

But rather than do away with mathematical modeling, the supply-side risk identified by Pilkey-Jarvis and Pilkey is a strong argument for more demanding and discerning consumers (that is, better education) and stronger management involvement in their development. Indeed, lesson 9 ["The mathematically challenged need not fear models and can learn how to talk with a modeler"] should resonate with educators. Too often what is identified as lack of mathematical or programming skills is actually a deficit in critical thinking and clarity in decision makers' understanding of the underlying process. Do current graduate program curricula and learning objectives produce graduates who are prepared to identify appropriate decisions for quantitative modeling, to provide direction and oversight to those who perform the modeling work, and, finally, to make informed decisions and implement them based on independent judgment and knowledgeable assessments of the strengths and limitations of the process? In my judgment, the answer is "yes."

Still, the points made by Ashley, Ferraro, and Taylor cannot be dismissed and speak to the way graduate programs must address modeling literacy as a core competence of professionalism in the field. Specifically, their critique calls for a different approach to educating professionals in training and establishing standards of practice that facilitate critical and independent external appraisal of modeling results. What is more, there are models from economics education that might help us move in this direction. For example, Robert Frank (Henrietta Louis Johnson Professor of Management and Professor of Economics, Cornell University) and Ben Bernanke (current Federal Reserve Chairman) (2004) determined that conventional teaching methods for economics were ineffective and offered a promising alternative to procedural quantitative instruction: narrative teaching. Frank and Bernanke dropped the traditional mathematical approach to introductory economics, focusing instead on narratives from daily life that elaborated a limited number of core principles: scarcity, cost-benefit, unequal costs, comparative advantage, increasing opportunity cost, equilibrium, and efficiency.

It seems to me that this process of narrative learning is directly relevant to the quantitative education and training of public administrators if their experiences in the classroom are to be durable and apply to their thinking in practice. Narrative learning requires a student to identify, and then to write about, an interesting question that demonstrates one (or more) of the main course concepts, such as opportunity costs or marginal costs. By using narratives, having students identify the questions, and reinforcement through applications over the term, students become more active learners and appear more likely to retain the practice of economic reasoning.

One outcome of this approach is Frank's 2007 best-selling book, The Economic Naturalist, a collection of insightful and interesting narratives that creatively question and explain everyday enigmas by applying core economic principles. Shouldn't this approach be considered for the quantitative analysis courses in our graduate programs? For most students, the gain in understanding major concepts through narratives would more than offset the loss of time for procedural learning that dominates the conventional approach. In public administration and public policy, quantitative analysis cannot produce certainty, but it can structure and make transparent how the relevant main factors in a decision relate to one another, as well as the most likely result of multiple sources of uncertainty in the model.

Conclusion

Software and methodological advances put even relatively sophisticated quantitative modeling within the reach of more consultants, academics, interest groups, and public administrators. Pilkey-Jarvis and Pilkey performed a service for the public administration profession by questioning the role of these models and offering lessons on their boundaries. Their arguments against quantitative models would, however, benefit from being more nuanced. A celebration of limitations discourages progress in professional education and practice. Students look to faculty and practitioners for cues about what is important and as models for how to be effectively engaged in performing the business of government. The present work suggests that quantitative modeling is a failed experiment that may be safely discarded, perhaps ignored. What does this say to students and junior practitioners? I believe that it says that they should be looking for a graduate degree that prepares them to be better consumers of quantitative models, rather than reject them outright.

As I have written elsewhere in regard to pharmaceutical education:

The basic requirement of professionalism is the exercise of independent judgment, but for many public administrators and policy analysts, a wealth of pertinent information from…analyses currently lies beyond their reach. University programs, continuing education activities, workshops, and publications need to include these professionals among their target audience and set learning objectives that will fill the gap in competence. (2007, 2095)

I would contend that it needs to be no more acceptable to declare mystification over managing and using quantitative models for decisions than it is to plead shyness for not speaking in groups, or poor spelling and grammar for not communicating in writing. Professionals in training and in practice require familiarity and comfort with managing activities that may require the development or use of quantitative models. Otherwise, we fail both, needlessly handicapping the next generation of public administrators and perpetuating this dysfunctional asymmetry between those who produce quantitative models and those who must make decisions based on them.

References

Academy of Managed Care Pharmacy. 2005. The AMCP Format for Formulary Submissions. http://www.fmcpnet.org/data/resource/Format~Version_2_1~Final_Final.pdf.

Ashley, Garland O. 1964. A Declaration of Independence from the Statistical Method. Air University Review, March/April.

Ferraro, Paul J., and Laura O. Taylor. 2005. Do Economists Recognize an Opportunity Cost When They See One? A Dismal Performance from the Dismal Science. Contributions to Economic Analysis & Policy 4(1): Article 7.

Frank, Robert H. 2007. The Economic Naturalist: In Search of Explanations for Everyday Enigmas. New York: Basic Books.

Frank, Robert H., and Ben S. Bernanke. 2004. Principles of Microeconomics, 2nd ed. New York: McGraw-Hill.

Lyles, Alan. 2007. Public Administration and Public Policy: Is Pharmacoeconomic Evidence Considered Indecisions on Pharmaceutical Policy? Clinical Therapeutics 29(9): 2094-95.

Maslow, Abraham H. 1966. The Psychology of Science: A Reconnaissance. New York. Harper & Row.

Paulos, John A. 2001. Innumeracy: Mathematical Illiteracy and Its Consequences. New York: Hill and Wang.

Redelmeier Donald A., Allan S. Detsky, Murray D. Krahn, David Naimark, and Gary Naglie. 1997. Guidelines for Verbal Presentations of Medical Decision Analyses. Medical Decision Making 17(2): 228-30.

Richter, Curt P. 1953. Free Research vs Design Research. Science 118(3056): 91-3.

Wolfers, Justin, and Eric Zitzewitz. 2004. Prediction Markets. Journal of Economic Perspectives 18(2): 107-26.