Methodology

Undergirding the Calvert-Henderson Quality of Life Indicators is a research methodology developed in 1994 to organize, synthesize, and analyze a host of statistics on the social, economic and environmental state of the nation in ways that allow bytes of data to be transformed into meaningful indicators that can help citizens understand complex phenomena. The Calvert-Henderson models serve as a framework through which a complex issue can be condensed into discrete components and understood by people who are not necessarily well-versed in the topic. Each model presents both a systematic method of capturing the relationships among the variables of interest and a technique for handling a host of statistical data simultaneously.

The methodology is described in detail below and in the Calvert-Henderson Quality of Life Indicators book.

Research Methodology
by Patrice Flynn, Ph.D.

The American public has become accustomed to social scientists disagreeing with each other about the condition of the nation. The print and broadcast media presents information on the social, economic, and environmental state of the country in a "cross-fire" format. This pits individuals with different research findings against each other in public debate. Social scientists invited to participate in such media events are trained to ask beforehand on which side of an issue they will be positioned.

As a result, public faith that scientists somehow cooperate with each other in enlarging and advancing our collective understanding of the world is giving way to doubt as new knowledge assaults existing knowledge. We are challenged to ask whether science is indeed progressive and cumulative. Are forays into the unknown characterized by researchers adding a modicum of new and better data to advance a more complete explanation of reality? Or does science advance when scientists modify or even discard previously established truths and disagree vehemently with each other?

The research methodology developed for the Calvert-Henderson Quality of Life Indicators seeks to shed light on when and where social scientists agree and disagree with each other about the overall conditions in the United States. The approach builds on the knowledge of scholars who have devoted their entire careers to the fields of study (i.e., domains) examined in this volume. Each author was presented with the challenge of: (a) identifying and rectifying apparent clashes in our public discussions about the respective domain; (b) educating the readers about the state of the field; and (c) taking us to the cutting-edge of thinking about the topic as it fits into the whole. Methodologically, the challenge was to condense complex issues into discrete components to be understood and used by the broad public. The aim was to provide meaning and context for an information driven society, using rigorous empirical techniques.

The Calvert-Henderson Quality of Life Indicators research methodology combines reliable and verifiable numerical results with new research methods that provide an antidote to the increasingly chaotic output of contemporary research on well-being. The methodology may also serve to renew the public's belief in and support for nonpartisan, rigorous scientific research to inform practitioners, policymakers, funders, scholars, and leaders about what is happening in the nation.

Below is a summary of the research methodology developed for the Calvert-Henderson Quality of Life Indicators. The methodology is grounded in the seminal work of Thomas Kuhn, described in Section I, which provides a theoretical framework for the systems approach we adopted. Section II discusses how this approach adds transparency and traction to current measurement efforts (e.g., index analyses and community-based indicators projects) while advancing the rigor with which we develop quality of life indicators. Section III identifies the institutional structure within which the research was conducted and the experts who created the indicators. Section IV describes the Calvert-Henderson models and underlying data and their utility in developing rigorous measures of well-being.

I. Transitions in Science
The research methodology designed for the Calvert-Henderson Quality of Life Indicators is grounded in the seminal work of Thomas S. Kuhn as articulated in The Structure of Scientific Revolutions (1962). Dr. Kuhn, the Laurance S. Rockefeller Professor of Philosophy at the Massachusetts Institute of Technology and later a Fellow of the Institute for Advanced Study at Princeton University, examined the process of transitions in science and how new theories emerge to explain the evolving world. Struck by the number and extent of overt disagreements between social scientists about the nature of legitimate scientific problems and methods, Kuhn attempted to discover the source of such differences. He came to recognize the role of scientific research in the development of new paradigms, defined as "universally recognized scientific achievements that for a time provide model problems and solutions to a community of practitioners" (1962:viii).

Kuhn advocated a reorientation in the evaluation of familiar scientific data as they continually are impacted by changing external intellectual and economic conditions. Along with fellow historians of science, he posited that perhaps science does not develop through the accumulation of individual discoveries and inventions. Rather, research will reveal fundamental novelties or anomalies that challenge substantive conclusions to key scientific questions. When this occurs, the natural tendency is for the scientific community to defend its preconceived assumptions. However, the anomalies will not be suppressed for long. At some point scientists begin the "extraordinary investigations" that lead to a new basis for the practice of science.

Each famous episode in scientific development necessitated the research community's rejection of a time-honored theory. New theories imply a change in the rules governing the prior practice of science. Such paradigms or theories add value by drawing from an existing body of concepts, phenomena, and techniques to help explain new facts or information. Kuhn writes that "in the absence of a paradigm or some candidate for paradigm, all the facts that could possibly pertain to the development of a given science are likely to seem equally relevant" (p. 15).

This initial volume sets the stage for further scientific developments in the field of rigorous empirical measurement of quality of life. The Calvert-Henderson Indicators put forth a new model (or pattern) to organize our thinking about quality of life. We have not yet achieved a paradigm that will guide subsequent scientific research in the field. However, efforts to re-define quality of life are at sufficiently high levels to suggest that the development of new models and theories about quality of life is timely and useful to fellow researchers and practitioners in the field. Ours is one of many contributions to this emerging field of scientific inquiry. It allows us to investigate some parts of society in detail and depth that would otherwise be unimaginable.

II. Measuring Quality of Life
There are two standard approaches to measuring well-being employed by social scientists today. The first is the index number, which tracks changes in a selected phenomena over time. Indices are common in economic analysis. The Gross Domestic Product index, for example, measures production; the Consumer Price Index measures inflation. In the language of economists, the fundamental problem upon which the index analysis rests "is that of determining merely from price and quantity data which of two situations is higher up on an individual's preference scale" (Samuelson 1947:146-147).

Index number theory is limited in that we assume an individual's tastes do not change in the period under consideration or if more than one person is considered, that their tastes are identical. Another limitation is that unless the reader is thoroughly familiar with the model employed to develop the index it is not transparent what variables are included and excluded or the relative weights assigned to each variable. Scholars in the field of quality of life have documented these and other methodological difficulties over the past few decades in such journals as Social Indicators Research and Social Indicators Network News. While powerful when fully understood and well-fitted to the data, index numbers can be very limiting when trying to understand a topic about which a person is not familiar.

The second approach to measuring quality of life comes from the field of community indicators, currently involving over 200 groups in the United States. During the 1980s and 1990s, the quality of life movement re-gained the attention of citizen groups, scholars, practitioners, policymakers, and private foundations interested in alternative measures of well-being beyond those created by economists and other social scientists (Sawicki and Flynn 1996). An increasing number of groups began redefining well-being at the neighborhood, community, or city levels in ways that expand the traditional parameters of the National Income and Product Accounts (NIPA). A host of new and innovative data sets are being identified, collected, and analyzed. The quality of life literature is expanding rapidly as the concept is integrated into the mainstream of life in America and as the growing movement for livable communities intersects further with local, state, and national policy-making.

A certain level of frustration occurs in the process of pooling information on quality of life in this manner. In the absence of a theory or reason for seeking more recondite information, early fact-gathering is usually restricted to data that are readily accessible. Groups end up with a morass of facts that juxtaposed may or may not illuminate a situation. Kuhn writes that "only very occasionally...do facts collected with so little guidance from pre-established theory speak with sufficient clarity to permit the emergence of a first paradigm" (1962:16).

Hence, participants in the quality of life movement may be ready for the next phase of "measuring what we treasure," as Hazel Henderson noted in Paradigms in Progress (1995). It is common parlance among participants in local community indicators projects to describe the GDP as a less-than optimal measure of the progress of the nation or community. Alternative measures are in abundance; many bytes of data are collected and stored. Missing at this junction, however, is a methodology for organizing, synthesizing, and analyzing these myriad statistics in ways that allow the bytes of data to be transformed into meaningful "indicators" that can help citizens understand and influence complex social, economic, and environmental phenomena. The Calvert-Henderson models offer a solution to this problem.

Systems Theory
The Calvert-Henderson Quality of Life Indicators provide a methodology to add transparency and traction to current measurement efforts and at the same time advance the rigor with which to develop quality of life indicators. We adopted a systems approach, whereby all 12 dimensions of quality of life were viewed as integral to defining a broad picture of national well-being. It becomes clear to the reader where and how each indicator relates to the other indicators included in the Calvert-Henderson system.

Transparency is also created in the "unbundling" of the 12 Calvert-Henderson Indicators. As noted above, the reader will not find a single index or simple answer to the question: How well are we doing in a given domain? Rather, the approach unpacks the existing warehouse of information about a given dimension and presents the information or data in an organized fashion. Our intent is to make data on the various indicators accessible to people who have an interest in the topic, but are not necessarily experts in the field of study. This step in the methodology assumes that the general public not only wants more information about what is happening in the country, but can digest complex data when presented in a thoughtful, organized manner.

A systems approach that adds transparency to current discussions about quality of life is by necessity nonlinear. The connections that each author makes between components they have identified as critical within each domain do not follow a linear pattern in all instances. Rather, the information is presented in a circular, iterative fashion, which we believe characterizes the human phenomena in the long-term more accurately than a linear approach. As such, this book was not designed to be read from front to back. Readers are invited to jump into the volume wherever their interests are greatest. The algorithm for ordering the indicators is simply alphabetical.

III. Institutional Home and Authors
Another critical step in designing the Calvert-Henderson Indicators was to house the project in the Social Investment Research Department of the Calvert Group under the leadership of Jon Lickerman. Historically, quality of life research projects have been conducted by either social scientists at a non-profit think tank or academics at a university. One unique aspect of this project is that is was co-created by Hazel Henderson, an independent author and futurist in St. Augustine, Florida; the Calvert Group, an asset management firm in Bethesda, Maryland; and a group of scholars and practitioners located in various universities, government agencies, think tanks, and research firms across the county. In essence a new, virtual research institution was created to develop the Calvert-Henderson Indicators under the auspices of Calvert Group, which provided talented human capital and a strong research base from which to design and implement the project.

The scientists who took on the challenge of designing the Calvert-Henderson Indicators have devoted their careers to the study of the respective domains. They represent part of the nation's brain trust on these issues. The authors are attracted to scientific inquiry for all sorts of reasons. Some desire to show that social science research is useful and relevant to real world events. Others are excited about the prospects of exploring new territory. Some hope to find order in the complexity of the human and environmental condition. Others are driven to test established knowledge and beliefs in their respective disciplines. Each person rose to the challenge of making explicit the connections of their field of specialty with the greater social-economic-environmental whole as articulated by Hazel Henderson. While often frustrated and tested in the five year process, we were able to work together to solve a new puzzle that no one has explored so thoroughly before. For this, each author is to be commended.

IV. Calvert-Henderson Models
How do social science discoveries come about? Kuhn says that discoveries are not isolated events, but extended episodes with a regularly recurrent anomaly. The emergence of a new theory is generally preceded by a period of pronounced professional insecurity generated by the persistent failure of the puzzles of normal science to come out as they should. For example, economists debate anew when low unemployment persistently accompanies low inflation in defiance of the Phillips curve. Kuhn suggests that the failure of existing rules is the prelude to a search for new ones.

Hence, we began the search for better metrics to define quality of life by designing a conceptual model for each of the 12 domains. The Calvert-Henderson models serve as a framework through which a complex issue can be condensed into discrete components and understood by people who are not necessarily well-versed in the topic. Each model presents both a systematic method of capturing the relationships among the variables of interest and a technique for handling a host of statistical data on the topic.

When presented with one of the Calvert-Henderson models, the reader will be able to identify immediately what is and is not in the indicator. This is not the case with index numbers, which must be read in light of its construction. We have developed models that are easy to grasp and help tell a concise story about what is happening in the given domain. It is our intention to inform the reader of the state of knowledge about the indicator, not to ask why things are the way they are or how we got here. In this regard, the aim is positive not normative. We leave it to the readers to reach their own conclusions and come to judgment about how well we are doing in each domain.

Econometricians who conduct statistical analysis of economic data, learned to develop sophisticated models the hard way. Henri Theil (1971) reminds us that too often in the academy students wanting to conduct empirical research plunge into regression analysis with only a vague notion of what the technique is supposed to perform theoretically. Computer output forces the students to work backwards in order to interpret test statistics by studying general statistics and matrix algebra. The easier way would have been to first study econometric theories and devote attention to models and techniques before fast-forwarding to problem-solving.

The Calvert-Henderson models were thus specified at the beginning of the research process to reflect a theoretical understanding of each domain, albeit a simplified version of a complex reality. Only then were appropriate and relevant data assembled to enable the reader to determine the joint and simultaneous relationships between a number of variables of interest and come to judgment about our nation's progress. In the future, it may well be that different specifications need to be estimated as our world continues to evolve.

The principles guiding the collection of data presented in the Calvert-Henderson models include the following:

National Data:
The unit of analysis is the United States. Users are encouraged to extend the unit of analysis to the international arena and/or disaggregate to the local, state, or regional levels.

Annual Data:
The indicators track changes on a yearly basis for simplicity and to avoid seasonal biases.

Federal Government Data:
The United States statistical system provides a wealth of reliable, consistent, and verifiable data for most of the indicators. Wherever possible, authors used federal government data from public use files. Where gaps in federal data were identified, private data were used.

Time Series Data:
Data streams begin and end at periods specified by the authors to reflect salient moments in history for the respective domains. Most of the indicators include the most recent year of data provided by the United States statistical system.

Data and Values:
The data employed in this analysis are not value free. We emphasize that the selection of data draws attention to what each author deemed important to understand the state of the respective domain.

Data and Theory:
Scientific facts or data do not speak for themselves, they are read in light of theory. Hence the teachings that emerge from each indicator are captured in the respective models that represent a theoretical construct through which data can be easily conveyed and perhaps tested in the future.

Stratification of Information:
The Calvert-Henderson models have prioritized information on a given subject based on each author's theoretical understanding of the topic. The authors made critical and often difficult decisions about what to include in the initial model and what could be added in subsequent editions. Recognizing the constraints of developing the first national, comprehensive effort to redefine quality of life using a systems approach, it was understood that there are many layers underneath each model for future exploration.

V. End Result
It is our hope that the Calvert-Henderson Indicators will provide the readers with a solid body of empirical work to articulate a new theory and/or understanding of quality of life. We strived to resolve some residual ambiguities about these 12 aspects of life to permit citizens to solve problems related to key issues of concern. We hope to see this approach applied in various arenas and further articulated under new and more stringent conditions.

Kuhn's closing thoughts on scientific transitions are both hopeful and cautionary. The introduction of new theories and anomalies, in this case systemically measuring quality of life, is a sign of maturity in the development of any given scientific field. However, he predicts that there will be resistance when scientists are introduced to the new type of research. Kuhn warns that scientists wed to old schools of thought will devise numerous articulations and ad hoc modifications of their theory in order to eliminate any apparent conflict with the new research. This we have seen in the 1990s with the introduction of satellite accounts to the National Income and Product Accounts; chain-weighted productivity indices; calculating government expenditures on infrastructure as investments in the Gross Domestic Product; and "green GDP" analogues to redefine progress. Second, we can expect to see new and different analyses of science within which anomalies are no longer a source of dissonance (e.g., new theories of scientific knowledge). Third, more and more attention will be devoted to the anomalies by the field's most eminent people, demonstrated by the RAND Corporation and other mainstream firms' current interest in quality of life indicators.

We welcome such interest in improving the way we articulate and measure quality of life in the United States. The door is open to continually improve our methods, laws, and facts that constitute scientific techniques and theories. We believe the Calvert-Henderson Quality of Life Indicators provide a fruitful, rigorous methodology to help us come to grips with the central issues captured in the respective 12 domains. The significance of the indicators lies not in the numbers themselves, but in the larger reality toward which they point. Retooling social science research methodologies is a luxury reserved for special occasions. For this, the brain trust behind the effort is indebted to Hazel Henderson and the Calvert Group for moving us toward a perceptual transformation in our collective thinking about quality of life.

REFERENCES

Hazel, Henderson. 1995. Paradigms in Progress: Life Beyond Economics. San Francisco, CA: Berrett-Koehler Publishers

Kuhn, Thomas S. 1962. The Structure of Scientific Revolutions. Chicago, IL: The University of Chicago Press.

Samuelson, Paul A. 1947. Foundations of Economic Analysis. Cambridge, MA: Harvard University Press.

Sawicki, David S. and Patrice Flynn. 1996. Neighborhood Indicators: A Review of the Literature and an Assessment of Conceptual and Methodological Issues. Journal of the American Planning Association, Vol. 62, No.2 (Spring).

Theil, Henri. 1971. Principles of Econometrics. New York, NY: John Wiley & Sons, Inc.