Integrating Educational Research and Practice:
Reconceptualizing the Goals and Process of
Research to Improve Educational Practice

Nora Sabelli
Senior Program Director
Division of Research, Evaluation, and Communication
National Science Foundation
4201 Wilson Blvd.
Arlington, VA 22230
(703) 306-1655, ext. 5888
nsabelli@nsf.gov

Chris Dede
Professor, Education and Information Technology
George Mason University
Fairfax, VA 22030-4444
(703) 993-2019
cdede@gmu.edu

Abstract

In this paper the authors reflect on their learning while Senior Program Officers in the Directorate for Education and Human Resources of the National Science Foundation, where they have been involved in different capacities with a number of grant programs that fund peer-reviewed educational research in science, mathematics, and technology. Both as areas of focus and as central themes of full funding programs, their collective experience includes research into instruction and policy, learning and intelligent systems, applications of advanced technologies, and networking infrastructures in education. The authors argue for a taxonomy of research endeavors that places higher priority on crucial types of educational studies currently not conducted in a sustained fashion. Their strategy for increasing the impact of educational research is based on reconceptualizing the relation between scholarship and practice in the context of an urgent need for sustained, scaleable, quality educational reform directed towards all students. Central to this strategy is recognizing that, (a) if the education system is to change, the role of practitioners in the process of experimentation and adaptation must become paramount; (b) educational reform is an iterative process of improvement; and (c) the creative interaction between the definition of curricular standards and strategies for their implementation parallels the interplay between theory and experiment in other research situations. Improvement comes from within a system; and research, even when at its most supportive of practitioners, has not focused sufficient attention on the limitations and affordances of scholarship in the process of institutionalizing innovations.

The Research Context

Traditionally, the goal of research in education has been, with few exceptions, one of transfer-to the classroom or to policy making-of research results and insights conceptualized by scholars, in parallel or outside everyday practice. The relationship between researchers and practitioners has taken several forms, some quite close and supportive (Kennedy, 1997; Wagner, 1997), driven mainly by scholars' goals of contributing to the research community's knowledge-based on its epistemology, conceptual frameworks, and values-and disseminating such knowledge to practitioners as materials, directives, or rules. We are concerned in this paper with the implications for educational quality of this mode of transfer between research and practice, and our comments are designed to raise the visibility and priority of types of research that most empower the improvement of practice.

Our discussion focuses on what is missing from current approaches to conducting research that enhances practice-not on all aspects of research-and on the isolation of much educational research from learning outside schools. To a great degree, the need to revisit and enhance the impact of research on practice is a reflection of a paradigm shift that focuses on the 'systems thinking' that can achieve deep content learning (particularly strong in science and mathematics) by all students. In addition, at the basis of all our comments is the implicit assumption that quality content and pedagogy are central to all the educational research we discuss.

The most common strategies for research-based educational improvement have several limitations:

1. The educational system is reduced to isolated chunks of individual practice and of collective policy, not recognizing crucial systemic relationships and reinforcing links among these components.
2. Learning is seen as occurring only within a classroom setting (based on curriculum, guided by a teacher, managed by a schooling organization) rather than as a distributed process across all of everyday life shaped by the expectations of and interactions with educators, families, workplaces, and communities.
3. Insights and innovations from research provide valid new strategies for improvement to consider, but seldom make a general and sustainable impact on the field. This is in part because researchers often focus on issues peripheral to the concerns of practitioners and policy makers, or study highly restricted situations from which results do not generalize to other contexts, or report their outcomes in a scholarly manner that does not stress the types of evidence and process details that educational implementers find persuasive, or provide insufficient opportunities for practitioners to develop a deep understanding of the conceptual basis of the innovation, of its goals, and how it looks in the classroom when adequately implemented.

Even teachers' "action research" on their classrooms is shaped by a research-centric perspective on education as composed of independent classroom units in which teaching and learning can be incrementally improved through reflective analysis at the 'micro' level, and of a separate policy making sphere in which aggregated patterns and trends provide the data needed for decision making.

Within these constraints, research has led to many valid, useful and exciting models of pedagogy (Darling-Hammond, 1997; Dede, 1998; National Commission on Teaching and America's Future, 1996), and to policy commitments in support of research linked to educational improvement (President's Committee of Advisors in Science and Technology, 1997). However, collectively decades of funded studies that have resulted in many exciting programs and advances, have not resulted in pervasive, accepted, sustainable, large-scale improvements in actual classroom practice, in a critical mass of effective models for educational improvement, or in supportive interplay among researchers, schools, families, employers, and communities.

The root causes of this shortfall in research impact on practice rest partially with the research community, partially with the agencies that fund educational research, partially with the educational system, and partially with the slow pace at which strategies are developed to satisfy shifting needs for educational outcomes. For example, at the National Science Foundation (NSF), peer-review-based educational research investments from 1993-96 focused on:

• analyzing students' learning processes for understanding math and science concepts
• studying students' and teachers' knowledge and beliefs about math and science
• analyzing student achievement by family, student, school, and sociocultural factors
• conducting research on teachers and teachers as researchers
• studying instructional practices and curricula, as well as educational reform on a highly aggregated basis
• conducting data-driven analyses of change in math and science education
• developing and studying efforts to achieve educational equality
• analyzing curricular needs created by advances in science and mathematics practice
• assessing the impact and application of science and math skills in society
• developing and validating teaching and learning strategies
• designing and evaluating intelligent tools and technology-based learning environments
• developing innovative models for collaborative learning and teaching through innovative uses of telecommunications
• conducting demonstrations, testbeds and proof-of-concept tests of innovative technology applications

Some projects strive for greater descriptive understanding of educational and societal dynamics, most are laboratory or design experiment studies that create atypical "islands of innovation," and only the testbeds (a small fraction of the research portfolio) deal with value-added, scalability, generalizability, and sustainability. Yet practitioners and policy makers are focused (appropriately) on exactly these issues.

Not present in this picture as a dominant overall goal in innovation-oriented research is consideration of the educational system qua system and of the interplay between policy goals, research outcomes, and actual classroom changes that lead to coherent actions by individuals and their organizations. As numerous studies of the processes underlying educational change document (Cuban, 1990; Fullan, 1993), these systemic relationships are crucial determinants of whether the implementation of a strategy for improvement succeeds or fails in reaching its educational objectives. In addition, a lack of systemic analytical frameworks makes it difficult for researchers and practitioners to develop insights on the relationships among these isolated themes. Such frameworks are needed in order to identify understudied areas and pressure points where new ideas could lead to coherent strategies for educational improvement.

The NSF portfolio of research investments, centered primarily on improving school instruction, has not permeated the agency's implementation portfolio in a systematic way. Systemic and infrastructure issues were primarily explored in the context of evaluating systemic projects or of building the human resource base for a technological infrastructure. This approach has left largely undeveloped many opportunities for creating partnerships among educators, parents, employers, disciplinary experts, and communities; for enabling deeper and more rigorous study of complex mathematical and scientific content; and for continuous, affordable and content-rich professional development of teachers.

Mutually Developed Implementation Studies as a Strategy for Improvement

How could investigators and funders refocus the goals and processes of research to transcend these methodological limitations? Reconceptualizing research agendas, priorities, and processes to focus more thought on implementation studies mutually developed by scholars, practitioners, and policy makers is a promising strategy to develop the research base needed for sustainable impact. We believe too little emphasis is placed on reflectively scaling-up and adapting interventions, policies, and experiments that as isolated islands of innovation are very successful; and even less priority is given to modeling and generalizing the coherent processes that led these innovations to succeed. Moreover, the common practice of considering studies of large-scale educational outcomes primarily as evaluations to be conducted by evaluators does not create the deep research base we seek, partly because outcome evaluation methodologies and goals often omit crucial "outside" variables and partly because this removes the responsibility for practitioners and funders to be active participants in assessing value-added and implementation strategies, and thus learn via participating in this process.

If one is interested in coherent, sustainable, and scaleable change, we posit that understanding the process of innovating (i.e. of altering standard practices) is as important as its outcomes. In the few instances when we have seen, in an implementation context, the evaluation of new strategies for change lead to deep understandings of sustainability and a plan of action, the "evaluation" (both formative and summative) was performed by the project team itself, led by a strong researcher. In this situation, the objective was not only to "evaluate" the performance of the project-a worthy goal-but also in tandem to model and institutionalize its gains. Such internal reflective learning could take place in all large scale implementations and not be limited to those undertaken with a specific innovation research purpose, or subsumed under an external evaluation of a project outcomes or performance.

Certainly, this type of research on sustainability and scaling-up of reform is expensive, people-intensive, and time consuming; but these are not the only reasons why such studies are seldom done. Implementation (systemic, applied) research does not fit well within conventional academic career paths, demands a multiplicity of theoretical and methodological perspectives (difficult given the fragmentation of the academy); cannot be conducted by just one-or a few-investigators, and requires sharing research (i.e., decision) power with practitioners and policy makers. When such close partnerships are in place, researchers must not only relinquish sole control over the analytic process, but also act as brokers to guide and mediate the reflective interactions of other stakeholders. This is an skill for which most investigators are not prepared-in part because conceptual frameworks for this type of research are not well developed. This skill is best fostered through supporting practice changes within integrated implementation and research "testbeds," an uncommon experience for most scholars.

As has been recently suggested by the President's Committee of Advisors in Science and Technology (1997), if our ultimate goal is long-term, pervasive, quality educational improvements, we must find ways to invest a "critical mass" of funds and human resources in clinical-type research. This article argues for a definition of such research as reflective interplay between basic research and practice, a process that is bi-directional and helps both sides evolve towards increasingly sophisticated objectives. In such a relationship, implementation is not the blind adoption of recipes and materials for innovation developed by others, but instead the reflective adaptation of a process that enabled a similar group to succeed in improvements actualized somewhere else. Focusing on the process as well as on outcomes makes it possible for practitioners to start with objectives consistent with their current problems and worldview and evolve towards more powerful goals as they reflectively adapt innovations. This type of interaction among scholars, policy makers, practitioners, and other stakeholders in quality education undercuts the limitations discussed earlier and lies at the heart of effective, sustainable scaling-up through the mutual evolution of research goals and outcomes.

In this type of scholarship, the crucial object of study is the process of implementation research itself, rather than particular outcomes enabled by the project. The intent of the research endeavor shifts from achieving an expected outcome to planning for continual, reflective evolution. Even the best of predetermined goals, in our view, too often lead to either unattainable or unimportant outcomes, or worse, to a co-opting of the goals. Evolving objectives, on the other hand, require developing a shared, long-term vision of what excellence in education means. Some central questions in this process of mutually conducted implementation research could be: What are the critical insights needed now to plan for and achieve an educational system's long-term goals? What is the existing knowledge base in a particular situation, and where are the pressure points in that context for augmenting that base? What types of intermediaries can aid or subvert the institutionalization of an ongoing relationship between practice, policy making, and basic, applied, and systemic research?

Reconceptualizing the Goals of Educational Research

Our premise is that higher priority should be given by the educational research community and its funding organizations to:

1. developing generalizable conceptual frameworks that coherently advance knowledge in support of sustainable improvements in teaching, learning, cognition, motivation, and problem-solving for all students (defining "improvements" is a significant part of this challenge);
2. conducting large-scale interventions based on innovations with the potential to significantly enhance educational outcomes through enabling all learners to master more complex content via sophisticated and adaptable pedagogies; and
3. conducting, in association with such interventions, a reflective ("taking stock") analysis of the implementation process itself, so that researchers, practitioners and other stakeholders (such as parents and policy makers) can understand the process they are experiencing, adapt and modify it, and establish acceptable milestones for achieving more effective learning environments (in classrooms, homes, workplaces, and communities) that serve the needs of all students.

In summary, we see the need for reflective and adaptive research methodologies applied to the study of complex educational systems. These goals can be accomplished, for example, by a balanced portfolio of research studies, conducted not linearly but in a richly intergenerative fashion, categorized by methodology and objectives as follows:

1. basic research and small, focused laboratory-type experiments: designed to advance foundational knowledge (for example, cognitive science research, some aspects of cognitive neuroscience and neural networks, studies of how naive learners and experts in a given area organize their learning, effect of cognitive tool use in stages of intellectual development, etc.)

2. design experiments and applied research: design experiments (Collins, 1996) to advance practices of teaching , learning, and organizational management (for example, longitudinal studies of the uses of scientific and mathematical modeling as pedagogy, environments that support learning of increasingly complex tasks, theories of instructional design in environments where learners have differing repertoires of learning styles, etc.)

3. data gathering and analysis studies on prior and existing practice and on intervention experiments: designed to evaluate and rationalize outcomes across laboratory and intervention studies (for example, surveys of teacher certification in the topic being taught, issues of gender and career choices, etc.)

4. systemic research on reform consortia and on large-scale intervention experiments: designed to advance the implementation of intervention experiments and research-based innovations in large-scale and complex educational systems (research and implementation testbeds in specific contexts, studies of reform consortia, comparison of the economics of scaling-up models, role of motivation and intellectual ownership in sustaining reform, social and ethnographic studies, policy research, economic co-dependence of publishers, software developers, other vendors and schools; etc.)

The Educational System Context

Traditionally, the primary goal of educational practitioners and policy makers has been, with few exceptions, to efficiently follow standard operating procedures, altering these only when current learning outcomes fall obviously short of society's needs and expectations. In the event that innovations are necessary to remediate shortfalls, the interventions attempted are seldom based on prior knowledge, on validated experience, or on research evidence, but usually emerge from political expediency or currently popular strategies for change. In parallel to scholars' emphasis on research as an activity inherently outside everyday practice, implementation programs and projects often assume a stance "outside research" by not monitoring relevant research or by limiting their goals to "fine-tuning" current approaches rather than searching for more ambitious visions. Too often, practitioners and policy makers reject as "impractical" a reflective approach to implementation as an iterative process in sustainable reform, thereby undercutting the leverage research might provide in developing strategic objectives and effective evolutionary processes for improvement.

Intervention experiments are one means to explore and analyze how to introduce alternate, reflective innovation strategies in a large, complex, context. In order to succeed, implementation research must be based on insights from laboratory experiments and must focus on reflective adaptation to given contexts. Unfortunately, in practice this is not often the case; the balance between "general," "generalizable," and "localized" knowledge acquired from research is usually not clear or even present in research portfolios. In fast-moving environments such as the present, data-based research offers baseline information more often than guidelines for effective future action. Basic research is isolated from policy decisions about which innovations are most promising to perform and sustain. Interventions mandated by local boards, politicians, or the public are typically not viewed as applied experiments that build on laboratory trials through iterative refinement to match a particular situation. Rather, they are seen as 'silver bullets' that will magically solve a pressing problem.

Moreover, performers and key audiences are often different for laboratory research and for implementation studies; in the absence of a common research framework, connections between these types of experiments are not broad enough to support informed public debate. In addition, seldom are research methodologies in use by other scholarly fields brought to bear on educational systems, even though these analytic methods have contributed to the understanding of designed, human contexts (e.g., corporations, cities) and of institutions charged with rapid technology-based improvement (such as hospitals). Characteristics of the larger societal context that contribute to educational inertia, such as testing and textbook companies, are seldom included in our models of educational change. These disconnects among basic research, useful methodologies from other fields, implementation studies, and policy hamper aggregating experiments towards a larger goal and undercut an ownership of change in those who must implement innovations.

Since a compelling need exists for clinical and implementation research, the educational research community must undertake the task of appropriate capacity building in a new generation of applied researchers and research-informed practitioners. We need innovative, mutually evolved methodologies for this type of research. However, the scholarly community knows better how to perform laboratory and analysis studies (whose methodologies are akin to physical and social sciences research) than how to conduct large-scale implementation studies. Most investigators do not have good mental and methodological models of intervention and systemic research, which are similar to applied scholarly activities in medicine and engineering where the relation between general knowledge and individual interventions is crucial. In these fields, the ability and responsibility of the individual practitioner to apply accepted professional practices and keep abreast of how these practices evolve, termed "compliance;" this perspective is largely absent from educational research. As a result, practitioners and policy makers are often appropriately skeptical of "ivory tower" remedies unworkable in real world settings, and in practice they eschew or co-opt such remedies.

In addition, compliance in medicine and engineering is appropriately a professional issue shared by theoreticians and practitioners. Unfortunately, in education it is instead approached as an issue of control. For example, contrasting medicine and education, the expectations of each research community about the role of practitioners in enabling compliance are at opposite ends of a spectrum: Physicians are seen as partners in the improvement of public health, but teachers are too often viewed as impediments to implementing improvements in students' learning. Terms such as "teacher enhancement" demonstrate a rhetoric that views educational practice knowledge as a one-way flow from researchers to practitioners. Absent a supportive relationship in which the educational research community acknowledges teachers and administrators as professionals with insights to offer, and practitioners' institutions as learning organizations that shape the meaning of compliance, those in the field have few inducements to incorporate and sustain research-based improvements that reshape standard operating procedures.

Beyond these problems of disconnects among policy, basic, and applied research, as well as between researchers and practitioners, the value of innovations is too often reduced to and measured by only short-term improvements on standardized tests of student performance. This focus, though useful within its limits, provides too narrow a framework for evaluating the gains, successes and shortfalls of educational reform's process and outcomes. When taken as the only measure of worth-and this is how most discussions frame this issue- evaluation based on these limited measures does not provide the public and policy makers enough supportive data for a crucial next step in educational improvement: a continued progression of thoughtful innovations, developed from basic research knowledge and assessed in systemic practice, that sustain a coherent strategy for higher-level student learning to meet the long-term needs of society. The lack of a shared conceptual framework for sustainable educational improvement contributes to the inadequacy of generally-accepted measures for determining the pace and slope of the path towards improvement.

Rewards are fundamental for conveying to others what we truly value. Focusing educational assessment almost exclusively on students' short-term capacity to restate declarative information leaves strategic intellectual, instructional, organizational, and political advances unrewarded and does not provide policy makers with an accurate way to gauge or plan true progress towards achieving education's most important goal: developing human resources for the future. Further, an all-or-nothing improvement strategy focused on immediate results is a self-defeating one. Change is a process, not an outcome; a 'one shot' improvement implies that the environment that nurtures the current failure is left unchanged. Measures and milestones of the directions in which schools are moving and the pace at which they advance must alter to provide a constructive relation among research, policy, and practice. These three communities must develop shared conceptions of educational reform that allow practitioners and researchers to assess their continual individual and collective progress based on a dynamic, holistic view of the educational system, in which standard measures of success are only one part of the assessment strategy.

We think the central problem is not only to achieve success in current reform efforts, but to enable continued, deeper improvements in learning through an evolutionary process built into the structure of our society's educational system. Otherwise, we will find ourselves confronted with recurring cyclical calls for 'reform.' Other fields provide better models of evolutionary advance than does education, in which change is seen as sporadic and following a step function. Every two decades the field goes through a cathartic phase in which immense energy and resources (and political pain) are suddenly invested in "revolutionizing" education to patch up many years of accumulating shortfalls.

To avoid such unproductive and demoralizing attempts at instant innovation (Kaestle, 1993; Kaestle, 1997), the research and policy communities must alter their perspective on the educational system and reconceptualize schooling as a form of social organization that can learn from itself. Many of the problems with reform arise from a view of the educational system as dependent on outside largesse-funding, materials, "teacher enhancement," research, policies-where all its practitioners can do is "deliver instruction." Without shared expectations among researchers, practitioners, policy makers, and local communities, no model for educational experimentation to enable educational evolution will succeed. Even though a collective model of innovation dominates organizational evolution outside of schooling and is driving the economics of the workplace, it is largely absent in both practitioners' and researchers' views of educational innovation.

In the context of the educational system, we believe enhancing reflective innovation and evolution requires both researchers and practitioners to consider that:

• Any sustainable change must at some point be 'owned' by an individual, organization, or system rather than being seen as imposed by some external mandate.
• No sustainable, coherent change can take place without reflective, continual experimentation by small groups of students, teachers, and managers within the target organizational context.
• Not only strategies evolve; the meanings attached to goals also shift as we learn. Worthwhile strategic goals must not be set in stone; if frozen in time, the objectives sought will become inconsequential, eschewed, or co-opted.
• Scaling-up does not only mean multiplying practices or disseminating outcomes, but also determining the contextual conditions for a similar process of innovation to take place--engineers know the fundamental role played in design by incremental improvement based on analyzing failures.
• No seminal experimentation will take place without a research mentality (the formation of testable hypotheses, the habit of reflection), supported by appropriate resources and embedded into the cultural self-image of the institution as a learning organization.

The research ethos must exist not only inside the "system" as a single unit of analysis, but inside each of the critical components of the system (i.e., the practitioner, administrative and policy maker communities); otherwise each component's approach to innovation will be shaped by divergent goals and measures. Under these adverse conditions, reform is necessarily sporadic, dependent on temporary circumstances, and unsustainable.

Bridging the Research and Educational System Contexts

The impetus for these shifts in research-to-practice assumptions and beliefs must initially come from the research community. But research can play a significant role in sustainable educational reform only when scholars working in this area admit that advances in school reform theory come from the interplay of research and experiment (developing and testing hypotheses in the challenging real world settings where implementation problems surface)-and seriously include in development, testing, and modification of hypotheses the insights of practitioners involved in reflective innovation.

Historically, science and mathematics educational research have moved closer and closer to enactment in contexts of authentic instructional practice. We have progressed from laboratory studies and mass testing to detailed research in classrooms and careful analysis of students learning complex concepts. This article calls for expanding this trend by giving much higher priority to research on whole systems of educational practice, including schools, families, mass media. Such research is currently rare, for all the reasons described earlier, but urgently needed.

To enable conducting the mutually developed, coherent implementation studies suggested earlier as a crucial strategy for educational improvement, the scholarly community and its funders must become more reflective and self-critical about the current process and goals of educational research. Studies of education are similar in some (not all) ways to studies in the sciences, social sciences, engineering, and mathematics. Analogous to research in engineering and the social sciences, educational studies involve developing knowledge about designed, human contexts less constant in their attributes than natural phenomena. As Salomon (1991) describes, aircraft design is a quintessential activity that must embrace complexity in order to reach solution; similarly, educational reform is a complex, multifaceted enterprise. The situations studied by educational researchers are can be seen as complex systems with sophisticated feedback and non-linear causality, similar to biological or ecological systems, and similarly benefit from integrated system research strategies. Beyond independent scholarship, educational researchers also should play a role as intermediaries who enable experts in other disciplines, educational practitioners, learners, funders, and policy makers to jointly develop exemplary practices.

As the field of education changes, the types of research requested and needed alter. In the 1980s, societal concern about educational outcomes led to a variety of descriptive studies designed to assess and understand problems in performance. In the late 1990s, now that the causes underlying educational dysfunctions are better understood, practitioners and policy makers are asking researchers to focus on applied studies that improve practice in a sustainable, affordable, and scaleable manner. Recent requests from the field for research results that inform practice have centered on evaluative studies that provide evidence that current educational interventions are worth the cost and trouble involved in implementation. We believe these questions should be answered in a manner that focuses on how to implement a process that optimizes investments in innovation and increases the quality and depth of students' learning outcomes.

Another factor influencing the nature of educational research is the rapid advance of information technology and the affordances new media provide for both learning and sustaining change. Including studies of educational technology in the research portfolio is vital because the following possibilities need to be examined, and, if true, the conditions for realizing their potential developed in an equitable manner:

Drivers of Educational Change

• Information technology is changing the types of student competencies the U.S. needs from its educational systems, as well as from workplace and informal learning.
• Fluency in information tools and the ability to validate information should be central to inquiry-based learning and standards-based content in an information-rich environment, just as they are for scientific practice and other forms of collective, organized problem-solving.

Potential Capabilities of Information Technology

• Through individualization, providing concrete representations of abstract concepts, and social interactions with experts, technology can make mastery of more complex knowledge and high-order skills achievable for all students. (However, enabling all students to participate in inquiry learning and in the scientific enterprise depends on proper attention to equity and access to information technology and its tools.)
• New media can enable educators, parents, scientists, employers, and local communities to partner as a means of enhancing student learning, and provide teachers and students with access to human and archival resources outside the confines of schools and classrooms.
• Information tools and virtual environments can enable sophisticated theoretical studies of the nature of learning and of distributed intelligence, which then can be applied in practice
• Using advanced collaboration technologies to enable "knowledge networking" among educational practitioners, researchers, and policy makers can enhance the scalability of systemic reform and the professionalization of the instructional workforce

Studies of alternative approaches to learning that emerging technologies enable are an important strategic investment for educational research, as these innovations might help resolve current problems of schooling and of scaling-up change in a more effective and affordable way than present strategies for improvement. The rapid advances made by computer science in developing powerful tools-advances based on a systems research paradigm of building complex tools to study the power of the embedded concepts-has much to offer for developing educational reform processes that are scaleable and sustainable.

In response to these shifting needs, a portfolio of educational research should include substantial numbers of projects that:

• develop a broader base of researchers and scholars capable of studying the complex situations characteristics of large-scale educational innovation in complex environments at many different points in the continuum of basic to systemic research;
• promote thinking on the integration of content areas, parallel with the emerging inter-disciplinary understandings characteristic of today's scientific research, as well as the scientific literacy needs of a knowledge-based economy in which such research has practical implications;
• develop new types of research methodologies to address significant problems of practitioners and policy makers at appropriate levels of aggregation;
• sponsor a series of research summaries that present validated current research results to a larger audience;
• build partnerships with practitioners and policy makers in the design and analysis of research based on or leading to projects that answer useful questions, provide analyses of results generalizable to other real-world educational situations, and develop insights that are widely discussed and adopted; and
• foster the potential capability of emerging information technologies to enhance learning and teaching; to improve the organizational efficiency and effectiveness of schools; to enable partnerships for learning among teachers, parents, employers, and communities; and to develop new tools and methods that empower the process of knowledge generation and communication.

To assess the quality and potential of educational research, we need a balanced portfolio of research capable of sustaining educational reform, including:

• fundamental innovation rather than minor enhancements to existing practices
• a focus on strategic issues of major importance to practitioners and policy makers throughout the multi-year time frame in which research outcomes are developed
• the development of a conceptual framework based both on current challenges in educational practice and on emerging opportunities from potential innovations in technology or theory
• closer research links to basic scientific research that has the potential to significant improve practice, or to shed light on human learning and coping strategies, and that can provide applied educational research with productive questions of relevance to human knowledge acquisition, retention, and generation
• a long-term perspective, making convincing assumptions about coming advances in technology or knowledge that require immediate, strategic action to prepare educational practice and policy for the future
• a model based on systems thinking rather than reductionist approaches to understanding the interrelationships between an educational intervention and the other processes occurring in its organizational context
• an analysis of already existing, related research that enables this project to extend, complement, or refute similar studies
• analysis of project products and outcomes, including testing the falsifiability of research hypotheses by controlled studies against "current best practice" alternatives to the proposed innovation
• studying successes to "reverse engineer" their generalizable implications for learning, teaching, and organizational structure
• proof-of-concept demonstrations that validate an insight or innovation through pilot implementation in a realistic, authentic setting, rather than simply in laboratory trials
• the generation of fundamental insights that generalize to advance the conceptual frameworks educators use in decision making, or the development of innovations in practice that transfer beyond the local context into many similar situations nationally
• innovations that are affordable, scaleable, and sustainable
• integrating aggressive and proactive dissemination strategies that both inform a broad base of researchers, practitioners, and policy makers and provide support for those interested in adopting or extending the innovation

Not all of these criteria are appropriate for every type of educational research, but they suggest the type of approach that could be used when making competitive judgments about what scholarship to propose or what funding decisions lead to an effective research portfolio.

Lagemann (1997) analyzed the history of educational research from the very useful perspective of how the professionalization of this field has influenced linking knowledge and action in education. As a complementary perspective, limits on the professionalization of practitioners in U.S. education (i.e., not supporting the development of their capacity for independent judgment and knowledge of their subject matter and its pedagogy, not providing time for their reflection and sharing ideas with colleagues) hamper linking knowledge and action. Comparisons of the role of teachers in the US and other countries-or of the role of teachers relative to practitioners in other fields such as medicine and engineering-illuminate how underestimating the professionalism of U.S. educational practitioners in turn undercuts the sustainability and quality of systemic educational reform.

Conclusion

Given that our nation's educational system must evolve to meet the needs of a global, knowledge-based democratic society, then all the groups within that system must contribute towards mutually agreed innovation strategies and outcomes. Any group that is not given validity as a respected and empowered contributor towards improvement will (wittingly or unwittingly) co-opt the change into trusted and known patterns of work and thus defeat any purpose not mutually developed. This is why the prevalent conceptualization of educational research as academic and separate from practice has direct deleterious effects on the relationship between research, policy, and practice.

Reconceptualizing research agendas, priorities, and processes to focus more on implementation studies mutually developed by scholars, practitioners, and policy makers is a promising strategy to remedy this situation. For such a reflective, evolutionary innovation process to succeed, researchers, practitioners, and policy makers must all alter their views on the value and conduct of research, as well as on the roles played by these three groups in research and implementation. Acceptance of these shifts in assumptions and actions must initially come from the research community and its funders; this article suggests first steps in this direction.

References

Collins, A. (1996). Design issues for learning environments. In S. Vosniadou, E. D. Corte, R. Glaser, & H. Mandl (Eds.), International perspectives on the psychological foundations of technology-based learning environments. New York: Springer-Verlag

Cuban, L. (1990). Reforming again, again, and again. Educational Researcher, 19(1), 3-13.
Darling-Hammond, L. (1997). Teachers and teaching: Testing policy hypotheses from a national commission report, Educational Researcher, 27(1), 5-15.

Dede, C. (1998). Learning with technology (the 1998 Yearbook of the Association for Supervision and Curriculum Development). Alexandria, VA: ASCD.

Fullan, M.G. (1993). Change forces: Probing the depths of educational reform. Bristol, PA: Falmer Press.

Kaestle, C. (1993). The awful reputation of educational research, Educational Researcher, 22 (1), 23-31.

Kaestle, C. (1997). Improving the awful reputation of educational research, Educational Researcher, 26(7), 26-28.

Kennedy, M. (1997). The connection between research and practice, Educational Researcher, 26(7), 4-12.

Lagemann, E.C. (1997). Contested terrain: A history of education and the United States, 1890-1990.Educational Researcher, 26(9), 5-17.

National Commission on Teaching and America's Future. (1996). What Matters Most: Teaching for America's Future. New York, NY: National Commission on Teaching and America's Future.
President's Committee of Advisors in Science and Technology. (1997). Report to the president on the use of technology to strengthen K-12 education in the United States. Washington, DC: USGPO.

Salomon, G. (1991). Transcending the qualitative/ quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher,. 20 (6), 10-18.

Wagner, J. (1997). The unavoidable intervention of educational research: A framework for reconsidering researcher-practitioner cooperation, Educational Researcher, 26(7), 13-22.