Applying What We Know to Improve Teaching and Learning

Nora H. Sabelli
SRI International
2008

Prepared for the Carnegie-IAS Commission on Mathematics and Science Education

“In all of this, I think you know we are not only interested in understanding what is possible, but also in having you spell out in concrete terms what are the next logical and practical steps that ought to be taken to reap the benefits of today’s technology.”

My comments are therefore thoughts on why we don’t use what we know, why we don’t know what we need to know, and how we can use the knowledge we have to do better for all students and all teachers. The emphasis is on action for long-term impact. I use my experience with science, technology and education to present some ideas that can lead to effective action—meaning positive, significant, sustainable STEM education improvements, with eventual large-scale impact. Theoretical arguments derived from the study of complex systems, of which education is an excellent example, can provide guidance for action, including new forms of research from which the system itself can learn. The arguments suggest the simultaneous need to inform the public debate about education with a vision of what technology makes possible for all students, and endeavor to aggregate and make known the existing models of how to achieve the vision.

One thing that technology has done is to move policy and innovation thinking beyond linear models. Timescales of change are too fast to think “research to prototype to scale and be done with it.” The end user intrudes into the model. We are in the decades of designer materials—YouTube, and Open Source Research—where those that study and those that do jointly contribute to advances.

We know quite a bit about how to use technology in education. What we know is not always “useable”¹ knowledge for practice because it is not tied to contexts so that we can, with confidence, predict how we can make it succeed under a plethora of classroom environments, student interests, parent expectations, teacher knowledge and beliefs, and policy mandates. Until we confront what is called “localization or implementation research” we will not be able to refine the design of materials and pedagogies so that they can be effectively applied under many conditions, and be also “educative” for teachers.

Policy must be confronted, and collaboration between researchers and practitioners must become more prevalent and sustained.

Teachers are crucial for large-scale impact. They are most effective when they can expose student thinking, and have pedagogical content knowledge, which differs from disciplinary content knowledge. And they need support. Could we make the current ineffective process more efficient? Yes—but clarity of goals is important. After Sputnik, we succeeded in “catching up with the Russians.” But we established “science is for the best and the brightest,” which still lies behind many current problems. My comments assume that we want to educate all students for a scientific and technically savvy citizenry that feeds a knowledgeable workforce, not only a professional one.

Caveat: This is a position paper on what we know, not a review of existing literature. It is a reflection on what I learned through my years at NSF, as well as at NCSA, SRI, and LIFE. My comments are all based on research results viewed from a policy perspective. They are the result of many discussions with researchers and other colleagues, and based on the unique programmatic experience that I was fortunate to acquire. Thus there are many relevant references missing. My research experience is computational chemistry. I have not conducted or written education research. I remember lessons and people, not references. Adding them would require the contributions of many in the field whose work I have followed, and to whom I owe a debt of gratitude.

Executive Summary

Wherever I go, I see impressive advances. But the center of gravity of engineering education has not moved (paraphrased from William Wulf)

Unlike other education reform periods of the last twenty years, the knowledge and skills all students need to prepare for the world of work and the work of science are not socially agreed upon, nor do we know how to achieve what we agree. A clear vision of the future of education is absent from curricula and public perception. “Strengthening the basics” looks like an option to many, but there are new social and economic imperatives, not only an educational imperative, to add to the traditional three Rs new requirements for scientific and technological literacy.

This position paper is based on two premises.

First, that we know enough about improving the teaching and learning of science, technology, engineering and mathematics (STEM) to engage in large-scale implementation of the education we envision.

Second, that we do not yet know enough about expanding, disseminating, accessing, and sustaining what we already know about STEM education.

Based on these premises, the paper argues that we must engage in the second—“implementation research” to achieve long term, sustainable, improvements in STEM education.

The paper draws on what has been learned from research in many aspects of education practice in order to achieve sustainable improvements in STEM education across the multi-faceted, complex, and varied education system. The paper concludes by recommending several possible actions, identifying examples of programs which embody the best of what we know, and listing some implementation “lessons learned” thus far.

What We Know: Action for Impact

Research does show that technology makes a significant impact in STEM education, but we are not applying broadly enough what is known. What we know is not always “useable”² knowledge for practice because we do not understand the process of implementation well enough. We need to learn from successful implementations how we can embed them in the fabric of the education system, so with confidence we can predict how we can make new approaches succeed under a plethora of classroom environments, student interests, parent expectations, and teacher knowledge and beliefs.

To change this situation, we need to understand and act on the educational system’s constraints. Policy must be confronted, and collaboration between researchers and practitioners must become more prevalent and sustained.

To achieve significant and lasting improvement requires considerations of infrastructure as well as technology, pedagogy and content. By infrastructure I mean the human and technological underlying support that shifts some of the weight of implementing change from individual teachers or schools to a shared support system. To get a sense of what we need to achieve think of libraries without librarians and card catalogs.

Although I am very excited about the curricular changes that can be attained with modern uses of technology, I will not talk about most of them here³. Nor will I emphasize how cognitive research on learning should be integrated into education. I will also not comment in detail on measures of learning and accountability. Though not as forward looking as research on using technology for STEM education, exciting new assessment ideas are being studied and implemented, and we know the limits of one-size-fits-all measures. Much can still be done in these areas, but I am more concerned here with using what is known and what will be known in the future—areas that do not receive the same level of attention.

Page 4

Context for Education Research

“Research” refers to a continuum of work. At one end of the continuum, research is defined by researcher questions that push the boundaries of knowledge. At the other end of the continuum, research is defined by large-scale and contextual experiments, as defined by implementation research questions around robust applications. Research on the center of the continuum, as much of education research does (for example, by considering content and pedagogy as a given tenet, or by analyzing current classroom practice) is a crucial knowledge base for change, but is not enough to understand how to change the features of the system that prevent sustained improvements. Research has to study prototypes and models of infrastructure reform that can scale-up and become sustainable and systemic to offer new insights into solutions. Case studies are not enough.

Although the ideas behind implementation research are not new, they have taken a new life within the public health community, confronted with the gap between rapid advances in the medical sciences and continued public health problems such as nicotine addiction, obesity, sexually transmitted diseases, etc. The argument behind education implementation research is precisely that with current advances in research on technology, pedagogy and cognition, we are in a position to focus on understanding how best to use what we know, and build on this understanding the next stages of R&D, in an iterative improvement cycle.

This implies that resources available for research should be sufficient and coordinated, and can support sustained collaborations inside and outside of education to ask hard questions. Funds for a few large-scale implementations have been available, but were never sufficient to treat them as experiments—to document the process and analyze their progress and pitfalls.

We need to think of a society where scientific and technological literacy are a requisite for participation for all citizens. One significant prior “social experiment” whose success we should remember is the agricultural system of land-grant universities and extension services⁴ that made possible the step-wise supported “translation” of research from the first-level research universities to the local needs of the farmer (e.g., local climate, soil characteristics, transportation costs, etc.).

Page 2

Disseminating Innovations

Successful diffusion of an innovation is an explicit process of adaptation—it is never just adoption. Neither artifacts nor ideas are sufficient by themselves as mechanisms for successful dissemination. Adoptions that fail waste precious time and resources.

Adaptations require insights into the process of implementation. Successful adaptations need long-term collaborations and benefit from a support infrastructure (literature, specific expertise, the design of the innovation, colleagues’ experiences, additional research, etc.). This is where the agricultural extension system model has much to teach us.

Each “innovation,” when implemented, is in fact a system of innovations that include changes in classroom management, professional development, the interaction between practitioner and parents, colleagues and supervisors, union rules—a whole set of lateral innovations that must be taken into account for the innovation to succeed. An implemented innovation can’t be independent of the system that uses it.

What We Can Learn From a System Science

Often, optimal solutions are counterintuitive. But in education, everybody’s intuition—informed or not—leads to action, when it should be subject to evaluation.

The fruits of reform tend to become visible only after at least three to five years. School administrators and parents want to know earlier if changes are leading in the right direction. Research should develop dynamical models of educational reform that allow ongoing assessments of individual and collective progress while the process is still developing. Such models are being developed in other systems for studying management practices.

The nature of a system is its dynamics (i.e. how the different parts interact with each other). Often, as in education, change takes place at different timescales for different parts of the system—students, teachers, schools, districts, and state policy, teacher education. That is why intuited changes usually focus on one part of the system; understanding the complexity of the dynamics of interaction between all the parts requires a different approach. Complex system models are needed to understand these dynamics, particularly what conditions lead to change. Realistic models, based on detailed data from case studies of reform efforts, may help plan for a successful process of adaptation.

What We Know: Content, Pedagogy, Instructional Workforce

If the goal is impacting the system and not only an individual classroom, we must think five years ahead, at a minimum. This has implications for content, pedagogy, and teacher education. Although STEM education standards will evolve, their current form allows for content that highlights what we need to say about the science literacy needed in five years; there is no need to wait.

Pedagogy makes a crucial difference in what students learn, and changing pedagogy implies changing the teacher’s classroom activities. As many have noted, classroom pedagogy is what provides stability within the ever-changing school policy and personnel environment. Teachers’ time for becoming proficient with new pedagogies and content is a barrier associated with significant improvements.

We can choose content, pedagogy, and instructional workforce activities that support each other and offer the promise of being useful in the long range. Prior research, primarily funded by NSF, has developed and tested cognitive-based educational applications of technology focused on advanced content. Research shows the importance of new scientific methodologies for engaging teachers in learning. The “facets” assessments explored by Minstrell and others show clearly what type of knowledge teachers of science must possess to be truly effective.

Content

Methodological advances are challenging the compartmentalization of science in existing disciplines. There is no such a thing as a disciplinary nanoscience. Science can be appropriately considered a distinctive, unique discipline in terms of its interplay between theory, models, and observations. The sequential, discipline-oriented science curriculum does a marvelous job of culling learners by focusing on the “how” at the expense of the “why” and “what”. The proper balance among why something merits scientific study, what that study can provide, and how to conduct the study is as yet unclear. To understand it, we must gather learning data from projects that attempt different emphases.

Pedagogy

The standard science curriculum and thus teacher education is based on a deductive (science in its final stage or ready-made-science) paradigm, and approaches scientific literacy mostly as knowledge of contemporary scientific theories and explanations, topic by topic. But inquiry practices (another aspect of scientific literacy that reflect the process of science) remain at the margins of science education curriculum expectations and have not entered into the standard ways of assessing student learning. Sustained improvements require that this marginalization has to disappear.

Teachers have to be able to adapt innovations, and for this they need to understand the connections between content-based, skills-based, and inquiry-based views of learning science so they can be flexible in their pedagogy.

Complex systems and modeling approaches may provide an answer to the frequent “less is more” calls to streamline. Such approaches often appear as calls for a curriculum based on conceptual strands (form and function, conservation of matter and energy, energy exchanges, and so on) that are common across STEM disciplines. These arguments are behind the calls for a “Physics First” high school curricula.

Instructional Workforce

We owe teachers in the trenches the tools for building their own capacity and knowledge not as a haphazard mix of fads and common sense intuition, but as a foundation that can be built upon and refined.

Success implies longer-term commitments for supporting teachers. Primarily, outcome measures must go beyond end of year tests since the retention of ideas and the ability to apply them develop not course by course, but across courses. Looking at outcome measures that take longer to become visible, such as increased graduation and attendance rates, enrollment in advanced placement or other high level classes, college or graduate study enrollments, even the nature of summer jobs, will help parents and schools allow teachers the space to change their pedagogy. Some high schools won the flexibility to institute significant changes in their science curriculum by negotiating upfront their graduates’ admittance to good colleges without consideration of test scores. This commitment allied parents’ concerns and shows that experimentation is possible.

Success implies longer-term commitments for supporting teachers.

We also know that when teachers undertake changes in pedagogy they go through the same “productivity dip” we all experience when we start on a new methodology. We should keep in mind that, when support is sustained over time outcomes can be more significant.

We know much about teacher enhancement, given its recognized importance and the significant resources it uses. But though credible examples of the effective use of technology to support teacher peer learning and receive professional content support exist and are ready for scaling up, most teacher enhancement activities use discredited methods.

Final Words

None of the ideas and implied proposals in this paper is new. In addition to providing a digest of lessons learned, its contribution, if any, is to present a coherent view of many aspects of the complex fields of learning and education. The intention in presenting what we know in each of them is to enable an emergent view of coherent next steps that is more than the sum of its parts.

If all of these ideas were to be attempted together, this is a tall order. But the goal is achievable if we work purposefully in a scientific experimental mode—iterating thought, action, analysis, and reaction. The main point I wish to make is that these experiments MUST involve attention to the whole—at least to document and understand the dynamics of the whole in such a way that the knowledge obtained can be analyzed coherently.

RECOMMENDATIONS FOR ACTION

The recommendations suggested distinguish between direct and indirect action. By direct action I mean direct intervention in schools—specifying content, process, and a theory of action to be subjected to implementation research. By indirect action I mean supporting the design and study of experiments that probe aspects of the infrastructure, for example by defining an educational parallel to the agricultural extension service as a testbed for analysis. This type of “for the good of the order” action, calling on the appropriate collaborators from practice, research, and policy, may not result in direct student learning in the short term, but will lead to a the creation of a research community that does not currently exist, to significant advances in the knowledge about policies for sustainable change, and hopefully to increased STEM learning in an increasing number of schools.

Indirect actions that the Commission can undertake will influence the context for education reform; these recommendations for action promise to influence policy at many levels, from the public to Federal agencies and legislative action. Indirect action has broader systemic goals that can strongly influence how resources are allocated⁵ and that direct action projects often can’t address.

One step to consider for each case is creating a working group of visionary stakeholders to formulate an action plan and perhaps act as an eventual Steering Committee. Please note that partial steps toward these actions may exist and can be built upon, but because they are not coordinated it takes too long to learn from them and transfer their lessons. The objective of these recommendations is to hasten action by building on what is known.

Possible Indirect and Systemic Actions

• Define a feasible educational parallel to the agricultural extension service as a testbed for implementation, study and analysis.
• Create dynamic agent-based models of education systems or subsystem prototypes (for example, a series of Sim-Schools or Sim-Districts) that can be run and tested against existing data from reform cases, and use the models to train practitioners in understanding where systems constraints impede progress.
• Create tools to support the study of implementation around specific cases, such as models to be used by practitioners to study the process of implementation.
• Select and study some successful long term interactions of intermediaries with schools or districts—intermediaries could be museums, university groups including but not limited to colleges of education, non-profit research centers, etc. that bring flexibility and experimentation to the system.
• “Embed” researchers, journalists, scientists or policymakers in some projects to involve more directly the public at large in understanding the efforts needed to carry out sustainable reforms, and the need for “clinical research” in education that parallels what transpires in medicine, law, and engineering.

Page 3

Possible Direct Actions

• Create school district “pedagogical” laboratories for teacher enhancement around the use of challenging STEM content. Have expert teachers teach students after hours (or use videotapes), with other experts (e.g., researchers) present and have other teachers discuss the pedagogy.
• Choose a system with capacity to absorb a particular challenging STEM innovation and the infrastructure to maintain it, and help the system work with a school that has already implemented the innovation to study the process of adaptation to local conditions.
• Help schools develop a new culture by integrating technology throughout—across courses, analyzing assessments, developing joint interdisciplinary learning goals, discussing video journals of good practice (or of their own practice), streamlining communication with parents, etc.

Summarizing the main ideas: None of the ideas and implied proposals in this paper is new. The paper’s contribution, if any, is to present a coherent view of many aspects of the complex fields of learning and education. The intention in presenting what we know in each of them is to enable an emergent view of coherent next steps that is more than the sum of its parts.

The goal of STEM education improvement is achievable if we work purposefully in a scientific experimental mode—iterating thought, action, analysis, and reaction. These experiments MUST involve attention to the whole—at a minimum to document and understand the dynamics of the whole implementation process, including both policy and infrastructure.

Context for Education Research. There is a need for research that analyzes large-scale implementations. To offer new insights into solutions, implementation research has to create and study prototypes and models of infrastructure reform that can scale-up and become sustainable and systemic.

Disseminating Innovations. Each innovation is a system of innovations that include lateral innovations that must be taken into account for the innovation to succeed. An implemented innovation can’t be independent of the system that uses it. Successful diffusion of an innovation is an explicit process of adaptation. Neither artifacts nor ideas suffice as mechanisms for successful dissemination.

What We Can Learn From a System Science. Often, optimal solutions are counterintuitive. But in education, everybody’s intuition leads to action, when it should be subject to evaluation. Understanding the dynamics of interaction between all the parts of an education system (be it a classroom, school, district, state) requires looking at the many changes taking place in successful or failed implementations.

What We Know: Content, Pedagogy, And Instructional Workforce. We must choose content, pedagogy, instructional workforce, and policy activities that support each other and offer the promise of being useful in the long range.

Inquiry practices called for by scientific literacy and that reflect the process of science remain at the margins of the science education curriculum and testing. Sustained improvements require this marginalization to disappear.

Outcome measures must go beyond end of year tests to facilitate pedagogical explorations by teachers—for example, increased graduation and attendance rates, enrollment in advanced placement or other high level classes, college or graduate study enrollments, even the nature of summer jobs that students take.

What We Know: Action for Impact

While new technology applications in different aspects of society have improved our national well being and supported international competitiveness, they have increased the demands placed on the educational system by increasing the price we pay for a scientifically and technologically disenfranchised segment of the population. The features of a world where technology is pervasive require the education system to prepare everyone to a standard that used to be reserved for a small elite.

Unlike other education reform periods, the knowledge and skills all students need to prepare them for the world of work and the work of science are not socially agreed upon. Many of the lessons from the past do not seem as relevant, but a clear vision of the future of education is absent from curricula and public perception. “Strengthening the basics” looks like an option to many, but there are new social and economic imperatives, not only educational ones, to add to the traditional three Rs requirements for scientific and technological literacy.

Educators and policy makers have not been inactive. As science and technology play increasingly important roles in our society, educators have tried to incorporate significant features of technology into the educational process. However, the need for present improvements forced most of the efforts to be add-ons to traditional classroom practices and thus minimize the impact that new, emerging innovations can contribute. Interactive, high performance uses of technology, which are a staple of scientific and economic practice and a driver for change in the workplace, are not pervasive in schools. More troublesome, to the extent they are available they reflect socioeconomic levels more than educational needs. The fault is not of the tool, which is value-free and has proven its worth outside schooling. Nor is it the fault of individual educators and school boards, who often find themselves too strapped for resources and knowledge and subjected to external pressures, to risk innovating on their own without some assurances of success.

What stands in the way of more general positive changes? In the words of Richard Elmore:

The pathology of American schools is that they know how to change. They know how to change promiscuously and at the drop of a hat. What schools do not know how to do is to improve, to engage in sustained and continuous progress toward a performance goal over time. So the task is to develop practice around the notion of improvement.⁶

And improvement is iterative.

If we are to affect education and reach the national goal of making science and mathematics education competitive in the world, it is unlikely that improvements in delivery and in efficiency will suffice. We’ve known this for quite some time. We’ve also known the critical role that technology could play in a more profound transformation. I can point to the 1997 PCAST report as a thoughtful analysis with its call for significant investments in education within a technology framing.

Research does show that technology makes a significant impact in STEM education, but we are not applying broadly enough what we already know. And to change this limitation, we need to act on the educational system’s constraints. Policy must be confronted, and a different type of collaboration between research and practice must become more prevalent. Implementing an innovative (transformative, in current parlance) strategy requires a different way of framing the problem.⁷ After all, the aim of innovation is not to make small affordable changes, but to make a significant difference.

If these statements are correct—we know the need, and we know how we could use technology to improve education, including STEM education—why is that knowledge put only sporadically to practice? My experience with a plethora of research programs around the overlap of science, technology, education, and research, has made it quite clear that we need to think of education and its funding together as a joint system, not as isolated components of two independent systems. The many commendable federal research programs implemented in the past derived from a linear view of knowledge transfer that posits that development and application are separate activities. In fact, as Elmore states in the lines cited before, the result is continuous and disconnected change, when what is needed is an iterative integration of research and implementation. Research programs were not followed by the necessary support responsive to the needs of implementers, nor by the research required to understand and adapt to such needs. Implementers need resources (which include time to learn) to translate what is known in general into what they practice locally. Mandates and exhortations to be effective are not sufficient.

Note that the policies that treat teachers as “deliverers of a given, fixed curriculum” who need to be “enhanced” to use it, often prevent schools from iterative improvements in learning, and, more injurious, do not promote teacher learning. Schools are the units whose culture of improvement must be supported so that improvements become iterative and sustainable, since individual teachers and principals move. The reasons behind Elmore’s quote speak to the concept of linear change (research to curriculum/pedagogy to implementation without feedback) leading to disconnected school changes, rather than to iterative school improvements.

A colleague once made the following analogy: you go to a doctor’s office and sit with other patients. Then the nurse stands up and says “The doctor says take one aspirin with a glass of water and call tomorrow.” We laugh because we know that doctors and nurses have skills in diagnosing an individual’s problems and suggesting the relevant treatment, not only in applying general rules. We should look at teachers as diagnosticians able to understand what individual students learn. Delivering a regimented curriculum does not help them accomplish this. And just like in medicine, increasing the professionalism of the teaching workforce does not imply teachers will have total autonomy.⁸

Although I am very excited about the curricular changes that can be attained with modern uses of technology, I will not talk about most of them here. Nor will I talk about software, even when my natural tendency would be to delve into modeling and simulation as ways of thinking that underlie STEM practice and education. I am a computational chemist, after all. Nor will I emphasize how cognitive research on learning should be integrated into education, as we are doing with my LIFE colleagues. Much can still be done in these areas, but I am more concerned here with using what is known and what will be known in the future—areas that do not receive the same level of attention.

Most relevant to this discussion are uses of technology that deepen the understanding of how science matters, and expand students’ ability to apply what they know. Particularly exciting examples are: modeling and simulation software under student control; software to use existing scientific databases in solving specific problems, probes to gather data and software to analyze it, software supporting collaborations that allow individual groups to develop and communicate their particular expertise to others, software supporting argumentation about scientific controversies, and so on. A fuller account will be given in Appendix B

For the same reason, I will not comment on measures of learning and accountability. Though not as forward looking as research on using technology for STEM education, exciting new assessment ideas are being studied, and we know very well the limits of one-size-fits-all measures. Again, more could be done with existing knowledge. The barrier is the increased cost of using more appropriate measures. Standardized one-size-fits-all tests can be used with many standards and are also cheaper to administer. The possible advantages of scale for more research-based assessments, including those embedded in software that can influence teachers’ pedagogy towards deeper learning goals, require implementation research investments that only now are being considered⁹.

Drucker’s warning is worth remembering: if we want change, we should not think in terms of investments that lead to improvements but of opportunities that lead to fundamental change. The question that I pose is: Why is the knowledge we have not put to use in education with the same vision and consistency with which knowledge is put to use in medicine or ecology? Are there aspects of our own practice as researchers or policy makers that we need to rethink and change?

We need to consider infrastructure, not only pedagogy, if we want to make a significant, lasting, and consistently relevant difference.

We need to consider infrastructure, not only pedagogy, if we want to make a significant, lasting, and consistently relevant difference. By infrastructure I mean the human and technological underlying support required to shift some of the weight of change from the education practitioners to a support system—libraries would be half as useful without librarians and card catalogs. We will need significant large-scale experiments with infrastructural changes to exercise our thinking since the existing ones have not come up to the challenge posed by new science and technology requirements. Examples could be studies of integrating teacher preparation with teacher enhancement in given school districts; using museums or other technology-rich institutions in supporting classroom activities; continuous on-line support for teachers in forming communities of practice, etc. All these and similar programs exist at smaller scales and we need to learn from them and study how to scale them up organizationally so they become embedded in the fabric of the education system. Where would science be without the R&D on the human and technological infrastructure and its uses in practice supported by over fifty years of NSF funding?

There is nothing wrong with incorporating an experimental ethos into interventions on education practice; but this is often seen as “experimenting with our children.” In fact, supporting teachers and schools in experimentation from which lessons can be learned can benefit students. Just consider that most educational mandates from above are in fact experiments that do not allow the benefits that could accrue from analyzing failures and taking corrective action.

Context for Education Research¹⁰

“Research” refers to a continuum of work undertaken to answer questions posed by a given group or an individual to provide information or knowledge usable to the same group or to a different one. At one end of the continuum, research is defined by researcher questions that push the boundaries of knowledge. At the other end of the continuum, research is defined by large-scale and contextual experiments, as defined by implementation research questions that frame robust applications. What changes from one extreme to the other is the relation of the individuals that pose the research questions to the audience that needs to act on the basis of the knowledge generated.

Research that tends to focus its sights towards the center of the continuum shown in Figure 1, as much of education research does, (for example, by developing solutions where content and pedagogy are a given tenet, by analyzing current classroom practice, or by evaluating materials) is crucial as a knowledge base for change, but falls short of understanding and enabling the paradigmatic changes required by the educational system, represented by the right half of Figure 1.

Basic innovation research and development in the content and in the tools of science and mathematics education has to inform and be informed by, and be refined based on applied, robust, large-scale testbed experimentation. The steps in Figure 1 go both up and down. Research and development on technology-impacted curriculum and pedagogy has to be coupled with the creation and study of prototypes and models of infrastructure reform that can scale-up and become sustainable and systemic. Case studies are not enough.

Table 1 differentiates between types of classroom studies and their outcomes to place implementation research ^11,12 in an ecology of studies that bridge traditional research and large-scale practice.

Although the ideas behind implementation research are not new, they have taken a new life within the public health community, confronted with the gap between rapid advances in the medical sciences and the continued public health problems such as nicotine addiction, obesity, sexually transmitted diseases, etc. ¹³ Ref. 13 states: “Over the past decade, the science related to developing and identifying “evidence-based practices and programs” has improved-however the science related to implementing these programs with fidelity and good outcomes for consumers lags far behind… The results of this literature review and synthesis confirm that systematic implementation practices are essential to any national attempt to use the products of science – such as evidence-based programs – to improve the lives of its citizens.”

Ideally, there need not be a separation between the styles, which exist in great part because the communities that engage in each are separate. Reducing or eliminating this separation implies the development of different funding and tenure criteria on the academic side, and a more experimental ethos on the practice side. The argument behind education implementation research is precisely that with current advances in research on technology, pedagogy, and cognition, we are in a position to start focusing on understanding how best to use what we know, and build on this understanding to frame the next stages of R&D.

Note that all the types of research are needed and all call for a strong collaboration between researchers and practitioners. Whereas in the first three the bulk of the support usually goes to researchers, the opposite should be true for the last two types, which answer implementation research questions.

The level of resources available for research is usually based on the needs of studying individual innovations in isolation. What is needed now are sufficient resources, coordination, and people to ask the crucial questions of significant impact, sustainability, and scalability that can make a visible dent in the pressing national problem, and ensure technology’s seminal participation in large-scale systemic interventions. It is neither efficient nor effective for different agencies with a say on education (e.g., States, federal agencies such as USDoE, NSF, NIH, private foundations) to support contradicting policies that do not maximize resources and minimize the timeline for improvements.

Funds for a few large-scale implementations have been available, but the research that should take place for the field to benefit from what are really experiments—to treat them as research testbeds—must become available as well.¹⁴ One significant prior “social experiment” whose success we should remember is the agricultural system of land-grant universities and extension services that made possible the step-wise supported “translation” of research from the first-level research universities to the local needs of the farmer (e.g., local climate, soil characteristics, transportation costs, etc.).

If we think of what stands in the way of such a supported “translation” in education an issue that comes to mind is the lack of appropriately prepared people in Education Agencies (local or State) and even Department of Education centers. This lack of preparedness is a consequence of the lack of appropriate levels of research funding.¹⁵ Research produces more than knowledge—it produces people with a different mindset about research. Not all people trained in STEM disciplines will be new researchers. If it were so, industry would not have the crucial personnel to evolve new knowledge into more practical knowledge and new products. It may be interesting to consider the differences between extension services at their heyday, the integration of laboratories in industry, and existing LEAs. Can we explore a research-savvy integration of learning objectives by focusing existing mechanisms within schools and districts on R&D learning? Advantages of such experimentation would be localized knowledge, integrated testing and instruction, and well-prepared people that can support continuous improvements.

Additional research and development and research-informed systemic experiments by themselves will not lead to sustainable change, unless we recognize that the audience for the outcome of these experiments is also policy makers, including local school boards. The public at large has to support sustainable change, either directly or through taxes. The whole process, from research to testbedding, has to be open with researchers, policymakers, and the public participating actively in the definition of the problem, and in the analysis of prototype results. Demonstrations are not convincing; participatory experiments can be.¹⁶

Disseminating Innovations

Research shows that teachers always modify the materials they use, and that “teacherproofing” materials does not work. This is especially true in STEM areas where prior student common sense knowledge leads to errors. Teachers adapt materials¹⁷ rather than adopt them, whether we want them to do so or not. Research also shows that, properly supported, teachers learn content in deeper ways in the process of adapting well-designed materials for their own use.¹⁸

Diffusion of an innovation should be seen as an active, dynamic, and explicit process of adaptation. In this sense, neither artifacts nor ideas are sufficient as mechanisms of diffusion, which can be defined as appropriate use of an innovation by others.

An “innovation” never comes alone—each “innovation” is in fact a system of innovations that includes changes in classroom management; professional development; and the interactions between practitioners and parents, colleagues, and supervisors—a whole set of lateral innovations that must be taken into account for the innovation to succeed. A successful innovation is in fact a strategic set of innovations for the multiple levels of the particular system. It is not a series of random walk innovations, and cannot be independent of the system in which they are applied. How the system responds to the innovation has bearing on how the innovation will fare—treat the innovation as “this too shall pass” rather than give it the supports it needs to become embedded. For example, when “problem-based learning” is instituted in a classroom, students will move around and talk among themselves as part of learning to apply their acquired knowledge. But principals or parents that equate noisy classrooms with lack of learning or discipline will not accept the change. The same can be said about problems with multiple valid answers, or unorthodox mathematical procedures. The system as a whole must have the capacity to absorb the innovation at several lateral levels and, in the ideal case, develop the infrastructure to maintain and perfect it.

The innovation should be designed with attributes that permit principled adaptation, such as templates and information about design characteristics. Different innovations bring up different system requirements, so at some point there should also be knowledge of what is required in other parts of the system that will adapt the innovation: Do we need teacher development for the teachers that will have the students next year? When and how can we evaluate the outcome of the innovation? If this innovation works such that other teachers want to use it themselves, how will we facilitate it? Should we prepare a more savvy technical support person, or one that supports pedagogy changes?

The system that receives the innovation must be aware of its deficiencies and be able to self-correct to maintain the innovation’s functionality. Skills, values, beliefs, a critical mass to support the change, data-based decision making driven by assessing learning and cognition are all important. Participants must allow and expect failures in the process—there must be space for exploration and experimentation.

Diffusing an innovation is an induction process that requires additional intellectual power and takes time to show results. Successful adaptation may need the long-term participation of a researcher to act as a flexible reflector of ideas. It will also benefit from access to a support infrastructure focused on learning (e.g., literature, specific topical expertise, modifications to the innovation, colleagues’ experiences, etc.). It will need specialists available to teachers and others, to provide advice and data on the hardest problems. This is where the agricultural extension system model has much to teach us.

Education should adapt—rather than adopt—ideas from agricultural extension. Characteristics such as distribution of formula grants may not be optimal for education. But two lessons are worth keeping in mind.

One is the vital role that state universities play in the process. The Cooperative State Research Education and Extension Service (CSREES) was formalized in 1994 as the federal member of a cooperative group that included not only extension services of long standing, but also the Land-Grant University of each State. These institutions have a critical mission within CSREES—extension. “Extension” means “reaching out,” and— along with teaching and research—land-grant institutions “extend” their resources, solving public needs with college or university resources through non-formal, non-credit programs. They also have a state mandate for this service. Thus, universities have become an integral part of the process of localization—of implementation research under local conditions.

In addition, “these programs are largely administered through thousands of county and regional extension offices, which bring land-grant expertise to the most local of levels.”

Service-oriented, research-based, and multilevel partnerships that include federal government grants, clear state mandates, and links between service-oriented groups within universities and local offices (the hallmarks of the extension system) are able to look at timely priorities from many perspectives and work on ways to influence policy.

What We Can Learn From a System of Science

“The aim of adaptive management (AM) is to improve (environmental) management through ‘learning by doing’ and understand the impact of incomplete knowledge, but AM more commonly consists of ad hoc changes in managing (environmental) resources in the absence of adequate planning and monitoring.¹⁹,²⁰“

Replacing “environmental” with “educational” in the above quote is instructive. Ecology and education share a structural problem: to act locally from general scientific knowledge on the basis of incomplete local knowledge, through many independent local actors with different backgrounds, and without major sources of funding to understand the all-important local details (including economical impacts, the beliefs of local actors, and local power dynamics).

Three ideas are important here. One is that the focus is on managing action. Another is that action is local, and so must be its support. The third idea is that experimentation to learn more about the operation of complex systems is an essential feature associated with adaptive management. Adaptive management has the attributes of being flexible, encouraging public input, and monitoring the results of actions for the purpose of adjusting plans and trying new or revised approaches.²¹

Adaptation and flexibility go hand-in-hand. If we want adaptation, we need to think of flexibility as a core aspect of managing a professional classroom. Foremost, by creating conditions that support a classroom professional in “being flexible, and monitoring the results of actions for the purpose of adjusting plans and trying new or revised approaches.” That is to say, an ethos inimical to the practice of mandating solutions based on fads and beliefs, to be discarded when they do not provide immediate success, to be replaced by another mandate. Note that there is a use of experimentation in ecology missing from education: the use by researchers in developing intuition, looking for leverage points, and testing conjectures via simulations, including agent-based modeling.²² Interestingly, such theoretical experimentation leads to the knowledge that very often, optimal solutions are counterintuitive.²³ Counterintuitive optimal solutions have not been given much space in discourse about education. Just the opposite, everybody’s intuition—informed or not—leads to action.

Change is a process, not an outcome; a “one shot” improvement implies that the environment that nurtures the current failure is left unchanged. School administrators want to know as soon as possible if the changes being tried are moving in the right direction—well before sufficient time has passed to gauge its final outcomes. Research can help develop intermediate measures of progress, dynamical models and other ways that allow formative assessments of individual and collective progress.^19,20

Page 5

The Education System and How to Influence Its Core^24,25

The U.S. K-16 educational system is conventionally defined as the system of public and private schools and colleges that offer students formal education from kindergarten to college graduation. For research purposes, however, the system must ultimately be defined by its constituent elements and its dynamics, such as what material and human resources are tightly enough coupled and interdependent in their behavior. Likewise, we need to know what is the range of timescales characteristic of the critical processes that enable the system to maintain itself. What are its significant levels of organization-not only control hierarchies, but characteristic emergent processes and patterns at each level? If we examine all the institutions that can act as resources for students and their families, we must include informal educational institutions such as science museums and learning sites afforded by mass media, print publishing, and interactive communication technologies.

Formal organizational hierarchies propose one starting point for identifying levels within the core educational system: individual learners and teachers, small groups, classrooms, departments, schools, districts (LEAs), States (SEAs), and Federal Agencies. How would a dynamic analysis take into account the varying timescales at which different levels of the system function, and what would the units of analysis be? How do the changing priorities, populations, and problems of a local community influence the larger educational system’s agendas and programs?

The classrooms studied by educational researchers can be seen as complex systems with sophisticated feedback loops, local constraints, use of resources, and nonlinear causality, similar to biological or ecological systems. Such systems cannot be understood by considering isolated pieces of the whole, and could therefore benefit from integrated system research strategies, whose development needs to be supported. Beyond independent scholarship, educational researchers should also play a role as intermediaries who enable experts in other disciplines, educational practitioners, funders, and policy makers to understand each others’ views of these perspectives.

We can expect that the new tools of complex system analysis will help us understand the potential impact on the educational system of innovations and interventions, and help us predict the paths that different efforts at systemic reform can follow. These techniques are in common use (and expensive) in the private sector. But we don’t know, we have not tried them in education in a large and systemic enough way. We can expect that these modeling techniques can help identify critical relationships within the educational system that resist systemic change or that afford opportunities for new alternatives. If the answers to any of these questions are to be ”yes”, we will require collaboration within a diverse new community of researchers seeking a common framework for sharing ideas from different disciplines and approaches to both complex system analysis and to education. There is urgency for the formation of such a community. The response of the educational system to the new demands of the public for reform and new opportunities technology affords can either be guided by the best ideas of the research community, and by research- and data-driven decision-making, or it will be guided by other forces.

Whichever level of organization or subsystem is the focus of our concerns at a particular point, we can always ask a series of key questions motivated by the perspectives of complex system theory-starting with what is the system and what is ”environment”?

For example, what next higher level of organization determines constraints on the dynamics at the level at which you intervene? What degrees of freedom remain after the constraints are allowed for? What characteristics of lower-level units determine the range of possibilities for action? What kinds of matter, information, and energy are exchanged? What social networks provide structure and stability?

Any pedagogical innovation introduced into a school system is in fact a series of embedded innovations at levels above and below the intervention, and strategies for all levels have to be considered coherently. I emphasize the importance of the concept of “drivers for change” with respect to three other ideas (1) the use of hypothesis testing; (2) the need for computational experimentation (modeling) to predict patterns of change, and (3) the inescapable fact that all education is local, and thus the championing of change, and its drivers, must be localized.

How would we model and analyze issues like these using the concepts and techniques of complex systems theory and adaptive management that explore the flow of information across system levels, such as developmental or ecological biology? Given access to data and expertise about the educational system, how would one approach information flow and use in school systems? Given the collaboration of others who could offer different insights about complex system behavior, how would policymakers, educators, and researchers begin to formulate any one of these problems for actual study, using the knowledge generated by organizational theorists? Essentially, how can education benefit from work in organizations and systems that have been better studied?

How well could we design today a ‘SimSchool’ or ‘SimDistrict’ school or school district simulation program-not just to model an existing system, but to enable us to create alternative systems and study their evolution over time, their needs and problems, and their probable outcomes? And, more important, to develop intuitions about these processes. What kinds of schools would students design if given access to an appropriate version of this software? And how would they evaluate various designs proposed by others? Who would we enlist in the team to create such a software package? What research literatures would we want to consult? What is not yet known that would be needed to complete the project? What kinds of data would be needed to realistically attempt such a project?

Given the interest in easily quantifiable parameters of the system, such as school budgets, teacher qualifications, and student test scores, we need to know how much value added there might be from a complex system model compared to more static statistical analyses. Agent-based dynamical simulation models hold the promise of enabling us to explore potential effects of changes in both quantitative parameters and in qualitative interaction of variables to produce observable statistical relationships. Complex systems models are designed to model change and dynamics, especially qualitative change. To build effective dynamical models of educational institutions we will need to know not just what people do, but why they do it, how they might imagine things being different, and what they would really want to do.

Even if such systems models are not predictive in any detailed way, they can still be useful in identifying possible alternatives, potential problems, and overall qualitative features of the change process which may not be intuitively evident to a linear logic of cause and effect. In complex systems every causal chain is mediated, and many chains branch and loop back on themselves in complicated webs of mutual interdependence, self-regulation, and amplification of effects. This conceptualization is consistent with Michael Fullan’s “systems at the edge of chaos” view of education.²⁶

No mention of the data needed for analysis and the development of theoretical models can leave out considerations of sharing information across projects (i.e., across localized case studies). This sharing and aggregation is a major problem for a topic as dependent on localized conditions as education reform.

Perhaps the most important lesson is that adaptation of models for system reform to local conditions matters more than efforts to replicate successes elsewhere, without extensive knowledge of how the systemic variables differ between environments. This “localization effect” points to the importance of determining whether any single complex system model can be both general and specific enough that it can include design templates to identify key local parameters that need to be set.

Discussion among researchers working with large school systems²⁷ indicates that in most cases it takes of the order of 5 to 10 years to establish effective collaborations between researchers and school systems²⁸, and that during this period there may be a need to re-negotiate and re-commit to goals and strategies developed together whenever there are major changes in leadership or personnel on either side of the partnership. The fruits of reform efforts tend to become visible only after at least three to five years. Any evaluation and tests of scalability require at least a second or third cycle of enlargement or replication, implying a minimum of 10 years’ scope for models of effective change. Clearly, this timeline is problematic for more than one reason. First, it implies that we need either to wait for “final resolutions” on “what works”-or look only at small improvements that can be studied much faster. Second, it implies that what we know from research will remain static while we study how to use it. Obviously, neither implication is true. The answer lies in starting as soon as we know “enough”—the contention of this position paper—and focusing on mechanisms for continued learning and improvement that will take us much further ahead that what we can envision now.

For reform efforts to be maximally adaptive to changing environmental conditions, an iterative process is needed in which plans are continuously modified in response to issues that only come to light once implementation has begun. Successful multi-year reform processes include periods of consolidation of gains; these periods provide a respite to plan for needed changes and for people to become comfortable with one set of changes before contemplating others. In this sense, sustainable change should be viewed as a “stepwise” process, in which advances alternate with periods of consolidation. This stepwise strategy promotes buy-in from skeptics, allows for non-disruptive change, and establishes a culture of continuous improvement. Under these conditions, modeling of different “scheduling paths to innovation” may lead to a more integrated and sustainable organization that is resilient with respect to changing future conditions.

Sustainability was found to have two key aspects. The first is the need for a match between stakeholders’ expectations regarding the nature and pace of results and the ability to provide persuasive demonstrations of timely effects. Early successes, as judged by stakeholders, are crucial for sustaining the reform process. The second is the relations among the timescales of change processes in different elements of the system, and between the system and larger social-political-economic systems in which it is embedded and on which its functioning depends.²⁹ These lessons point to the importance of multiscale modeling techniques for educational change, and particularly to multiple timescale models. When we consider that many key structural features of educational practice (e.g., student-teacher ratios, use of textbooks, age-grading, local-taxation funding, curriculum areas, teacher training institutions) have been stable on timescales of a century or longer, we can infer that there are powerful system-regulatory relationships maintaining this stability. Reform mandates and implementations, on the other hand, are formulated and expect results on timescales of the order of a decade or much less.

Complex system models are needed to understand why so many features of the educational system do not change, and under what conditions they will change. Realistic models, based on detailed case studies of reform efforts and on general system modeling principles, may help us understand if such assumptions are realistic or not. We need to know whether or not current modest reforms have any realistic chance of producing major gains at the large scale in realistic timeframes. If it should appear that more radical re-engineering of the educational system is needed, we will need to understand very well the functional roles and interdependencies of current structural features.

What We Know: Content, Pedagogy, Instructional Workforce

It is probably fundamentally wrong to imagine that the way to ‘progress’ is to educate each generation up to maturity to be exactly like its predecessors, and then expect them to radically innovate. That model is a recipe for inhibiting social and cultural change.³⁰

If the goal is impacting the system, not only a few classrooms, the timelines for achieving impact become much longer. Focus should then be on content that reflects the evolution of science and technology; it will not do to limit one’s thinking to the way science is expected to be taught at present. We must be squarely focused on the middle of the 21st century, with all this implies for both content and technology, including using technology as a cognitive, transformative tool that leads to individual empowerment. As many projects have proven, this is doable within current science education standards. Nanotechnology can be used as an example, as demonstrated by the work that led to the identification of nanoscience learning goals and their correlation with physics, chemistry, and biology education standards.³¹ Nevertheless we should not expect these standards to remain unchanged. Rather, we should remember that the way to achieve a desired future is to build it.

Though we should act on the basis of forward-looking content, content does not need to be our primary driver for action. Based on what we know from research, pedagogy makes a crucial difference in what students learn, and focusing on pedagogy implies changing the teacher’s classroom activities. This is what Elmore, Cohen, Cuban, and others refer as the ‘unchanging core of education,’ what provides stability within the ever changing policy and personnel environment. We can certainly move without paying attention to the core. But then we can surely expect to have to take similar actions over and over again. Reinventing the wheel is a common occurrence in education, derived in great part by the localized nature of educational action, and by the failure of attempts to change the core. But it also depends on the beliefs of scientists (outside the learning sciences) who know so little about learning in general and about learning in their own fields in particular, to say nothing about pedagogy, that their ideas about improving education are often counterproductive. Were your actions to change just this perception, and focus all scientists on the importance for action of learning about learning, your contribution would be significant.

Fortunately, there is a plethora of content materials that has been studied and that could be used. Prior research, primarily funded by NSF, has developed and tested cognitive-based educational applications of technology focused on deep, meaningful content, and most of it deals with topics suitable for forward-looking education. Publications such as the AAAS Atlas of Scientific Literacy provide critical interdisciplinary connections between fundamental scientific concepts. There is also significant research that shows the importance that focusing on teaching deeper concepts using new scientific methodologies has for engaging teachers in learning content in transformative ways.³² As Hestenes points out, teaching teachers how to use new materials—in his case, modeling as a basis for physics and chemistry—is a nonthreatening way of educating them as professionals and providing them with a firmer base with which to engage student thinking. The “facets” assessments explored by Minstrell and others^33,34 show clearly what type of knowledge teachers must possess to be truly effective, and how to see where a given student’s problems with understanding lie. Such leading edge systems promise to give teachers better diagnostic tools. Other advances, such as the PADI (Principled Assessment Development for Inquiry) will help, and hopefully simplify, the work of assessment developers.³⁵

We are in a position to choose content, pedagogy, and instructional workforce activities that support each other and at least offer the promise of being useful in the long range. We should not be swayed by ready-made materials that may have solid evaluations but point to past methodologies. Experimentation and scaling up should be based on forward-looking materials.

Page 6

Content

Choosing materials that serve the needs of all students does not mean staying close to basics-just the opposite. Learning outcomes for all graduates should be defined starting from what has become possible to teach and learn with the advent of technologies and methodological, fundamental changes in the practice of science. Thanks to technology, science is becoming more interdisciplinary and more evolutionary (for example, biochemistry and materials science are diverging practices of what was chemistry; looking at the energetics of reactions and their dynamics has changed chemistry). Mathematical experimentation around models that embody what we now about phenomena have become commonplace, and visualization techniques support the uses of multiple representations of abstract concepts in professional practice.

Many researchers and faculty are exploring curriculum changes, and the need for change is not under discussion. The curriculum is slowly changing and both practical and philosophical arguments for deeper or for specific changes are discussed even in the popular press. Whether these changes are as deep as they need to be is another matter. The objective should be to address directly the intellectual issues at the core of the science curriculum itself. In practice, the core is defined by what students should be expected to know as embedded in standardized tests. These tests suffer for the most part from assumptions about science and science education that do not reflect what students will need to know and be able to do upon graduation. The tests are static, memory-driven, and disciplinary when they could be performance-oriented and reflect the state of science and technology in society. For example, the standards for both mathematics and social sciences include issues of data analysis, which could easily be integrated into social science problem-solving based on mathematical manipulation of data. Tests that probe such data-based argumentation could lead to changed pedagogies in both disciplines that reflect the standards we endorse.

Methodological advances are challenging the compartmentalization of science into existing disciplines. There is no such a thing as a disciplinary nanoscience. Science can be appropriately considered a distinctive, unique discipline in terms of its interplay between theory, models, and observations. In my view, the sequential, discipline-oriented science curriculum is better adapted to educate a limited number of scientists and engineers, and to take its time to do so. It does a marvelous job of culling learners by focusing on the “how” at the expense of the “why” and “what”. The pace of content introduction into the curriculum is driven by the attention paid to techniques of performing tasks (“how”). Even students with scientific professional expectations do not acquire scientific ways of thinking (“why”, “what”) that could help them address complex issues of everyday life before higher division college education and then only in their own specialty. The proper balance in science education, at different levels, among why something merits scientific study, what can that study provide, and how to analyze and solve the problem, has not yet been seriously characterized.

The above comments do not imply that the curriculum has not changed or evolved in a hundred years, but that its underlying expectations and structure assume the current sequencing and does not adapt easily to anticipating and shaping the future. The sequencing of courses, the use of mathematics in science, the concepts to be learned in each domain area, culminating in what students are expected to know and be able to do at the end of different levels of schooling, are changing too slowly despite profound changes in the tools accessible to students and in the levels of scientific knowledge required of the citizenry.

This holds true even when research has shown that there are tools that can help all students master more challenging and more relevant scientific concepts, and that the current curriculum is not serving all students well. Calculus—the mathematics of change—is an area where research shows this clearly, and where we know that failure closes access to higher education for many students.³⁶ The disconnect between science and mathematic education as generally understood and society’s needs has become too large. The mechanisms to sort students into those predetermined to be science-bound and those not-science bound are not serving the evolving demographics of the country’s needs, not to mention the individual students’ needs.³⁷

Pedagogy

It should be noted that the goal is to better educate all students, not only to elevate the learning of the bottom half, or entice more of the top half to pursue scientific and technological careers. Deeper, more conceptual knowledge and exposure to modern techniques and thinking will enhance the learning of all students.

Technology is the reason both for the urgency in opening the discussion and for its role in fostering cross-disciplinary scientific research, and is perforce an integral part of the conversation. The standard science curriculum organization is based on a deductive (science in its final stage or ready-made-science)³⁸ paradigm, and approaches scientific literacy mostly as knowledge of contemporary scientific theories and explanations, to be provided topic by topic. Current constructivist, hands-on pedagogies that attempt to impact the curriculum are based instead on a science-in-the-making approach. But inquiry practices that redefine scientific literacy remain at the margins of science education curriculum expectations, do not by themselves provide a rationale for setting priorities, and have not entered into the standard ways of assessing student learning.

The shift in educational research from theories of “learning” to theories of “cognition” has lead to greater emphasis placed on developing thinking and problem-solving skills (inquiry) as practiced by experts in various domain fields. Rather than focusing on teaching and learning based upon disciplinary taxonomies, “constructivist education” is often criticized (and misunderstood) on the basis of the lack of proper prior and concurrent mathematical and scientific training by learners to use it as an entry point into deeper modes of thinking. Content-based, skills-based, and inquiry-based views of learning science are likely to all be needed at some point or another: science as a body of accepted, validated content knowledge; science as a set of problem-solving skills; and science as a way of understanding phenomena.³⁹ But we need to understand when to use each and how the balance among them, or their sequence, depends on student goals.

We have not yet thought hard enough about what is a proper balance between prior known facts and inquiry skills for different audiences and at different points in their education progression. Constructivist-based research does not provide by itself strategies for choosing what content coverage to emphasize as critical, or provide by itself the materials needed to support student’s formal and/or self-directed content learning on top of their constructivist experiences. The need to understand how to best integrate pedagogies is a very important issue and a problem that science education research must address as it integrates cognitive and systemic studies. It is also an area where the deep knowledge of science researchers is needed, particularly in refining and advancing science content standards.

A pedagogy that incorporates multiple visual representations and allows novices to start from a concrete representation of phenomena that can be closely correlated with observable reality promises to empower a larger group of students, since not everybody learns best through abstract, symbolic, or mathematically-oriented expressions. We also now understand much better the importance of motivation for audiences traditionally not well served by STEM education. Changing demographics and the dearth of good students in advanced STEM areas make this a necessity. We should update the knowledge and skills goals for graduates, and on this basis, how to include the needed symbolic, abstract, organized professional ways of thinking that characterize scientific thought. But we cannot contemplate appropriate mixtures of pedagogies without a better-informed definition of goals. Potentially, this also addresses the economic issue of shortfalls in skilled technical personnel—not scientists and engineers, but technicians, programmers and systems maintenance personnel—an important issue for society but also an important mobility mechanism for underrepresented groups.⁴⁰

A new conceptual structure, responsive to the new ways of doing and using science, is needed if a coherent science curriculum vision can be both argued for, refined, and discussed with the many stakeholders in science education and in education in general. There are as of now few if any general frameworks that conceptualize a coherent curriculum that incorporates the content changes that information technology can bring to a science education research agenda—though examples of individual activities abound.⁴¹

There are good reasons for choosing a few topics to be explored in depth as a prelude to a conversation on a more generative science education agenda. Without a coherent framework it is difficult to propose or assess principled decisions on content coverage. It seems likely that the perceived fragmentation of student knowledge reflects in part a no longer tenable fragmentation of conceptions of science and mathematics knowledge. Reality, after all, is interdisciplinary.

There are, in my view, two useful organizing ideas for contemplating the task at hand. One is the role of mathematical modeling and experimentation in contemporary science,⁴²,⁴³ and the other is the complexity (i.e., the systemic, self-inferential, multilevel nature) of the scientific issues that confront citizens in making sense of political discussions and reaching personal decisions. Global climate change and risk assessment of genetically engineered foodstuffs are examples. Interdisciplinarity or ideas behind complexity can be used as a lens to analyze ways in which the science and mathematics curricula relate to science education goals.

Although modeling and complexity may appear superficially very different from each other, they are in fact closely related. Complexity describes a phenomenon while modeling is a methodology for exploring the phenomenon without reducing it to independent, separable parts. Modeling is more than a technique; it is a way of thinking that permits the integrated exploration of complex conceptual spaces.⁴⁴

Physical classroom experiments by themselves are seldom able to empower students to understand most phenomena, complex or not; in practice, laboratory experiments leave students bewildered. As currently practiced, laboratories seldom engender an inquiry perspective to develop an understanding of even simple phenomena. Laboratories do not correlate the macroscopic nature of the observation with the microscopic nature of the explanation provided, while such bridging is required. Mathematical experimentation, when properly used, bridges the microscopic model and the macroscopic observation and can lead students to test their hypotheses in physical systems. The possible simultaneous use of both mathematical and physical experimentation to correlate theoretical concepts at a microscopic level with macroscopic observations could help to reverse the neglect into which laboratories have fallen. An obvious way to do this is the use of microscopic-based simulations to predict observable macroscopic behavior.

Mathematical experimentation can help make abstract, microscopic concepts concrete and manipulable, and thus empower more students to think about these concepts. It can help students better understand mathematics as a powerful, dynamic tool, particularly students whose preferred mode of learning is not analytical or verbal (e.g., chemists and architects, for example, are often visual, three-dimensional, dynamic thinkers).

The move towards seeing mathematics as an experimental science has been deeply coupled with the important role of difference mathematics in the more experimental sciences. The specific role that constructing models plays in mathematics is implicit in the goals and expectations for the use of modeling proposed for the K through12 curriculum.⁴⁵ This “building mathematical models” view of modeling overlaps only to some degree with the specific role that mathematical modeling (simulation or mathematical experimentation) plays in the physical sciences (physics, chemistry, biology), where its uses correspond more closely to “what if” thought experiments using models that encompass what we know about the phenomena.⁴⁶ As DelRe writes, “Let us admit that models are the tools of scientific thinking: physical models are tools of descriptive analogical thinking, mathematical models at large are the tools of argumentative analogical thinking.”⁴⁷

Making models of systems has always been key for understanding their behavior. All models are imperfect, but even imperfect models are very useful in understanding the world. The imperfections in models are of critical importance in guiding future work, because they illuminate where next explorations and enhancements should be aimed. By inputting assumptions into a model, students can subject their assumptions to a process of testing—the results from the modeling can either falsify or validate the assumptions. By going through this process, students can learn the difference between assertion, argument, understanding, and other modes of thinking and discourse.

We acquire knowledge incrementally, building on what we already know and believe to be true. The conceptual distance between a student’s prior knowledge and beliefs, and the new and often counterintuitive scientific knowledge that he or she is expected to acquire in science education could become smaller with a pedagogy that incorporates exploring the mathematical, existing, validated, scientific models of a complex system. Such pedagogy can focus on conceptual advances, rather than on procedural manipulations, and can easily lead to the development by students of testable conjectures.

The country’s science literacy needs of next century—which include understanding of acceptable though partial explanations of detailed, complex phenomena—are often based on microscopic considerations, a scale where intuitions about forces may be invalid, and thus limit students’ capacity to build models. We need to know how to integrate current approaches to helping students understand and construct (mathematical) models while using existing (scientific) models as repositories of what the scientific community knows about a given problem.

A different line of argumentation arising from cognitive science, artificial intelligence, and engineering converges on the importance of qualitative (or semiquantitative) scientific thinking.⁴⁸ Qualitative reasoning (reasoning about signs and magnitudes) has been used to give computers a more natural way of expressing how quantities on the same scale relate to each other, and to make inferences from relatively limited information. Such thinking follows a well-defined qualitative calculus.⁴⁹ Like Moliere’s Monsieur Jourdain,⁵⁰ qualitative calculus has been used in education, not always explicitly.⁵¹ We often forget that estimation and qualitative reasoning should be integral part of quantitative literacy, and explicit theoretical recognition⁵² is needed to eliminate the artificial opposition of visual learning modes (i.e., often qualitative) and more rigorous representations (quantitative).

Bredeweg and Winkels⁵³ categorize the different qualitative ways in which humans interact with physical systems as follows: (1) controlling and operating, (2) designing and constructing, and (3) diagnosing and repairing. This taxonomy is useful for categorizing scientific literacy, since many different groups are often called upon to perform the three functions.

Modeling as an integrative scientific process, with its accompanying simulations and visualizations, can be part and parcel of the process of diagnosing and repairing physical systems, as much as it is part, for specialists, of the process of designing them. The excitement caused by robotics and similar design-oriented competitions attest to their usefulness. Such design and testing processes lead to learning science at high cognitive levels by supporting students in working with many different representations and learning modalities—verbal, analytical, visual, concrete.⁵⁴

Results from this and other research⁵⁵ suggest that the empowering effect of understanding what it is to understand, facilitated by technology, stays with the student and transfers to other school activities. In a similar manner, it would be interesting to see what approximations students make on their own in going from a complex simulation to a constructed conceptual understanding of the problem, and compare these approximations with those embedded in the historical scientific progression. Complex systems and modeling approaches may provide another systematic means by which to answer the frequent “less is more” calls to streamline yet deepen school curricula. Such approaches often appear as calls for a curriculum based on conceptual strands (form and function, conservation of matter and energy, energy exchanges, and so on)⁵⁶ which are well suited to exploratory approaches in systems where complexity appears as stabilizing feedback loops. Hyphenated sciences and cross-disciplinary phenomena in fact often arise from consideration of feedback loops within systems that cannot be properly called closed. Despite the importance of open systems in real phenomena, they tend to disappear in the compartmentalization of disciplines, which arises from a separation of the whole into parts assumed non-interacting. Integrative trends may provide an opportunity for developing coherent educational experiences with diverse problems within a common framework, helping students build the conceptual tools to understand their ever more complex and rapidly changing world.

Page 7

Instructional Workforce

Extracted from Hestenes “Wherefore a science of teaching?”⁵⁷

Teaching, I say, is an art, and not a science . . in no sense can teaching be said to be a science. These words, written by F. K. Richtmyer in 1931 were recently reiterated in this journal by R. A. Goodwin. Professor Goodwin seems to think that all the great truths about teaching are already known, so that recent attempts to improve teaching techniques can hardly be more than transitory “fads.” I am sorry to see someone who is concerned with the quality of teaching take such a divisive stance. Perhaps a reply will help some readers develop a more constructive point of view.

I will argue that an ample foundation for a science of teaching exists already today, but that the “science” remains in a primitive state primarily because it has not been fostered and cultivated by those in a position to do it, namely, the university professors.

Art or science?

Let us agree at the outset that good teaching is an art, fully deserving our respect and admiration. It does not follow, however, as Goodwin seems to think, that there cannot also be a science of teaching. Who will not agree that there is an art of experimental physics and an art of mathematical thinking? Nobody, let us hope, confuses the art of doing science with the body of knowledge which it produces. Nor should anyone confuse the teaching skills acquired by individuals with an objective body of knowledge about teaching. Medical practice is widely acknowledged to be an art, but who doubts the possibility of medical science? Is teaching so different because it ministers to the mind?

Given what we know and what we want, can we work with the existing instructional workforce? For me, than answer is yes, for the most part, provided we do not ignore the role of other parts of the educational system in supporting or opposing our goals, and that we understand at least something of the science behind teaching. We owe teachers at the trenches the tools for building their own capacity and knowledge⁵⁸ not as a haphazard mix of fads and common sense intuition, but as a foundation that can be built upon and refined.

Because of the system, the question should not be “can we?” but ‘how can we?” Citing Drucker again, implementing an innovative strategy (in business) requires a different way of framing the problem. He warns that one must reject input-driven questions such as “is this effort necessary?” or “what is the minimal level of support that is needed?” Instead, we should ask performance or output-driven questions such as “what is the maximum of good people and key resources which can be put to work at this stage?” And “what does it take to achieve the goals?” Priorities are the critical factors in any decision, not money and time. For these priorities to be acceptable to schools, there must be an assurance that they will have to refine their plans, but they will not need to replace them with a totally different plan with its attending disruptions.

If we accept this position, there is much that we know about how to work effectively and efficiently with teachers in classrooms. Efficiency here is not of cost, but of time and ideas. Bob Tinker is fond of saying that bad education can be very cheap.

Success implies longer-term commitments for support.^59,60 Primarily, outcome measures must go beyond end of year tests; retention of ideas and ability to apply them develop not course by course, but across courses. Looking at outcome measures that take longer to become visible, such as increased graduation, attendance rates, enrollment in advanced placement or other high level classes, college or graduate study enrollments, even summer jobs, will help parents and schools grant teachers the space to change their methods and become expert at them. In the private sector, measures such as market share, customer loyalty, market positioning, and niche marketing, not just profits, are a consideration for planning.

Most certainly, we know that teachers that undertake changes in pedagogy go through the same “productivity dip” that we all experience when we start on a new topic or a new methodology. If we ignore the time this takes, almost anything besides doing the usual will be deemed a failure. Becker notes:

It takes time for teachers to master computer-based practices and approaches. The Sheingold and Hadley survey shows at least five to six years. Teachers who have had students use computer software in a substantial way for several years are the same teachers who are most apt to report that their teaching practice has changed substantially.⁶¹

We should keep in mind, and continuously remind the public and policymakers that, when support is sustained over time outcomes can be more significant⁶². A particularly significant result from Center for Children and Technology (CCT)⁶³ indicates that teachers that use technology to empower students to learn independently report more time spent with the students that need help most, contributing to the desired classroom gains, and indicating a more efficient, professional use of their time. And all this with the technology of the 1990s!

An area where policy promises more than it delivers is use of student data for targeting instruction to individual student performance. Means and Olson⁶⁴ found:

Even though nearly half of all teachers (48 percent) reported having access to student data systems, they did not necessarily have the information or tools they needed to make use of the student data available to them. Less than 40 percent of teachers with access to a student data system reported having access to standardized test scores from the current year for their students.

Teacher enhancement is an area where we know much, given that its importance is recognized and that significant resources go into providing such opportunities, though very few look at technology’s role in it. But though credible examples of the effective use of technology to support teacher peer learning in “communities of practice” and in receiving professional content support exist and are ready for scaling up, most teacher enhancement activities use traditional methods that have been discredited.⁶⁵ As with standardized assessments, the cost in time and money of effective teacher enhancement is a barrier to their use.

Final Words

None of the ideas and implied proposals in this paper is new⁶⁶. In addition to providing a digest of lessons learned, the paper’s contribution, if any, is to present a coherent view of many aspects of the complex fields of learning and education. By the nature of the funding and of the education research enterprise as constituted, it appears to be a series of loosely related topics, all of them needed to achieve more than a passing solution. The intention in presenting what we know in each of them is to enable an emergent view of next steps that is more than the sum of its parts. If all of these rules were to be attempted together, this is a tall order. But the goal is achievable if we work purposefully in a scientific experimental mode—iterating thought, action, analysis, and reaction. The main point I wish to make is that these experiments MUST involve attention to the whole—at least to document and understand the whole in such a way that the knowledge obtained can be analyzed coherently.

The influential book “Pasteur’s Quadrant”⁶⁷ introduces three models of research: Bohr’s (emphasis on pure science, the NSF model), Edison (emphasis on applications, the NIST model) and Pasteur (joint emphasis on pure science and on its applications, the NIH model). The fourth quadrant is left unassigned. Perhaps we should call it the Linnaeus quadrant (gathering data where taxonomy does not exist), emphasizing the importance of data to theorize a new area. We must find our way to collective data that could lead to collective wisdom. Researchers know well how to devise a doable study within a larger problem, such that the study does not violate the system requirements and that its results are valid. They know when generalization is possible, and when more data is needed. They also know that they are part of a community and both look back to what came before their work and forward to what will come after their work.

Researchers read and publish. They keep a lab book for notes on the process to be replicated. Those that will want to follow in a researcher’s footsteps will need to understand why the work was designed the way it was, and what could be done differently with the benefit of hindsight. Any work undertaken should be considered “educative” for those doing it and for those that will want to replicate it. What needs to be shared is the equivalent of science-in-the-making. Methodology and strategy, the ups and downs of implementation, are what empower others to make the best possible use of the work.

In perusing anew the literature for preparing this position paper, I was struck again by the nature of what is available: what has been called “supply research” that reports studies or R&D, mostly in isolation. Choosing from this literature requires both time and relevant expertise or, conversely, hunches based on prior beliefs. There is not a “demand side” research literature to provide the first step toward helping teachers and districts to plan for their own reform, with the consequence that vendors and other interested parties are the source of information, much like pharmaceutical representatives provide advice and samples to physicians. One useful action that can be contemplated is the creation of impartial consulting groups, with an appropriate mix of experiences from research to superintendency that could visit, analyze, and inform local STEM education planning along the likes of Lauren Resnick’s Institute for Learning⁶⁸. I end with a quote from the “taking stock” meeting on research in urban systems, already seven years old (Ref. 27):

The most fundamental message of the meeting and of this report is that researchers have produced systemic knowledge that is both warranted by data and useful for guiding the continuous improvement of urban schools under a range of conditions. In all the cases discussed, trust and sustainability of research collaborations were based on thoughtful, flexible, and mutually respectful partnerships with teachers, schools, school districts, and communities. This collaborative-systemic research paradigm can provide a framework for integrating the results of studies that examine the individual components of urban educational systems separately. It demonstrates the possibility of continuous feedback between educational research and practice, making it unnecessary to postpone practical improvements until the completion of long research cycles. Finally, it provides an example of “research in the public interest” at its best: research rooted in social and community values that offer guidance for building up the very basis of democratic community—an educated citizenry.

Page 8

APPENDIX A
RECOMMENDATIONS FOR ACTION

The recommendations suggested distinguish between direct and indirect action. By direct action I mean direct intervention in schools—specifying content, process, and a theory of action to be subjected to implementation research. By indirect action I mean supporting the design and study of experiments that probe aspects of the infrastructure, for example by defining an educational parallel to the agricultural extension service as a testbed for analysis. This type of “for the good of the order” action, calling on the appropriate collaborators from practice, research, and policy, may not result in direct student learning in the short term, but will lead to the creation of a research community that does not currently exist, to significant advances in the knowledge about policies for sustainable change, and hopefully to increased STEM learning in an increasing number of schools.

Indirect actions that the Commission can undertake will influence the context for education reform; these recommendations for action promise to influence policy at many levels, from the public to Federal agencies and legislative action. Indirect action has broader systemic goals that can strongly influence how resources are allocated⁶⁹ and that direct action projects often can’t address.

One step to consider for each case is creating a working group of visionary stakeholders to formulate an action plan and perhaps act as an eventual Steering Committee. Please note that partial steps toward these actions may exist and can be built upon, but because they are not coordinated it takes too long to learn from them and transfer their lessons. The objective of these recommendations is to hasten action by building on what is known.

Possible Indirect and Systemic Actions

• Define a feasible educational parallel to the agricultural extension service as a testbed for implementation, study, and analysis.
• Create dynamic agent-based models of education systems or subsystem prototypes (for example, a series of Sim-Schools or Sim-Districts) that can be run and tested against existing data from reform cases, and use the models to train practitioners in understanding where systems constraints impede progress.
• Create tools to support the study of implementation around specific cases, such as models to be used by practitioners to study the process of implementation.
• Select and study some successful long-term interactions of intermediaries with schools or districts—intermediaries could be museums, university groups including but not limited to colleges of education, non-profit research centers, etc. that bring flexibility and experimentation to the system.
• Embed researchers, journalists, scientists, or policymakers in some projects to involve more directly the public at large in understanding the efforts needed to carry out sustainable reforms and the need for “clinical research” in education that parallels what transpires in medicine, law, and engineering.

Possible Direct Actions

• Create school district “pedagogical” laboratories for teacher enhancement around the use of challenging STEM content. Have expert teachers teach students after hours (or use videotapes) with other experts (e.g., researchers) present, and have other teachers discuss the pedagogy.
• Choose a system with capacity to absorb a particular challenging STEM innovation and the infrastructure to maintain it, and help the system work with a school that has already implemented the innovation to study the process of adaptation to local conditions.
• Help schools develop a new culture by integrating technology throughout—implementing across courses, analyzing assessments, developing joint interdisciplinary learning goals, discussing video journals of good practice (or of their own practice), streamlining communication with parents, etc.

In all of these projects, the following consideration will apply:

• Help teachers be diagnosticians focused on what each student knows and needs to learn, and use documents such as the AAAS Atlas of Scientific Literacy to foster thinking about the evolution of and interrelation of concepts.
• Help teachers use tools to analyze their own student data.
• Openly analyze the reasons behind failures.
• Build models of the system and the intervention to conduct what-if experiments on how the system works under different assumptions.
• Pedagogy makes a crucial difference in what students learn, and focusing on pedagogy implies changing teachers’ ingrained classroom behavior and beliefs.
• Consider science as a distinctive, unique discipline in terms of its interplay among theory, models, and observations. Look for the overlaps between different disciplinary standards, which are many.
• Use a pedagogy that incorporates multiple visual representations, allows novices to start from a concrete representation of phenomena correlated with observable reality, and is more generally accessible to non-science oriented students.
• In many cases students have verbal and mathematical deficiencies that stand in the way of learning science. Language and reading teachers should be involved.

APPENDIX B
EXEMPLARS OF TECHNOLOGY USED FOR SCALING UP SCIENCE EDUCATION

I will refer here to a few examples of what I consider exemplary integration of conceptual content goals, good software, and appropriate pedagogy that can be scaled up. These examples reflect my personal biases—programs and projects that I know well—and point to ways in which education research, technology, and cognition have been integrated and used well. The programs chosen reflect significant types of technology use, but no attempt was made to make this a complete list of what is exemplary in the field.

Each case listed has received many years of research and implementation funding. An older group has been validated by specific impact criteria in more than one implementation; and the criteria used are of importance by themselves for choosing promising projects. The more recent group does not have the same level of external validation, though its positive impact on learning has been well documented. This means that often the projects listed work with volunteer teachers, schools, and districts.

Three of the four exemplars come from the 2000 U.S. Department of Education publication Exemplary and Promising Educational Technology Programs, selected by a panel of research and policy experts following specific criteria developed by the group⁷⁰. The report analyzes why individual projects or software packages by themselves are not sufficient to indicate if implementation succeeds or fails, and thus lists programs of multiple implementations that have proved successful in practice and, more important, the criteria and associated rubrics used to select them. The report is available at http://www.ed.gov/pubs/edtechprograms/index.html.

I copy here the criteria for quality from the USDoE Report, but the reader should go to the publication itself for more details on the criteria and the rubrics used.

The Expert Panel developed evaluation criteria and program indicators based on four categories of criteria:

1. Quality of program
2. Educational significance
3. Evidence of effectiveness
4. Usefulness to others
 

The evaluation criteria were carefully field-tested and resulted in the final review criteria.

An exemplary program:

• Addresses significant educational issues and identifies goals and a design supported by research;
• Improves preK-12 learning;
• Contributes to educational excellence for all;
• Promotes organizational change;
• Makes possible educational gains; and
• Serves as a model for other educational institutions because it is sustainable, adaptable, and scalable.

Page 9

Two of the programs are for high school science, the other two are for middle school (one science, the other mathematics). These four programs reinforce the fact that we know what to do; we do not know how to apply it in a large enough scale to “move the center of gravity of STEM education.” The three older programs are⁷¹:

1. Maryland Virtual High School—Integrating Technology and Teacher Professionalism with Science Education Reform. http://mvhs1.mbhs.edu/mvhsproj/project2.html and http://www.ed.gov/pubs/edtechprograms/virtualhighschool.html.

2. Modeling Instruction in High School Physics. Hestenes, D. (1987). Toward a Modeling Theory of Physics Instruction, Instruction, American Am. J. Phys. 55, 440- 454 (1987); Wells, M., Hestenes, D., and Swackhamer, G. (1995). A Modeling Method for High School Physics. Journal of Physics, 63. http://www.ed.gov/pubs/edtechprograms/modelinginstruction.html.

3. One Sky, Many Voices. http://www.oercommons.org/courses/one-sky-many-voices http://www.ed.gov/pubs/edtechprograms/manyvoices.html.

A fourth program whose external validation and scaling are the focus of current studies is:

4. SimCalc—The Mathematics of Change. SimCalc, Technical Report 02 December 2007. Jeremy Roschelle, Deborah Tatar, Nicole Shechtman, Stephen Hegedus, Bill Hopkins, Jennifer Knudsen, Margie Dunn. Also http://ctl.sri.com/projects/displayProject.jsp?Nick=simcalc; http://math.sri.com/research/index.html; http://math.sri.com/curriculum/index.html

One-page descriptions of each of these programs appear below, followed by shorter decriptions of five other exciting and successful projects.

Maryland Virtual High School—Integrating Technology and Teacher Professionalism with Science Education Reform.

The Maryland Virtual High School (MVHS) CoreModels Project has two primary goals:

1. To use computer modeling to help all high school students achieve state and national science standards

2. By developing and refining a process of network-based peer leadership and collaboration, to give teachers in their classrooms the support they need to integrate modeling activities into their instruction.

Started in 1997, CoreModels is based on, and is intended to institutionalize, the gains achieved by the Maryland Virtual High School program that connects rural and underserved Maryland schools to a magnet high school via the Internet. Through this connection, MVHS supports teachers in developing and implementing computational science projects with their students.

Crucial components of the MVHS program are:

1. Its focus on using computational modeling to teach complex science content; and
p(. 2. Its mentoring and support for remote teachers and students provided by magnet teachers and their students. The mentors won national recognition for their computational science projects and gained experience through a student-run laboratory at Montgomery Blair High School Magnet Program.

A model of distributed support creates the conditions for sustainability of the changes and gains achieved by schools. The program⁷² was active in 6 to 15 schools in each of three CoreModels regional centers in Maryland. Students construct their own models utilizing the software STELLA in areas of science such as wildlife populations, the carbon cycle, hurricane prediction, projectile motion, chemical reactions, and rock formation cycles. Students hypothesize about the results of their model based on the parameters used in the graphical model definition. They compare their results with other predictions, such as those from scientists or developed by other students, or with actual results as measured in practice, thus gaining an understanding of important recurring scientific concepts involving equilibrium processes, feedback, and multiple causal relationships, among others.

MVHS CoreModels increases students’ in-depth understanding and competence in their current science subject area by focusing on an analysis of change over time and on the transfer of understanding from one activity to the next. One of the hallmarks of the project is the alignment of CoreModels activities with Maryland Science Core Learning Goals (CLGs). CoreModels materials also focus on the American Association for the Advancement of Science (AAAS) Benchmark common themes (similar to National Science Education Standards themes), which emphasize connections between seemingly disparate science content.

Modeling Instruction in High School Physics.

Modeling Instruction in High School Physics, started in 1990, uses computers to teach models and modeling, central components of modern science. These components are focal points to develop the content and pedagogical knowledge of physics teachers, who then serve as local experts on the use of technology in teaching and learning science.

Science, and physics in particular, is a content area for which students need to learn how to use computers as a scientific tool for observation, data acquisition, analysis, and problem solving. Teachers are trained to support technology-based learning in up to eight weeks of intensive Modeling Workshops conducted over two summers, and with ongoing year-round electronic network support. Teachers are thus engaged in a complete revamping of high school physics to incorporate both technology and the insights of educational research in full accord with the National Science Education Standards. The training provides them with a robust new teaching methodology that greatly increases students’ understanding of basic physics.

In the Modeling Workshops, participants are introduced to modeling as a systematic approach to the design of curriculum and instruction. Teachers identify a small number of models around which to base their physics course and learn strategies to help students develop those models. They collaborate on the redesign of the high school physics course to enhance learning and employ technology to achieve their goals. They learn how to use computers as an integral part of their teaching practice. They implement a student-centered instructional strategy that engages students in active scientific inquiry, discourse, and evaluation of evidence. Further, they examine the implications of educational research for physics teaching. They do all this while immersed in studying the content of the entire year, which also provides extensive remediation for under-prepared teachers. Teachers learn to:

• Organize course content around a small set of basic models as the core of physics;
• Engage students collaboratively in making and using models to describe, explain, and predict;
• Design and control physical phenomena;
• Involve students in using computers as tools for collecting, organizing, and analyzing,
• Visualize and model real data;
• Provide students with basic conceptual tools for modeling physical objects and processes, especially mathematical, graphical, and diagrammatic representations;
• Show how scientific knowledge is validated by engaging students in evaluating scientific models through comparison with empirical data;
• Assess student understanding in more meaningful ways and experiment with more authentic means of assessment; and
• Improve continuously and update instruction with new software, curriculum materials, and insights from educational research; and work collaboratively in action research teams to mutually improve their teaching practice. Students learn to understand scientific claims and to make sense of their experiments. They must articulate coherent opinions; defend their findings in a variety of formats using graphs and/or diagrams, and through algebraic expressions of the relationship.

One Sky, Many Voices⁷³. One Sky, Many Voices (OSMV) is a research-focused learning program based on the student’s application of knowledge to solve science problems using real-time weather data. Visualization software allows students to track real-life events such as hurricanes, blizzards, floods, and tornadoes. The program, started in 1992, covers 4 or 8 weeks of coordinated study in middle school science, teacher support in the form of local study groups and focused networked discussions by experienced teachers, daily scientists’ updates, and a suite of state-of-the art technological tools including current, customizable weather imagery and message board systems. OSMV has been implemented, with varying levels of support, with a large number of students in impoverished urban schools, as well as suburban and isolated rural schools. A science- and data-based interaction among students from these diverse populations is a strength of the program.

OSMV promotes important attributes associated with significant learning experiences. Middle school students use an Internet browser for enhancing their investigations of current weather phenomena. Guided by their teachers and linked with experts in the atmospheric and environmental sciences, students view real-time images and converse with their peers on Web-based message boards as an integral part of their studies. Thus, students might track and predict current hurricanes, collaboratively study and discuss current weather fronts in their region, or develop content-rich explanations of weather phenomena in their area to be shared with students who do not live with the same extreme weather patterns.

OSMV engages and empowers students as scientists and provides support for teachers using the associated technology and pedagogy. Students take an active role in the teaching and learning experience. The exchange of roles between students and teachers in this program reflects a sound, progressive school reform effort.

Students increase the abilities necessary to conduct and understand scientific inquiry as well as to make sense of their predictions themselves. They hone their communication skills while conducting online interactive discussions with content mentors and peers distributed nationwide on their focus topics. Peer explanations and predictions are critiqued and discussed in a group forum. After participation in OSMV, students display knowledge of significantly more weather terms, make more scientifically valid claims, and engage in more sophisticated measures of scientific thinking.

Teachers learn to:

• Provide students with basic conceptual tools for understanding the use of modeling processes, especially in visual, mathematical, and graphical, representations;
• Assess student understanding in more meaningful ways and experiment with more authentic means of assessment;
• Improve continuously and update instruction with new software, curriculum materials, and insights from educational research; and
• Work collaboratively in action research teams to mutually improve their teaching.

SimCalc—The Mathematics of Change⁷⁴ The SimCalc project seeks to democratize access to the mathematics of change, making concepts of proportionality, linearity, and rates of change accessible to ordinary middle school students. The project teaches core concepts in the strand that leads to Algebra and eventually to Calculus. To accomplish this, SimCalc develops and studies restructured curriculum and innovative graphing technologies, seeking an integration that supports students in developing a robust, integrated, multi-faceted understanding of the concept of “rate of change.” See SimCalc Tech Report #2: Extending the SimCalc Approach to Grade 8 Mathematics.

The project studies the viability of research-based, technologically enhanced mathematical pedagogy as a large-scale innovation across a range of teachers, students, and educational environments. Emphasis is on the challenge of understanding whether a wide variety of teachers can succeed with materials that integrate technology, curriculum, and teacher professional development.

The current scale-up studies are based on the student and teacher learning demonstrated in prior research. The starting point is the investigation of the robustness of a replacement unit model for teaching core concepts in complex and conceptually difficult mathematics, specifically rate and proportionality. To do this we are developing techniques for improving and measuring teachers’ content, pedagogical content, and technology integration knowledge.

The project in 2006 completed a randomized, controlled experiment with over 100 7th grade teachers in Texas over 2 years. SimCalc also expanded into the 8th grade and designed another replacement unit focusing on linear function. In both years, outcome data is collected to look at: (1) strong evidence on which educational leaders can base decisions about using technology to address the national challenge of improving students’ ability to do complex and conceptually difficult mathematics, and (2) new insights into specific teaching and professional development practices that leverage technologies to improve learning.

The current work is based on prior projects: Phase I was a pilot of a randomized, controlled experiment with 7th grade teachers in Texas that found significant learning gains for teachers who received training in our intervention (compared to those who did not). Furthermore, we found significant learning gains for their students.

A related project, NetCalc explored the potential of wireless handheld devices in implementing the SimCalc vision in the context of an 8th grade month-long introduction to concepts such as the meaning of the area under a curve. The research examined learning gains, patterns of attention and engagement, and the new learning opportunities possible within classroom activities supported by handheld devices. Students increased their proficiency in the mathematics of change and variation during the NetCalc curriculum. Furthermore, the NetCalc eighth-grade students performed better on AP Calculus items than high school students taking the AP exam, according to published test results⁷⁵.

Additional software that exemplifies other types of technology uses in the classroom that are in wide use and thus make them attractive candidates for scaling up

Page 10

Artemis Middle Years Digital Library Bos, N., Drabenstott, K., Krajcik, J., Soloway, E., Talley, M., Woolridge, S., & Miller, J. (2000). Students’ Searching and Evaluating with the Artemis Interface to a Digital Library. In B. Fishman & S. O’Connor-Divelbiss (Eds.), Fourth International Conference of the Learning Sciences Mahwah, NJ: Erlbaum (www.umich.edu/~icls/proceedings/pdf/Bos.pdf) See http://www.hi-ce.org/artemis/index.html Artemis™ is an Internet based research tool (enhanced digital library) based on using a Driving Question to guide inquiry learning in science. Rather than focus upon simply finding answers to simple questions, Artemis supports inquiry-based learning by helping students create a meaningful research question around which they can learn, explore and discover. Accessed via subscription, Artemis provides teachers and students with a continuously evolving collection of resources and links to relevant sites on the web that have been selected by librarians for their appropriateness of content and reading level. Students search and find optimally sized sets of vetted material that often include contrasting information. Students receive hints about how to read the materials. Artemis helps students organize and save their searches in “driving question” folders stored on personally dedicated server space, to keep track of their research questions, view past search results, and share their results with each other. Teachers are able to monitor the entire system in real time. The process helps students focus their energies and develops competence in seeking, evaluating, and using information. Artemis features include the following:

• There is registry of over 5000 educational web sites, hand-picked by teachers and librarians, updated regularly, and aligned with national and state standards.
• Teachers can monitor and assist their students’ online activity.
• Students can:
  • define –and refine– a driving question to guide their search;
  • save good sites and research notes in a permanent storage folder; and
  • share relevant information and web sites with other students and classrooms.
• A classroom-tested and scientifically validated set of curriculum materials help teachers and students integrate standards-based content with Artemis technology.

Artemis has received many distinctions, including being voted “by far the most popular digital library choice” by eSchool News online, and being selected as a NetDay Compass Spotlight Site for offering students “an excellent online tool for project-based research.”

The GEODE Initiative: The SSciVEE, Worldwatcher and Global Warming Projects.  Edelson, D. C. (2007). Environmental Science for All Considering Environmental Science for Inclusion in the High School Core Curriculum. Science Educator, 17(1), 42- 56. http://www.covis.northwestern.edu/sciviz/sciviz.html, http://www.covis.northwestern.edu/sciviz/sciviz.html, http://www.letus.northwestern.edu/projects/gw/

The GEODE Initiative is dedicated to the improvement of Earth and environmental science education through the use of data visualization and analysis tools to support inquiry-based pedagogy. Data visualization and analysis technologies have transformed the practice of science in recent years by capitalizing on the power of the human visual perception system to identify patterns in complex data. The GEODE Initiative explores the potential of this technology to improve science education in similarly dramatic ways. Our research is exploring the hypothesis that scientific visualization, incorporated into inquiry-based learning, can enable students of diverse abilities to develop an understanding of complex phenomena in the Earth and environmental sciences. Our challenge is to identify the specific software and external supports that are necessary to transform data visualization and analysis into an effective educational technology.

The transformation of tools and techniques developed for scientists into environments to support students is a significant challenge. Understanding the requirements of such supportive visualization and data analysis environments for education is an important goal of this research. We are developing and evaluating geographic data visualization and analysis environments for the study of a diverse range of topics in the Earth and environmental sciences. These supportive environments enable learners to examine data sets created by the scientific community and to create their own data. They allow students to view geographic data in the form of interactive maps at a variety of spatial and temporal resolutions.

WISE Design for Knowledge Integration Scaffolds for scientific reasoning in an inquiry classroom. Linn, Marcia C.; Clark, Douglas; Slotta, James D. (2003). Science Education, v87 n4 p517-38 Jul 2003. http://wise.berkeley.edu/

Web-based Integrated Science Environment (WISE) started in 1997. WISE capitalizes on the synergies between Internet connectivity and integrated science to advance our understanding of inquiry science instruction. The WISE learning environment incorporates recent cognitive and social research and gives new authoring partnerships a head start on effective inquiry designs and includes the following :

• Projects that meet design criteria and can be used by teachers everywhere
• The WISE mentored professional development model that guides teachers to implement inquiry practices
• WISE server technology that collects and stores both student work and teacher activities such as lesson plans, contributions to discussion, and reflection notes.
• WISE Knowledge Integration Assessments that measure progress in inquiry and can be used to detect transfer from the topic where inquiry was taught to new topics.
• WISE science, technology, and language literacy measures, and connections to standards-based measures of inquiry from NAEP, TIMSS, and state testing programs to establish broader impacts.

WISE research looks at how these elements of inquiry instruction jointly contribute to sustained improvement of science learning to address four research themes:
(a) What is the longitudinal impact on students of one or more WISE projects?
(b) What are typical trajectories of teachers from diverse schools (urban, rural, and suburban) who adopt inquiry practices such as WISE?
(c) What value do new technologies for visualization and modeling, including our new grant of Palm technologies, bring to inquiry instruction with WISE?
(d) How do Curriculum Design Partnerships progress in response to the design review and classroom trials of their curriculum projects?

Technology Enhanced Elementary and Middle School Science (TEEMSS). Learning Science in Grades 3-8 Using Probeware and Computers: Findings from the TEEMSS II Project. (2008) Zucker, Andrew A.; Tinker, Robert; Staudt, Carolyn; Mansfield, Amie; Metcalf, Shari. Journal of Science Education and Technology, Volume 17, Issue 1, pp.42- 48 (Feb. 2008) 02/2008. http://teemss.concord.org/

TEEMSS is a long-term initiative at the Concord Consortium to infuse computer-based data collection and analysis across the elementary and middle school science curriculum. The TEEMSS project is a research and demonstration project designed to gather solid data on the educational value of low-cost probeware. Schools could greatly reduce their total costs of implementing probeware by shifting from desktop computers to handhelds and by having students construct their own probes. The questions addressed by the project are whether this is feasible in typical schools and whether educational materials can be developed around this capacity that improve student learning of standards-based concepts.

The project is structured to increase the field’s understanding of how teachers at the middle school level can incorporate electronic technology and whether their students learn mathematics and science concepts more effectively through the use of the technology. Measurements are made using inexpensive, but sensitive, probeware connected to handheld computers. Models and simulations allow students to explore behavior that is difficult to understand by traditional means. Teacher support materials encourage teachers to change practice for a few weeks.

Quest Atlantis. (2007) Our Designs and the Social Agendas They Carry. Sasha Barab, Tyler Dodge, Michael K. Thomas, Craig Jackson, Hakan Tuzun. The Journal of the Learning Sciences 16(2), 263–305. http://atlantis.crlt.indiana.edu/. See other publications listed in http://inkido.indiana.edu/barab/rsrch_qa.html

Quest Atlantis is a learning and teaching project that uses a 3D multi-user environment to immerse children, ages 9 to 12 in educational tasks. Currently over 4,500 registered users from 5 continents use Quest Atlantis in formal school environments as well as in after-school settings. Building on strategies from online role-playing games, Quest Atlantis combines features used in commercial gaming environments with lessons from educational research on learning and motivation. The core elements of QA are: 1) a 3-D multi-user virtual environment; 2) learning Quests and unit plans; 3) a storyline, presented through an introductory video, novel, and comic book, that involves a mythical Council and a set of social commitments; and 4) a globally-distributed community of participants. The narrative helps to establish continuity among the QA elements and helps to bridge the fictional world of Atlantis with the real world of Earth, an act of interpretation by each individual child. Central to this narrative is a group of young activists, the Council, who communicate with participating children and help scaffold their activities. The activities of Quest Atlantis take place in registered Centers, typically schools, under the direction of teachers who have undergone professional development and training. QA includes both curricular and optional projects that unfold both online and away from the computer, as children work alone or together to accomplish tasks within the international QA community.

Central to the team’s work in the Quest Atlantis (QA) project has been designing a context for learning, which sits at the intersection of education, entertainment, and social action. Designed to support social commitment and real-world action, QA is an immersive context with over 20,000 registered members worldwide. The project is intended to engage children ages 9–12 in a form of dramatic play comprising both online and off-line learning activities, with a storyline inspiring a disposition towards social action.

Page 11

Appendix C
Putting What We Know to Use

I will attempt in this section to start the process of developing implementation considerations that emerge from what we know about: (a) achieving sustainable impact for all students; (b) working within the system constraints so as to learn from our work and help the system adapt to change; (c) developing materials and processes that can be adapted by others to fit their situations; (d) content that anticipates the science of this century; (e) pedagogies that can help all students acquire knowledge and skills to become scientifically literate; (f) supporting teachers in undertaking the needed changes; (g) documenting the process of ups and downs and the lessons learned from the work.

These guidelines reflect a vision, partial at this point, of what can be considered steps towards refining what we mean by success. It attempts to hypothesize what were the conditions that allow some interventions to succeed while others do not. Nevertheless, I am conscious that if these ideas are taken only as “exhortations to be good,” they will not be as useful as if they are considered as bases for reflection and refinement. Science is, after all, data-based argumentation.

Interacting with practice

• Have as the goal what is needed for long-term school improvement, even when working with only one teacher, to know how to take advantage of any gains achieved.
• Establish clear goals and realistic timelines jointly with teachers and school administrators.
• Think strategically—should I start with a few classrooms, or with a whole cohort of students? Which one will be better at a particular location? How does information disseminate within that particular location?
• Plan for yearly measures that can give the school a sense of successful continuous improvement, while helping the school keep in mind that more encompassing gains will develop as time goes by. Improvement is an iterative process but higher administrators need to know that progress is being made. The only way to counteract strong external pressures for quick success is to define partial success as an accomplishment.
• Help communicate what it means for a teacher to be a diagnostician focused on what each student knows and needs to learn, and how such knowledge can help both instruction and assessment.
• Observe successful teachers and consider the pedagogical knowledge they display to share with their colleagues (for example how they elicit comments from students, how they focus on each student’s needs, how they help peer-to-peer learning, how they keep a fruitful discussion moving along, etc.).
• Distribute the AAAS Atlas of Scientific Literacy wherever you go to foster thinking about the interrelation of concepts and their interdisciplinarity.
• Set goals for all students, and measure success accordingly. Multiple longer-term indicators of change will be important.
• Look for the reasons behind failures. Analyze the failures with school personnel and use data and your knowledge of the field to propose improvements.
• An innovation is in fact a strategic set of innovations for multiple levels of the system. Therefore, you need to understand the particular system you are working with and its flows of information, not to mention power structures.
• Any innovation brings changes at levels above and below the one at which you intervene and strategies for all the three levels have to be simultaneous and coherent.
• Do not limit your vision to existing resources. Work where there is sufficient flexibility in funds (i.e., the ones you bring and the ones the school has) to succeed.
• The schools most in need of help are also most in need of your innovation and vision, and often most receptive to change.
• Developing trust with school collaborators takes less time that developing it with a school system—on the order of two years. Teachers and principals are used to being guinea pigs for researchers. If more is asked from them, they have to know that support will be available when problems of practice start coming up, and when school boards want answers.
• Make five-year plans when possible. The complexity of human learning and study makes the first year of education research the equivalent of “calibrating the equipment.” In this case it involves not only measurement instruments but also trust (how far can you go) and validity. With the need for at least one iteration—the equivalent of refining experimental measures—there will not be time otherwise for reflection or for gathering the lessons that can lead to continuous improvement.

Interacting with the system

• Policy must be confronted, but the school is the expert, not the outsider who intervenes. Your role is to help and support schools in addressing policy.
• Don’t think of education and it’s funding as independent of each other. Schools tend to isolate sources of funding that come with different objectives, but benefit when they can use the resources coherently. Sustainability implies better use of resources, not always more resources.
• Make the whole process visible and interactive. Parents and school board members should understand the goals, the processes, and the outcomes. Nobody changes unless they see the advantages and feel supported.
• Choose a system that has the capacity to absorb the innovation and the infrastructure to maintain and evolve it.
• Pay attention to adaptive management since all action is local, and the local policy context is constantly changing.
• Help different levels of the system develop a common meaning for the language they use.
• Optimal solutions often are counterintuitive. Be ready to explain why common sense is often counterproductive.
• Consider using other local human resources to build internal capacity in departments, schools, districts (LEAs), and even local Chambers of Commerce.
• Discussions among researchers working with large school systems indicate that in most cases it takes of the order of 5 to 10 years to establish effective collaborations between researchers and school systems, and that during this period there may be a need to renegotiate and recommit to goals and strategies developed together.
• Do not forget informal resources available to children and teachers. Museums and after school programs are part of the system and may offer islands of flexibility used and understood by parents, at least in part.
• Consider the footprints you will leave: human infrastructure, new processes, increased human capacity. A significant percentage of funds should go towards what you will leave in place.

Learning from action

• Build jointly with local partners models of the system and intervention to share “what if” experiments on how the system works under different assumptions.
• Consider what kinds of data would be needed to realistically monitor all objectives, and work towards making the data accessible to those who need it.
• Successful multi-year reform processes include periods of consolidation of gains, and enabling consolidation periods should be part of your plan of action. These periods provide a respite to plan for needed changes and for people to become comfortable with one set of changes before contemplating others.
• Such stepwise strategy promotes buy-in from skeptics, allows for non-disruptive change and establishes a culture of continuous improvement.
• Modeling different “paths to innovation” may lead to a more sustainable organization that is resilient with respect to changing conditions.
• Bring the exciting new ideas in assessment, to be used in conjunction to the ones schools are mandated to use. Help schools see the additional instructional advantages of the new measures.
• Find ways of helping the school take part in existing online support communities that are being created, or create one with a local college.
• Teachers are professionals and deserve professional tools. Bring user-friendly data analysis and statistical packages that can be used to analyze student data.
• Teachers go through the same “productivity dip” that we all experience when we start on a new topic or a new methodology. If we ignore the time this takes, almost anything besides doing the usual will be deemed a failure.
• Teachers who use technology to empower students to learn report more time spent with the students with the greatest need, helping achieve desired classroom gains. (i.e., what is learned, and how it is learned).
• Be modern in your science-interdisciplinary, including design, mathematics and social science. Go beyond the standards, while respecting them.
• Choose interactive, high performance uses of technology (e.g. interactive simulations, online data access, sensor probes), which are a staple of scientific and economic practice and a driver for change in the workplace. The classroom itself is a workplace that must have access to such tools.
• Think across courses. If you can’t work with several disciplines, point to the ways in which such work could be carried out, so teachers can talk with colleagues.
• Pedagogy makes a crucial difference in what students learn, and focusing on pedagogy implies changing the teacher’s classroom activities. This should be your ultimate measure of success, in addition to student learning gains. Both measures happen in different timescales.
• Consider science as a distinctive, unique discipline in terms of its interplay between theory, models, and observations.
• New pedagogies are based on a science-in-the-making approach, which is more real and motivating for students.
• Education research has moved from theories of learning to theories of cognition, emphasizing the development of thinking and problem-solving skills.
• A pedagogy that incorporates multiple visual representations and allows novices to start from a concrete representation of phenomena closely correlated with observable reality is more generally accessible than one based on verbal, abstract, symbolic, or mathematically oriented expressions.
• In many cases students may have verbal and mathematical deficiencies that stand in the way of learning science. Use of multiple representations and qualitative reasoning can open the doors to learning and motivation.
• Motivation is crucial for educating audiences traditionally not well served by the pedagogies in use. Think of all students—not only future scientists and engineers, but future technicians, programmers, instructors and legislators.
• Laboratories are important in science and engineering education, and should be used well. Their messiness is crucial for developing observation skills. Have the students build connections between your examples and the real world as validation for ideas and models, and help them see the natural roles of experimental errors and of averaging results or discarding some data as integral to the process of science.
• Results from research suggest that understanding what it is to understand stays with the student and transfers to other school activities. Encourage reflection, either orally, or written.
• Help teachers access and use student data for targeting instruction to improve individual student performance.
• Consider and contribute to the sciences of learning and of teaching.
• Motivated students make a very effective transfer mechanism for ideas and practices. They have much to contribute to any project.

Page 12