AlgebraByExample: A back door entrance into the everday classroom

SERP History
The Strategic Education Research Partnership (SERP) was established in response to a National Research Council (NRC) committee report that envisioned a partnership between leading scientists and education practitioners to develop a problem-focused, coherent program of education research and development tightly coupled and interactive with practice (NRC, 2003). The vision called for an organization with a central coordinating capacity, a set of long-term partnerships with school district “field sites” (imagined at a number in the range of 12 to 15), and a set of networks composed of those in different field sites with common areas of expertise (assessment, school organization, etc.). Key to the vision was the proposed capacity to systematically build work over time, deliberately linking efforts across sites so that progress and knowledge accumulation would be accelerated.
Fewer than 10 years after the release of the SERP report, the SERP Institute, with its headquarters in Washington, DC, coordinates RD&D efforts in three field sites: Boston, San Francisco, and a multi-district field site in partnership with the Minority Student Achievement Network (MSAN).
SERP Infrastructure for RD&D: rationale
School systems are highly complex, loosely coupled (Meyer & Rowan, 1978; Weick, 1976) organisms dependent for their functioning on distributed responsibility and authority for the many inputs into education delivery. Change in such organizations – whether in education or other sectors – requires agreement from many parts of the systems, any one of which can stop the effort in its tracks (Garvin and Roberto, 2005). Creating the capacity for change through Research, Development and Design (RD&D) efforts therefore requires the ability to work across multiple components and levels of the system. But to do so successfully requires a major investment in establishing relationships, building trust, developing norms of interaction, and cultivating shared commitments. These commitments are not supported by the current incentive structures in either universities or school systems, and they cannot be quickly won. The SERP model is thus anchored by an investment in the creation of long-term “field sites:” ongoing partnerships with school districts in which the norms and routines of collaboration evolve with experience and become deeply rooted over time. The bet SERP is making is that with appropriate infrastructure a concerted and sustained effort to address the challenges of crossing institutional boundaries will produce work that is of high quality, high relevance, and high interest to the communities of researchers, practitioners, and policy makers.
Investment in “infrastructure” is common parlance in economics, where it refers to the structures—roads, airports, energy grids--that support economic activity. Whether an infrastructure investment is warranted depends on the importance and the permanence of the goal. An infrastructure investment in a grand performance hall is not made for a single Broadway show for example. But cities enhance infrastructure for the Olympics. More importantly, a city’s ongoing investments in infrastructure are for purposes of making possible whole classes of activity –commerce, employment, education, performing arts, sports events--that are highly valued and not specifically defined. SERP’s goal is to build an infrastructure that is the metaphorical equivalent of new roads that link researchers, practitioners, and designers in order to allow for more routine engagement in “problem-solving” RD&D.
The infrastructure investment would make little sense if the purpose were to get the quickest solution to a single problem, such as: “how can we improve the algebra performance of 8th grade students in District A.” There are many promising products and practices that, if matched appropriately to the student population and well implemented, are likely to produce improvements in student achievement. But these are not the questions that motivate the SERP enterprise. Because we are interested in sustainable, scalable, and continuous improvement, we ask how routine and supported interaction among researchers, practitioners, and designers can change three things: 1) the nature of the questions we ask (focusing on problems of practice and recruiting necessary expertise rather than shaping the problem to the expertise); 2) the complexity of our understanding regarding improvement as a continuous process (rather than as an event); and 3) the professional identity of participating education researchers and practitioners (drawing researchers into applications of their knowledge through interaction with designers, and subjecting practice to diagnosis and experimentation).
Research knowledge is critical to effective problem framing, but that knowledge does not stand on its own. It needs to be contextualized by those with knowledge of practice, and put to use by those with knowledge of design. The SERP model calls for “principled collaboration,” a term that signals the need for an intentional and managed approach to interaction that will support productivity. Identifying principles is critical to becoming better at the enterprise in which we are engaged. They are the vessels for carrying lessons learned from one effort to the next.

We touch on a few emerging principles below, with the proviso that their status is quite tentative: they are observations offered by, and discussed among, a group of SERP collaborators who are learning by doing. Because we are attempting to cut a path through poorly charted territory, the principles are in a stage of rapid evolution.

Principles guiding problem definition
SERP processes locate the framing of problems in teams that include members from the multiple disciplines in the research community, representation from multiple levels of school districts, and with people who bring design expertise. Norms for interaction include the following:
o district leaders determine the priority, and have veto power for any planned activity in schools;
o the problem, and the approach to solution, must be framed for relevance to a wide range of districts;
o researchers have ultimate authority in research design in order to ensure research validity and integrity, but logistics of research conduct are negotiated with the districts and schools.
In order to follow the contours of a problem, reframing or broadening the problem definition is expected over time, requiring greater flexibility in the RD&D process than is desirable in more traditional research settings.
Principles that guide co-design of programs, practices and tools
It is taken as a given that practitioners, researchers, and designers are all engaged as “co-developers” in the design process. Additional principles emphasize:
o the need for tools to have “handles” that allow teachers to pick them up and learn by doing, rather than requiring extensive training up front;
o the importance of designing tasks that stimulate the desired behavior directly rather than requiring teacher/administrator expertise before the task is potent;
o the need to design for teachers at three phases of expertise (Phase I: focused primarily on classroom management; Phase II: able to focus on student engagement, often going beyond the regular curriculum, and Phase III: able to focus on student thinking and learning, adapting curriculum as appropriate. (Schoenfeld, 2011).
o the need to test ideas early and often in practice contexts, avoiding natural tendencies to “make it beautiful” before the “it” is trialed.
Principles of designing for scale from the outset
Designing in context can lead to a local focus that will impair scalability. SERP work disrupts that focus with explicit and repeated attention to designing with scaling in mind. Lessons pertaining to scalability are harvested from every effort and incorporated into evolving design principles. Current principles emphasize:
o digestibility: new approaches must fit well with the routines of classrooms and schools, and live within the mechanisms for district management and decision making.
o interpretability: practitioners need to know how and why to use new tools, programs, or practices and need feedback on their impact—thus requiring the design of feedback mechanisms.
o sustainability: tools, programs, or practices must fit within the budget constraint and professional development model of districts or provide extensive guidance on how to reallocate resources.
Principles for knowledge accumulation
SERP work is made available through websites that are structured to share and support the use of SERP tools, knowledge, and research instruments. The goal is to make these sites as interactive as possible so that others can directly comment on all aspects of SERP work, and engage in its improvement. A companion goal is to represent related work by others when possible so that the sites help to organize ideas that contribute to, or extend in some way, work being done by SERP teams. A SERP website on internal coherence that will become public this calendar year is one example. While the SERP work around which it is organized has been led by Richard Elmore in the Boston field site, video interviews with Tony Bryk, Amy Edmonson, and Roger Goddard all appear on the site, directing viewers attention to related work on organization coherence. Funding constraints have limited our efforts in this area thus far however, but the goal is to routinely provide links to related work whenever possible.

From the outset, districts committed to participating in a process that included rigorous controlled experiments. We have conducted multiple studies and are still in the process of analyzing results from them and ongoing data collection efforts. Studies include (1) a year-long study investigating the implementation patterns and student learning outcomes when 28 AlgebraByExample assignments were used as compared to control assignments; (2) multiple “short-term” studies that span a single Algebra I unit; (3) (to begin in the 2011-2012 academic year) “mid-term” studies that span two Algebra I units; (4) (to begin in the 2012-2013 academic year) a year-long study of 43 assignments used throughout the entire year.
For the year long studies, pre and post tests include sample items from participating districts’ state wide assessments. The short term studies include pre and post test items that address students’ conceptual and procedural mathematical knowledge. In addition, to understand better how student’s assessments of their interest in and motivation to do math influence and influenced by our assignments motivation survey data are also collected.
During the first phase of this work: 2007-2009, with funding from the Goldman Sachs Foundation, a SERP-MSAN team designed a bank of 24 Algebra I assignments that includes worked examples with prompts for self-explanation, interleaved with typical procedural problems. This bank of assignments was completed in February 2009, and was tested for usability and feasibility with several teachers across five MSAN districts during the 2008-2009 school year. Teachers who used the example-based assignments indicated that the assignments prompted interesting classroom discussions about students’ misconceptions, and that they felt their students learned more from these assignments than from typical, problem-based assignments. Collaborating teachers who had individual students “think aloud” while working with an assignment also noted that the examples both caused students to confront their misconceptions and helped them figure out how to solve other problems in the assignment. Analysis of the data related to student learning demonstrated that compared to students who completed control assignments; those using the example based version performed better on the post-test controlling for pre-test performance. The effect was strongest for non-Asian minorities who attempted the highest number of worked example problems.
Further, results of a short-term pilot study with three classrooms of students (n = 51) using four of the intervention assignments were extremely promising. Half of the randomly chosen students (n = 26) in each participating class completed four of the example-based assignments over the course of their chapter on equation solving, while the other half (n = 25) completed control assignments that were closely matched in content, but in which all items were problems to solve (i.e., no worked examples). Students also completed a pretest and posttest of conceptual and procedural knowledge. Results revealed that students who received example-based assignments improved more than students who received control assignments. The effect of the example-based assignments was even more pronounced for minority students and for low-achieving students. Though an appreciable achievement gap was found at pretest (63% vs. 72% correct), no difference was found between groups at posttest (both Caucasian and minority students answered 72% of problems correctly).
Analysis of motivation survey results in short-term studies suggests this may be related to students’ expressed interest in mathematics; students who indicate they enjoy math appear not to benefit particularly from the worked example assignments as compared to those who indicate they do not like math.
The initial results from both teacher survey and pilot data point to the promise of the collaborative approach fostered in the SERP-MSAN field site. The design process provided the opportunity for practitioners to shape the design of the intervention, and to provide feedback each step of the way. While the research team drafted the assignments initially to target concepts and misconceptions that the research literature points to as problematic for students, a team of practitioner co-developers reviewed and in some cases reshaped them so that the language and level of challenge was appropriate to the student population. After the initial years of implementation, input was sought from a larger group of teachers who worked with the assignments. That feedback provides the foundation for the next phase of development that is currently underway funding by the Department of Education. It also provides critical information in our dissemination strategy.
In focus group discussions, we received several suggestions from teachers on how the approach could be more easily implemented and more useful for improving student learning. First, the teachers indicated that the scope of most of the 24 assignments was too broad, incorporating problem types that were typically taught over the span of several days, rather than a single lesson. This trait often made the assignments difficult to implement within the planned curriculum, forcing teachers to decide whether to use the assignment early on, when the last few problems would include content that was still unfamiliar to students, or to wait until all of the material was covered, when the early problems would constitute a review. Teachers were also concerned that individual misconceptions or common errors were typically only covered once in the whole set of assignments. They suggested that it would be better to have a higher dosage of assignments on key topics, so that misconceptions and common errors could be targeted. Finally, the teachers indicated that engaging in think aloud exercises with students had been of great value in supporting their understanding of students’ misconceptions, and in highlighting the value of the assignment design. They proposed incorporating thinking aloud training into the intervention. As a result of our focus group data, we decided that, rather than test the efficacy of the existing bank of assignments, further development of the approach and the materials in accordance with the teacher feedback is warranted.

In the current climate of shrinking federal and foundation budgets, SERP recognizes that we will also need to tighten our belts further. SERP already seeks and receives considerable support from faculty members who are sometimes but not often supported financially by our SERP won grants at levels that reflect the amount of effort they commit. Innovation and ideas emerge through our collaborative and intensive partnerships, but we also seek and use our websites to solicit input from the larger community of those who use our tools. The hardest sell for our organization remains the need for infrastructure support that nurtures partnership work before tightly defined projects have been identified.

SERP’s mission to address pressing problems of educational practice includes a goal to work collaboratively with other research-practice partnerships to share and distill principles that can advance the field and be used to improve the environments in which such efforts operate. As described, whenever possible our tools are made available freely on the web along with support materials that ease their adoption in varied school settings.