Academic Exchange Quarterly Spring 2001: Volume 5, Issue 1
Collaborative Action
Research
to Assess Student Learning
and Effect Change
Linda A. Catelli,
Joan Carlino,
Catelli is a
Professor of Education and Director of Project SCOPE II in the
Abstract
Higher education has been called upon to demonstrate
its commitment, capacity, and effectiveness in preparing "quality"
teachers in the 21st century. As part of the assessment/audit process, institutions
will be expected to provide accrediting agencies with evidence of their
effectiveness to prepare competent teacher candidates. This article presents
action research studies collaboratively engaged in by a college and a
partnering school district to assess the college's teacher education students.
Videotaped performances of the students are analyzed to assess their teaching
performances as indicators of learning and to establish baseline data for
instituting change and improvement. In this article a summary of the action
research video studies is presented as well as selected data/findings.
Introduction and Background Information
In the next decade the nation
will need over 2.2 million teachers (Riley, 1999). In general, colleges and
universities around the nation have been called upon, now more so than in the
past, to demonstrate their commitment and ability to meet the changing needs of
American society (Goodlad, 1999; Schmidt, 2000). With
regard to the need for teachers, higher‑education institutions have been
asked not only to meet this need, but also to solve the problem of preparing
"quality" teachers for
At present, the U.S. Department of Education, through the 1998 Congressional Higher Education Act, Title II, Section 207, is requiring states and higher‑education institutions to assess and publicize the effectiveness of their teacher education programs to prepare quality teachers. Section 207 of the law includes new accountability measures that require states and their colleges and universities to announce annually the percentage of students who have passed state teacher‑certification exams, and to report on other quality indicators and licensure requirements as well. In essence, for higher education there will be a national and state report card (see U.S. Department of Education, 2000a, 2000b). Subsequently, if a college or university does not achieve a designated passage‑rate set by the state in which the college or university resides, then the state's education department or its board of regents may close that institution's teacher education program or subject it to deregistration.
For example, the New York State Board of Regents has
set an 80% student passage‑rate for its higher‑education
institutions. That is, if less than 80 percent of the institution's teacher
education students pass one or more teacher certification examinations, the
institution's teacher education program is subject to deregistration. And if a
program, found deficient, does not demonstrate significant annual improvement
toward the 80% standard, it, too, is subject to deregistration (see New York
State Regents Task Force On Teaching, 1998, pp. 24‑25). For small
colleges and universities this could be financially devastating; and for larger
institutions it may mean that they will no longer be able to use their teacher
education programs as the institution's "cash cow."
Further, the U.S. Department of Education and state
education departments around the nation are requiring colleges and universities
to demonstrate their "institutional" commitment to prepare quality
teachers and their capacity to do so by mandating that the institution engage
in a comprehensive assessment and/or audit conducted by a designated
accrediting association or agency. What we are currently witnessing is not only
an assessment of programs in teacher education and the unit of the college that
sponsors the program (i.e., schools and departments of education), but also an
assessment of the entire institution in terms of its capacity to prepare
"quality" teachers. (See the National Council for Accreditation of
Teacher Education [NCATE], 2000, and Teacher
Education Accreditation Council [TEAC], 1999, for the standards, rubrics and
criteria used to assess institutions.)
As a result of these events,
college and university presidents are becoming more aware of the issues,
problems, and concerns related to accreditation (i.e., involving the liberal
arts and science faculties in the preparation of teachers). Also, they are
becoming aware of the financial consequences of not securing accreditation for
their institution. It is interesting to note that in
It is in the effort to meet these challenges that collaborative action research can play a critical role. There are three major concepts we would like to promote and put forth: one, that school‑college partnerships are critical to the process of preparing "quality" teachers in the 21st century; two, that collaborative action research engaged in by school and college personnel is an integral part of the process for instituting change and improvement in education and for demonstrating programmatic‑institutional effectiveness in the preparation of teachers; and three, that data/findings of action research studies that focus on analyses and assessment of student performances can be used (1) to establish baseline data for the program, (2) as indicators of student learning, and (3) as evidence of the students' professional competence and/or achievement of teaching‑learning standards.
For purposes of this article, we will focus on action research video studies that were conducted by members of a school‑college partnership project and the use of the data to analyze and assess the teaching performances of teacher education students. Our intent is not to describe the studies in great depth, but rather to highlight one or two of the findings and explain their function in assessing student performances and effecting programmatic‑institutional change and improvement. In the following sections we will present first, a synopsis of the school‑college partnership project; second, a summary of the action research video studies engaged in by members of the partnership; and third, an explanation of selected data and findings, and their use in the assessment and change process.
The Partnership and Its Partners
In June of 1998
In June of 1998, when Project
SCOPE was expanded and adapted to other academic settings at
Summary of the Action Research Video Studies
The action research video studies that were
conducted during the 1998‑2000 academic years were part of Phase One of
the partnership's overall strategic plan to effect change and improvement in K‑18
student learning (Catelli, Padovano, Costello, &
Diver‑Stamnes, 1999). The studies were
participated in and conducted by the partnership's graduate education students
(Dowling College), school teachers who served as cooperating teachers in the
partnership project (the North Babylon School District), student teachers
(Dowling's teacher candidates for initial teacher certification), and the
professor‑director of the project. All parties volunteered to become
members of Project SCOPE II and subsequently played a role in either the design
of the action studies and/or the implementation of Phase One of the
partnership. The primary focus of the action research video studies is on the
student teachers' classroom teaching performances and the behaviors of their
pupils. The overall purpose of the studies is to establish baseline data for
changing and improving the preservice elementary
teacher education program as well as provide a descriptive analysis and an
assessment of the teacher candidates' teaching performances.
Action research was the method employed in conducting the action studies. Traditionally known for its systematic and cyclical method of operation, (i.e., identification of a problem, data collection, data analysis, implementation and evaluation), "action research" is fundamentally employed as a methodological strategy for improving educational practice, and for the production of relevant and practical knowledge in education. Within the context of a school‑college partnership, action research may be more appropriately labeled as collaborative inquiry, denoting a process of self‑study involving the members of the partnership in a type of analysis and evaluation that is critical, reflective, and focused directly on the work of the partnering members in relation to the partnership's goals. The core, then, of the research that is conducted in a partnership program, and in particular in SCOPE's holistic partnerships, resides in the fact that it is collaborative, participatory, and critical; and that it is directed at changing and improving education on both levels in an integrative way.
Nine action video studies were conducted by thirty-five of SCOPE II's action researchers arranged in eight research teams. The eight teams identified and analyzed videotaped classroom instructional behaviors (i.e., asking questions) of twenty-nine elementary‑education student teachers (teacher candidates). Each research team was responsible for analyzing four to five student teacher videotaped lessons that were taught to classes of no more than thirty-five elementary school pupils. All twenty-nine student teachers were members of Project SCOPE II's partnership program during the 1998‑1999 academic year. Each student teacher was videotaped during the second month of his/her student teaching experience, after one formal observation by a college supervisor had taken place.
Using multiple observational systems and adaptations of systems, the eight teams of action researchers during the 1999‑2000 academic year coded the videotaped classroom behaviors for five second intervals and analyzed teacher‑pupil performances. Frequencies and percentages of time the student teachers and their pupils spent exhibiting classroom instructional behaviors were computed. The resulting percent occurrences of the behaviors indicate the amount of time the student teachers and their pupils engaged in specific instructional actions that facilitate pupil learning and that reflect teaching and learning standards. Assessment criteria and rubrics that are related to state and national standards were then applied to the classroom video performances in order to discern either levels of the teacher candidates' performance or the presence‑absence of an instructional behavior. Thus, the analyses of the teacher candidates' teaching performances were conducted in an effort to:
1.
identify emerging classroom teaching behaviors, instructional actions,
and patterns;
2.
identify classroom pupil behavior;
3.
establish baseline data for changing and improving the preservice elementary teacher education program; and
4. assess the teacher candidates' instructional performances in regard to criteria extracted and adapted from four sources:
a)
b) standards set forth by the
National Board For Professional Teaching Standards (1998);
c)
criterion measures identified by Educational Testing Services' PATHWISE program (1995);
d) the pedagogical preferences of a representative of the partnering school district.
The ultimate goal of the work is to institute the type of change that brings about student learning on both the school (K‑12) and college (teacher education) levels, and demonstrate programmatic and institutional effectiveness in preparing competent teacher candidates. The series of studies exemplify SCOPE II's integrative‑holistic approach to changing and improving preservice and inservice teacher education and K‑12 student learning in a coordinated fashion.
To achieve the overall purpose of the action video studies, a number of research goals were formulated. Two of the research goals are listed below:
Goal 1: To determine the percentage
of class time devoted to "substantive instructional teacher‑pupil
behaviors," (i.e., asking questions, giving corrective feedback), and the
percentage of time devoted to "non‑substantive behaviors,"
(i.e., managerial, organizational, and disciplinary behaviors).
Goal 2: To determine the number of pupils cognitively and/or physically "on task" during a class period. Figure 1 identifies the substantive and non-substantive teacher-pupil behaviors that were coded and analyzed (see Figure 1).
Selected Findings and Their Use in the Assessment and Change Process
Basically, for Goals 1 and 2, the preliminary data and findings reveal that twenty-six (or 90%) of the twenty-nine student teachers engaged in substantive instructional behaviors for at least three-fourths (75%) or more of their class time or lesson. Seventy‑five percent (75%) was set as the minimum "time" criterion. Lessons ranged from twenty minutes to one and a half hours. It should be mentioned that those lessons that exceeded one hour were coded only up to the first hour of the lesson. With regard to Goal 2, in all twenty-nine videotaped lessons, twenty-one or 72% of the student teachers had at least 85% of their pupils "on‑task" for the lesson. Thus, a high percentage (90%) of the student teachers conducted lessons in which 75% or more of their class time was spent in substantive instructional behaviors‑‑while a relatively low percentage of student teachers (10%) spent 25% (or higher) of their class time in management and discipline actions.
Upon closer examination of the data we found that the three student teachers who had below 75% of the time devoted to substantive instructional behaviors either had very young pupils (i.e., kindergarten age), or they were conducting an initial lesson using the method of "cooperative learning" which required more direction and pupil management. There were, however, cases where better management techniques could have been employed. The overall intent of Goal 1 is to have the partnership members, i.e., the professors, supervisors, and cooperating teachers, prepare preservice teachers to conduct lessons where management time is kept to a minimum, and time devoted to substantive instructional matters and behaviors is maximized‑‑thereby allowing for the possibility of increased pupil learning.
Both Goals 1 and 2 relate to or were adapted from the following sources: (a) item 3 of the New York State Teacher Standards which states that "The teacher effectively manages classrooms that are structured in a variety of ways, using a variety of instruction methods. . . . " (New York State Board of Regents Task Force On Teaching, 1998, p.14); (b) criterion C5 of Educational Testing Services' PATHWISE program which assesses the student‑teachers' classroom performances on "Using instructional time effectively." (Educational Testing Service [ETS], 1995, p.40); (c) a review and an analysis of the 27 New York State Student Learning Standards; and (d) a review of the NCATE's Standard 1, the Candidate's Performance . . . the Candidate's Knowledge, Competence, etc. (NCATE, May, 1999, pp. 3‑9).
The data and videotapes were analyzed further to reveal specific teaching behaviors that facilitate pupil learning and those instructional actions identified as pedagogical preferences by a cooperating teacher of the partnering school district. The instructional behaviors that were cited by the teacher include the following behavioral categories seen in Figure 1: [1] Accepts pupil(s) feelings and/or Sets the climate and tone of class; [4a‑4b] Asks lower and/or higher‑order questions; [3] Accepts, uses or extends ideas of pupils; [6] Gives corrective feedback to pupils; [5d] Articulates the goal‑objective‑rubric or expectation for the lesson; and [8] Pupil(s) responds to questions. Our findings reveal that for these instructional classroom behaviors, the student teachers, as a group, had low frequencies or percent occurrences of time engaged in each of the behaviors. These classroom behaviors are pedagogical areas or instructional actions that facilitate pupil learning. They are the areas of competence that should be targeted for improvement in the teacher education program.
In addition to conducting descriptive analyses, all twenty-nine student‑teacher videotaped performances were scored according to selected assessment criteria and rubrics adapted from the PATHWISE Program (ETS, 1995) and the National Board For Professional Teaching Standards (1998). The analyses of all that data have yet to be completed. However, with regard to learning outcomes or areas of teaching competence, the criteria set for assessing the videotaped performances, at an initial level, revolved around the following competency statements for student teachers:
o
To conduct lessons for which at least 75% of the class time is devoted
to substantive instructional teacher‑pupil behaviors/actions (i.e.,
asking higher‑order questions), and no more than 25% of class time is
devoted to non‑substantive behaviors/actions (i.e., managerial,
organizational, and disciplinary behaviors).
o
To conduct lessons that have at least 85% of the pupils cognitively
and/or physically "on task" during the lesson.
o
To demonstrate selected instructional behaviors/actions that facilitate
pupil learning and achievement (i.e., asking higher‑order questions,
giving corrective feedback).
o To demonstrate acceptable levels of classroom performances with regard to selected teaching‑learning rubrics and/or behavioral criteria that are linked to and aligned with national and state teaching‑learning standards.
We envision that the resulting analyses and continuous assessment of videotaped performances of future student teachers of the program will be used as indicators and evidence of student‑teacher learning.
Conclusion
To summarize, the preliminary findings cited previously as well as additional data and findings of the nine action research studies will be used
o
to establish baseline data for future studies that focus on effecting
changes and improvements in the preservice teacher
education program;
o
as indicators of student‑teacher learning;
o
as evidence of professional competence and/or achievement of national,
state, and local teaching‑learning standards;
o
to institute changes in courses and field modules of the program that
target those pedagogical areas where we find that the student teachers (teacher
candidates) need to develop higher levels of skill or at least have more
experience prior to student teaching (i.e., asking higher order questions;
giving corrective feedback to pupils, etc.); and
o to design content for inservice training sessions to assist the partnership's cooperating‑supervising teachers in coaching their student teachers in the instructional behaviors that relate to national, state, and local standards of teaching and learning.
Finally, it is our hope that the descriptive data and findings of the nine action research video studies conducted during the 1998‑2000 academic years of the partnership will serve to initiate change and improvement in the teacher education program, and serve as baseline data for subsequent institutional assessment studies for purposes of accreditation. Further, and on a larger scale, we envision that the work presented will serve as a template for higher education in its effort to prepare quality teachers in the 21st century and provide evidence of its capacity to do so. Also, it is our hope that collaborative action research as portrayed in this article will contribute to the knowledge‑base on assessment of student learning, and on research associated with effecting educational change and improvement on both the college and school levels.
References
Catelli, L. A. (1992).
Against all odds: A holistic urban school/college partnership‑‑Project
SCOPE. Action In Teacher Education, 14
(14), 42‑51.
Catelli, L. A. (1995). Action
research and collaborative inquiry in a school‑university partnership. Action In Teacher Education, 16 (4), 25‑38.
Catelli, L. A. (1997). An
holistic perspective on school‑university partnerships in the twenty‑first‑century‑theory
into practice. In D. Lambert and A. Hudson (Editors), Exploring futures in teacher education: Changing key for changing times
(pp. 228‑246).
Catelli, L. A., & Nix, W.
(1992). Project SCOPE: An urban partnership venture-‑descriptions and
predictions. In G. Graham and M. Jones (Editors), Collaboration between researchers and practitioners: An international
dialogue ‑ Proceedings from the 1991 AIESEP‑NAPEHE
World Congress (pp. 105‑110).
Catelli, L. A., Padovano, K., Costello, J., & Diver‑Stamnes, A. Strategic models of school‑college
partnerships to effect change and improvement in student learning and teacher
education: A comparison of two approaches. Conducted a symposium and presented
a paper at the 1999 International Congress for School Effectiveness and
Improvement (ICSEI).
Educational Testing Service.
(1995). PATHWISE®: Orientation guide.
Goodlad, J. (2000). Rediscovering
teacher education: School renewal and educating educators. Change, 31(5), 29‑33.
National Board For
Professional Teaching Standards. (1998). Middle
childhood/Generalist standards.
National Council for
Accreditation of Teacher Education. (May, 1999). Institutional orientation: State of
New York State Department.
(1995).
Riley, R. (February, 1999). Riley calls for fundamental changes to
improve American Education; Announces steps to address accountability and
teacher quality: The sixth annual state of American education speech.
Government Speeches. Webcast of the Secretary's
Speech. [http://www.ed.gov/Speeches/9990216‑a.html].
Schmidt, P. (June 30, 2000).
States set a course for higher‑education systems-‑master plans aim
to ensure cohesive response to technological and economic changes. The Chronicle of Higher Education,
Section: Government and Politics, vol. XLVI, no. 43, pp. A27‑28.
Sirotnik, K. (1988). The meaning and
conduct of inquiry in school‑university partnerships. In Kenneth A. Sirotnik and John Goodlad (Eds.),
School‑university partnerships in
action: concepts, cases, and concerns (pp. 169‑90). NY: Teachers
College Press.
Teacher Education
Accreditation Council. (1999). TEAC:
Prospectus. Washington, DC: Author ‑ One Dupont Circle Suite 320,
Washington, DC: Author ‑ Department of Education, Office of Post
Secondary Education, 1990 K Street, N.W., Washington, DC 20006.