Academic Exchange Quarterly Winter
2009 ISSN 1096-1453 Volume 13, Issue
4
To cite, use print source rather than this
on-line version which may not reflect print copy format
|
This article should not be
reprinted for inclusion in any publication for sale without author's explicit
permission. Anyone may view, reproduce
or store copy of this article for personal, non-commercial use as allowed by
the "Fair Use" limitations (sections 107 and 108) of the U.S. Copyright law. For any other
use and for reprints, contact article's author(s) who may impose usage fee..
See also |
The
Process of Creating Program Learning Outcomes
Lynn Henrichsen,
Brigham Young University, UT
Mark Tanner,
Brigham Young University, UT
Lynn
Henrichsen and Mark Tanner both work as teacher
educators in the TESOL graduate program in the Linguistics and English Language
Department at Brigham Young University.
Abstract
Higher education is
undergoing a paradigm shift that involves establishing and assessing learning
outcomes, but creating and measuring learning outcomes is not a simple
procedure. This article describes the process experienced by Linguistics and
English Language Department faculty at Brigham Young University in creating
learning outcomes and assessment measures for our degree program in TESOL
(Teaching English to Speakers of Other Languages) and the key lessons we
learned.
Introduction
A significant paradigm shift is taking place in higher education in the United States and other countries. This shift involves a stronger focus on course, program, and institutional “learning outcomes.” Very simply, learning outcomes are statements that communicate what students should be able to do as a result of completing their particular program of study. Creating learning outcomes for a degree program and putting them into operation, however, is anything but a simple process. Outcomes that meet specifications of clarity, learning focus, disciplinary focus, and measurability must first be created and published. Then, through regular and systematic assessment, evidence must be gathered to determine the degree to which students who complete the program have achieved the stated outcomes. Measures used to assess the achievement of outcomes must be valid and robust, as well as useful for making decisions on program design and curriculum improvement. Finally, evidence that the results of these assessment activities lead to improvement in teaching and learning must be provided.
In this paper, we highlight the results of a year-long process to create learning outcomes for a graduate program in TESOL. We first explain the importance of learning outcomes and how they should be created. Next, we share the process of creating program-level learning outcomes and the lessons we learned from the experience.
Learning Outcomes:
Their Nature and Importance
Over a decade ago, a
new dimension was added to the tradition of assessing student learning in higher education. Barr and Tagg explain, “In its briefest form, the paradigm
that has governed our colleges is this: A college is an institution that exists
to provide instruction. Subtly but
profoundly we are shifting to a new paradigm: A college is an institution that
exists to produce learning” (1995,
13, italics in original).
This paradigm shift has gained momentum in recent years as key educational and governmental leaders have become concerned that “higher education gives little return to society for the enormous resources it sops up” (Wood 2007, 1). Nowadays legislators, employers, parents of students, and the general public are all seen as “informed consumers of higher education” and they must be assured “that the goals of higher education have been achieved” (Middle States Commission 2003, 1). Likewise, university accreditation agencies have begun placing greater emphasis on the identification and assessment of learning outcomes. In this new climate of public transparency and accountability (Spellings 2006), higher education faculty and administrators must overtly and rigorously evaluate the learning accomplished by students in their classes and degree programs. In other words, academics will, sooner or later, encounter learning outcomes. That realization naturally leads to the question, how does one go about creating learning outcomes?
Guidelines for
Creating Quality Learning Outcomes
Our own recent experience has taught us that creating
quality learning outcomes and measuring their achievement is not a simple
process. The range of possibilities is broad. As the New England Association of
Schools and Colleges Commission on
Institutions of Higher Education (2005) explains: “Inquiry may focus on
a variety of perspectives, including understanding the process of learning,
being able to describe student experiences and learning outcomes in normative
terms, and gaining feedback from alumni, employers, and others situated to help
in the description and assessment of student learning.” Nevertheless, not just
any measures will do. The institution must devote “appropriate attention to
ensuring that its methods of understanding student learning are trustworthy and
provide information useful in the continuing improvement of programs and
services for students” (standard 4.50).
To guide faculty and administrators through the process of producing and measuring learning outcomes, some excellent books have been published (e.g., Middle States Commission on Higher Education 2003). Numerous websites have also been set up.[1] In this article, we do not repeat what these paper and electronic sources have already provided. Rather, we review our own experience developing learning outcomes for our academic program in TESOL and conclude with the lessons we learned from that process. The processes and lessons are, of course, relevant to educators in any subject area.
Our Experience—The
Process
The development of
learning objectives for our graduate program began in conjunction with the
university’s preparation for a re-accreditation team visit from the Northwest
Commission on Colleges and Universities (NWCCU). To prepare for this visit, the
NWCCU asked Brigham Young University to do the following:
1. Undertake a systematic process of identifying and publishing expected
learning outcomes for each of its degree programs.
2. Through regular and systematic assessment, demonstrate that students
who complete their programs have achieved these stated outcomes, using the full
range of appropriate primary and secondary sources of evidence.
3. Provide evidence consistently across its programs that results of its
assessment activities lead to improvement of teaching and learning (Frequently
Asked Questions 2006, 1).
As part of this university-wide process, each department was asked to review its major programs and determine core learning objectives that described how students involved in each academic program benefit from participation in it.
In our department, a team of TESOL faculty members met bi-weekly for an entire semester in order to establish the learning objectives for our graduate program. Some initial training in the writing of objectives was provided by an assessment specialist from BYU’s Faculty Center. The assessment specialist also met with the group regularly to provide feedback on learning outcomes and assessment measures that were being discussed.
The process we went through in creating the program-level learning outcomes began with initial brainstorming. Next, we connected courses currently offered in the program to the outcomes that had been identified. The third step involved revising, refining, and consolidating the outcomes into a set of simple, defined outcomes. The fourth entailed identifying assessment measures by which the program-level learning outcomes could be evaluated.
Initial Brainstorming
The TESOL faculty began the process of forming learning outcomes by brainstorming a list of items that we felt graduate students completing the program should either know or be able to perform. Accordingly, the outcomes were grouped according to what students should “know,” “do,” and “communicate.” For instance, within the area of “knowing,” faculty identified that students who were becoming English language teachers should have a core knowledge of linguistic systems. Program graduates, we decided, should also know and understand past and present theories of second language acquisition as well as current and historical trends and methods in language teaching. In the area of “doing,” faculty felt that students should be able to read, understand, and critically analyze professional research in the field. In addition, graduates should be able to conduct a thorough needs analysis, create effective lesson plans and teaching materials, and employ a range of teaching strategies. In the “communicating” category, sample learning outcomes included giving professional presentations and writing publishable reports of their own research and teaching ideas.
Connecting with Courses
Once the list of terminal learning outcomes had been generated, each item on the list was connected with the courses taught within the program. The matching of courses to the list of terminal outcomes helped faculty see whether or not existing courses did indeed lead to what had been identified as appropriate learning outcomes for the program. Through this exercise, a second list of more refined learning outcomes was created. This list identified 26 expected learning outcomes, which were subsequently grouped into three specific areas: 1) knowledge outcomes, 2) performance outcomes, and 3) communication outcomes.
Revising, Refining, & Consolidating
It was at this point in the process that we remembered the counsel from our academic vice-president to make learning outcomes “clear, simple and few” (Tanner 2006b, 1). We realized that having a total of 26 outcomes, to each of which multiple assessment measures had to be tied, would make the data collection process so unwieldy as to be unsustainable. Consequently, a few additional meetings were held to condense the outcomes down to a number that was more manageable, as well as consistent with other programs and departments across the university. In this spirit of simplification, we decided to have only two major learning outcomes, one for each major part of the program—the TESOL Graduate Certificate (first year of graduate study) and the TESOL MA.
The first major learning outcome was centered on what students completing the TESOL Graduate Certificate would be able to do. This learning outcome stated that all students completing the graduate certificate would be able to teach English language skills to non-native speakers of English in an effective and professional manner. We then reviewed the list of 26 previously identified learning outcomes to determine which ones might best serve as enabling outcomes for this major one. The result was a much shorter list, consisting of six enabling outcomes, such as (1) manifesting mastery of the English language, (2) demonstrating an understanding of the major systems of human language, and (3) demonstrating an ability to analyze and apply theories of second language learning and acquisition.
The second major learning outcome centered on what students graduating with the full MA degree should know and be able to do. This outcome was worded so that students would understand that in completing the master’s degree, they would need to go beyond the core competencies demonstrated in terminal learning outcome one, to develop specialized knowledge and skills within one or more of the following areas of TESOL: research, specialized teaching knowledge and skills, curriculum and materials development, assessment, and administration and supervision. Examples of the enabling outcomes listed under learning outcome two included: (1) analyzing, performing, and communicating professional level research; (2) developing professional quality curricula and materials for English language teaching; and (3) examining, selecting, and using methods conducive to effective learning of particular language skills and elements.
Identifying Measures
Once the learning outcomes were articulated and clarified, the next step in the process was to identify direct and indirect means of measuring student knowledge and performance. A careful review of course requirements and program level assessment tools helped faculty create program-specific direct and indirect measures, some of which are listed below:
Direct Measures
1. A well-written
master’s thesis or project
2. A professional
teaching portfolio
3. Supervisory
observation and mentoring reports of students’ classroom teaching
4. A written
philosophy of language teaching that embodies current language learning
theories and practices
5. A creative and
well-organized teaching materials file
Indirect Measures
1. Annual external
reviews of master’s theses and projects
2. Annual survey
and focus groups with graduating students
3. Surveys of
near-graduates in capstone course
4. Individual
course/ instructor evaluations
5. Alumni survey of
graduates
Lessons Learned
The drafting of learning outcomes and measures at the program level was a lengthy process requiring nearly fourteen months of effort by our TESOL faculty with input from an assessment specialist. In this process, we learned a number of key lessons that should be of value to others who may need to create learning outcomes for their academic programs.
Adjust Attitudes
When department faculty were first informed that they would be engaged in creating learning outcomes for each program and eventually each course, the initial temperament of many was that this process was some type of masochistic exercise being required by the upper administration. While the creation of learning outcomes was a requirement of the re-accreditation process, the academic vice-president for the university assured the faculty that this activity should be seen as “an imperative to become better” (Tanner 2006a, 2). His challenge to the university and to the faculty was “To become an even better house of learning by becoming more fully focused on learning” (Sorenson 2006, 1). We found that only after the faculty accepted the inevitability of our involvement in the process and acknowledged its potential benefits, could we begin to focus our energies on worthwhile discussion of what constituted learning outcomes.
Foster Collegiality
Within our department, the TESOL faculty was invited to be the first group to begin the process of articulating learning outcomes because of our high level of collegiality and our previously demonstrated willingness to tie courses to specific learning goals rather than to the special interests of individual faculty members. These favorable qualities notwithstanding, the process of discussing philosophical underpinnings of the program and specific areas of change still produced some heated discussions. Faculty frequently had to be reminded that the purpose of the meetings was to promote positive change in the program and its students. At times the discussion included honest acknowledgement of changes occurring in the larger profession that were moving in directions different from certain faculty members’ established areas of specialization. For such discussions to lead to success (and not discord), an overriding spirit of collegiality was crucial.
Think Innovatively
Even though the discussion in our learning outcomes meetings occasionally became emotional, the articulation of specific learning outcomes also provided faculty with a synergistic opportunity to look beyond “the way things have always been done.” This “thinking outside the box” allowed us to explore current and future needs of the students and changing directions of the profession and re-conceptualize the framework of our program and its course offerings.
Put Things in Writing
Another valuable lesson we learned was that faculty meetings were consistently more effective when discussions and assignments from the previous meeting had been articulated in written form. It was the committee chair’s responsibility to do this and see that faculty came with their assigned tasks completed. When we had written descriptions that could be reviewed and critiqued, we used our meeting time more efficiently and effectively.
Prepare for Long-Term, Ongoing Development
We were not far into the process of creating program-wide learning outcomes when we realized that we had only encountered the tip of the iceberg. Once the program-level outcomes had been determined, we then had to identify measures for assessing the learning outcomes. These processes had to become a permanent part of our normal operations.
Now, instructors are expected to specify learning outcomes for each course that they teach, and those course outcomes are tied to our program-level outcomes. Creating, coordinating, and refining these outcomes is an ongoing process that will require multiple iterations before everything is polished and the coverage of learning outcomes becomes complete.
Conclusion
While the process of articulating and assessing program-level learning outcomes was long and time-consuming, it produced a paradigm shift of sorts within our own program. Rather than focusing on what is taught and how it is taught, we now focus on what students learn and how they learn it. We are no longer content to simply describe what texts will be read and topics discussed in a class. Instead, our focus is on the learning activities that students engage in, as well as on identifying clear and measurable means to determine how (and how much) student learning has occurred.
Our experience is, of course, not unique. The creation of learning outcomes is now an expected part of the process in which all accredited colleges and universities participate. Higher education’s new level of transparency and accountability to both the consumers of educational knowledge and those who accredit such institutions is not likely to diminish in the future. For those who face the prospect of creating and assessing learning outcomes for their programs, we hope the information and lessons we have shared will be beneficial.
Endnotes
[1] See the Brigham Young University site at
https://learningoutcomes.byu.edu/, the North Carolina State University site at
http://www.ncsu.edu/assessment/resources/resources.htm, the University of
Illinois-Urbana/Champaign site at http://www.oir.uiuc.edu/assessment/home.html,
and the University of Nebraska Kearney site at http://www.unk.edu/academicaffairs/assessment/Academic/
for some good examples.
References
Barr, Robert B., and John Tagg. November/December 1995. “From teaching to learning: A new paradigm for undergraduate education.” Change, 13-25. Full text is also available at http://critical.tamucc.edu/~blalock/readings/tch2learn.htm
Frequently Asked Questions: Program-Level Learning Outcomes, Evidences, Assessment. December 12, 2006. Provo, UT: Brigham Young University.
Middle States Commission on Higher Education. 2003. Student learning assessment: Options and resources. Philadelphia, PA: Author.
New England Association of Schools and Colleges Commission on Institutions of Higher Education. 2005. Standards for Accreditation. http://www.neasc.org/cihe/standards_for_accreditation_2005.pdf (accessed February 12, 2008).
Sorenson, Lynn. Fall 2006. “Student learning outcomes.” In Focus on faculty: A newsletter for those who teach at Brigham Young University, 14, 1-2.
Spellings, M. September 26, 2006. Secretary Spellings' Prepared Remarks at the National Press Club: An Action Plan for Higher Education. http://www.ed.gov/print/news/speeches/2006/09/09262006.html (accessed April 17, 2008). See also Spellings, M. 2006 A test of leadership: Charting the future of U.S. higher education. http://www.ed.gov/about/bdscomm/list/hiedfuture/reports/final-report.pdf (accessed July 6, 2007).
Tanner, John. 2006a. Building a better house of learning. BYU Annual University Conference faculty session. 29 August 2006. 1-4. http://avp.byu.edu/documents/pdf/auc2006/pdf (accessed April 17, 2008).
Tanner, John. 2006b. Carrots, vision, and learning outcomes at BYU. http://avp.byu.edu/documents/pdf/carrots.pdf (accessed April 17, 2008).
Wood, Peter. March 27 2007. “Spellings bee”. National Review Online, http://article.nationalreview.com/?q=NzBjZDFmMjJhM2E4ODM3YzMxN2UwOTViMGZiMDlhNDY=