The State University of New York College at Plattsburgh
Comprehensive Assessment in Academic Disciplines
Assessment within academic disciplines must meet two seemingly contradictory criteria: on the one hand, it must reflect the particular perspectives of the departments involved; on the other, it must yield comparable data on student performance.
To reconcile these divergent requirements, the State University of New York College at Plattsburgh organized an assessment consortium composed of five SUNY colleges: Buffalo, Cortland, Fredonia, Plattsburgh, and Potsdam. The colleges sent teams of two faculty from several, but not necessarily all of the following disciplines: Biology, Teacher Education, Foreign Language and Literature, History, and Psychology.
The purpose of the consortium was to develop comprehensive examinations in each of the five disciplines. To achieve this purpose, teams met over a period of three years and engaged in a step-by-step process that yielded considerably more than a set of assessment instruments.
The teams began by defining the nature of their disciplines with respect to undergraduate learning, discussing not only content but the field's epistemology as well as its contributions to society, its purpose, values, and ethical standards. Faculty next examined the assessment strategies most appropriate to each discipline, frequently discarding approaches grounded in traditional psychometric theory in favor of measures that seemed most accurately to reflect learning in each field. Most importantly, at the conclusion of the process they engaged in discussions of the implications of assessment for curricular reform.
The Biology group developed an assessment of the following goals: 1) to provide students with a broad-based general knowledge and familiarize them with the language of science; 2) to provide students with some basic laboratory skills and training in the proper use of standard instrumentation; and 3) to increase student awareness of current issues in science, especially the relationships of science to society and the ethical questions inherent in the use of technology.
The Psychology group developed an objective, multiple choice test that meets rigorous standards of measurement. The instrument consists of three parts: 1) a content section that assesses knowledge of concepts, terms and principles in ten basic areas of Psychology; 2) a critical thinking or "process" section that assesses ability to analyze research; and 3) a section that assesses the relationship of course experiences to the scores on the first two sections.
The Teacher Education assessment model included a two-dimensional, 40-cell matrix generated with four domains of assessment and ten program elements. The Education group sought to find assessment strategies and data sources to label judgments in each of the cells of their matrix. They explored potential contributions from data sources such as vignettes, portfolios, observations, video tapes, questionnaires, interviews, written exams, jury by peers, etc.
The History group developed and refined an assessment instrument which allowed students to reflect in various ways on what they have learned during an undergraduate course in History. Students are assessed on their synthesizing/analytical capabilities as supported by comparative historical reasoning and evidence. An interpretive essay assignment allows students to synthesize diverse historical information as they attempt to validate or challenge provocative generalizations. Students are asked in addition to demonstrate their command of factual contextual historical knowledge. A battery of eight multiple choice tests allows them to choose areas of competency which they are certain to have encountered. The final version of the assessment includes questions on the nature of the historical discipline and its practices.
The Foreign Language and Literature group focused its activities on six key areas embodied in all foreign language programs: oral proficiency, writing skills, listening comprehension, reading proficiency in the foreign language, "proficiency" in literature, and knowledge of civilization and culture. The group constructed grids for each of these six areas modeled on the ACTFL/ETS guidelines for proficiency. Oral skills are assessed in an interview format and the writing and literary interpretive skills through written assignments. A multiple choice format assesses basic knowledge.
The project was evaluated on the basis of its achievement of three goals. The first was the creation and maintenance of an inter-institutional consortium within a large public university system. Achievement of this outcome was demonstrated by the actual sustained efforts of the consortium. Furthermore, participants engaged in an analysis that generated insight into the dynamics of operating such a consortium. These insights can be shared with other groups attempting to develop similar inter-institutional projects.
The second goal was development of assessment-related conceptual frameworks for each discipline so that assessment strategies could be devised in accordance with a comprehensive understanding of the key elements and boundaries of each discipline. These frameworks were evaluated in ways depending on the discipline in question. In some cases, external reviewers were asked to comment on the quality of the work. In other cases, the participants made their own judgments based on comparisons with existent frameworks within their disciplines.
The third goal was development of assessment strategies for each discipline. The effectiveness of these strategies and instruments was again evaluated in differing ways for each of the participating disciplines, but each involved some form of field-testing to verify its utility and viability.
The project resulted in the creation and maintenance of a consortium of institutions within a state university system. Not only did it yield many insights into the operation of such a system, but the coming together of faculty from different disciplines and institutions for the sake of a common endeavor produced in the participants an unanticipated sense of renewal and stimulation. Each disciplinary team developed an assessment-related conceptual framework that not only produced an assessment instrument as a direct result of the team activities, but one which can be used in the future to develop other measures.
Major Insights and Lessons Learned
When faculty joined the consortium, they believed that they were about to engage in a team effort to construct comprehensive examinations based on the knowledge they already possessed of teaching and learning in their discipline. Instead, they found themselves participating in an arduous intellectual exercise of questioning and rethinking not only matters related to their discipline, but the nature of teaching itself. As a result, participants became passionately committed to the work of the teams, and faculty development turned out to be one of the project's primary benefits.
Those involved in organizing the consortium concluded that the following strategies are essential to the success of projects such as this one:
Faculty must be given as much autonomy and freedom from bureaucratic interference as possible.
Participants must be selected according to their commitment to the project; their knowledge of the discipline; and their ability to communicate, to work within a team, and to provide leadership in their own departments.
Faculty must be given control over the data that the project will yield. This allays fears that assessment results might be used to harm individuals or departments, and allows the work to proceed within the norms of scholarly inquiry.
A retreat-like setting stimulates informal discussion and interaction.
Occasional interdisciplinary meetings should be held, since contact among individuals from different disciplines fosters creativity.
An adequate stipend (in this case, roughly the equivalent of remuneration for one summer course) not only allows administrative control over minimum compliance but, most importantly, conveys a sense that the work is valued.
The performance of each team is largely dependent on its coordinator; therefore, these individuals need extensive training and support.
The leadership and commitment of central administration are essential to the success of projects such as this one.
Unanticipated ProblemsAs many who have engaged in assessment know, it is difficult to involve students in the process unless it is mandatory and minimum performance levels are specified. Unfortunately, not only does such an approach restrict the kinds and extent of skills and values examined, but it also may proscribe creative or experimental approaches.
On the other hand, the traditional, more limited assessment measures yield information that is only of marginal value for improving teaching and learning. Yet assessment instruments should not be used to judge students or curricula unless their validity and reliability have been amply demonstrated--something that is of course more difficult to establish with locally-developed measures. The project yielded no solution to this dilemma, although participants emerged with a conviction that assessment is of great value in teaching and learning.
When conducted in the ways outlined above, assessment is not only labor-intensive but expensive as well. Thus, despite its unquestionable success in fomenting discussion of profound academic issues, it is not certain that the activities of this project will continue.
The project produced a number of papers and reports. Chief among these are the Final Project Report, as well as a report on each participating discipline. Furthermore, several papers based on the work of their groups have been presented at professional organization meetings by project participants. Information about them can be obtained by contacting the project director or the group coordinators listed below, all of whom are faculty at SUNY/Plattsburgh in the departments of the discipline they represented. The specific reports are as follows:
Biology Group Report, prepared by Bonnie Seidel-Rogol
Foreign Language & Literature Report, prepared by Craig Sample
History Group Report, prepared by Douglas Skopp
Teacher Education Group Report, prepared by Nick Stupiansky
Psychology Group Report, prepared by Henry Morlock
Copies of all these reports can be obtained by contacting the Project Director:
E. Thomas Moran
Vice President for Academic Affairs
Plattsburgh, NY 12901