A Survey of Faculty Opinions Concerning Student Evaluations of Teaching
Michael H. Birnbaum
California State University, Fullerton
Student evaluations of teaching were originally intended as devices to help measure the quality of instruction, for the purpose of improving the quality of education, but they may be causing more harm than good.  Because procedures used in research on student evaluations are confounded, research has not been able to demonstrate that student evaluations are valid measures of teaching quality.  However, the real effect of student evaluations depends not on their actual validity but rather on their perceived properties.  
	Because retention, tenure, promotion, and PSSI salary raises are influenced by student evaluations, faculty members will make changes in their courses that they believe will improve their evaluations.  This article explores causal theories held by members of the faculty concerning student evaluations; in particular, it explores instructors' theories of how changes in their grading standards and content of courses would affect student evaluations and student learning.
	An email request to complete a survey was sent to all members of the faculty of C.S.U.F.  The survey was completed (via Internet) by 208 faculty members.  The majority judged that student learning can be improved by increasing course content and by raising standards for grading.  However, they also stated that these improvements would hurt their evaluations.  The majority judged that the current system of tenure and promotion discourages raising standards, encourages lowering of standards, and promotes "watering down" of course content.  
	The data appear consistent with the hypothesis that most faculty believe that ratings will be lowered by changes that would improve learning, and that the effect of student evaluations of teaching is to decrease the quality of education.  Faculty theories are compared to students' judgments of their own policies.  The majority of students state that they would give highest ratings to courses with the least content and the lowest standards.  Apparently, the majority of faculty members understand the majority of student opinion.
Debate on Validity of Student Evaluations
	A recent issue of American Psychologist featured the controversy on the validity of student evaluations of teaching (d'Apollonia & Abrami, 1997; Greenwald, 1997; Greenwald & Gillmore, 1997; Marsh & Roche, 1997; McKeachie, 1997).  These authors noted that there are many possible interpretations of student ratings because procedures used in research on this topic are confounded.  Because different instructors teach different content, use different standards, wear different clothes, give different exams and tell different jokes to different groups of students who are not randomly assigned to classes, it is not possible to confirm or refute opposing theories of student evaluations.  D'Apollonia and Abrami (1997) performed a meta-analysis of studies that used different sections of the same class in which evaluations were correlated with measures of student learning.  They concluded that only 10% of the variance of student evaluations is related to valid measures of learning, and the data permit rejection of the statistical hypothesis that as much as one-sixth of the variance of student evaluations is validly related to educational performance.  Some consider student evaluations to be so complicated that anyone who might dare to use them for practical purposes must be familiar with nonlinear, nonadditive, multidimensional modeling of confounded judgment data (Marsh & Roche, 1997; McKeachie, 1997).
My own research specialty is the study of nonadditive, nonlinear models of human judgment, so I suppose these authors would consider me qualified to interpret student evaluation data.  In judgment research that uses similar techniques as those used in student evaluations, one can show that the number 450 is judged to be a significantly "bigger" number than 550 (Birnbaum, 1974; Parducci, 1968; Parducci, 1995).  Using procedures even more closely related to those of student evaluations than I used in 1974, I recently showed that the number 9 is judged to be a significantly "bigger" number than 221 (Birnbaum, 1999).  Since most of us know that 9 is actually smaller than 221, we should be wary of conclusions when procedures that lead to bizarre conclusions are applied to important, practical questions.  If 9 were really larger than 221, we might feel more confident in the conclusion that we should promote teacher A and fire teacher B because A gets higher ratings than B.  If we doubt that 9 is really "bigger" than 221, we should be wary of the conclusion that A is a better teacher than B simply because A gets higher ratings.  
Aside from the issue of the actual validity of student evaluations is a potentially more important question, namely, their perceived validity.  Although some teachers are fired because of student evaluations, most figure out how to get better evaluations.  The effect of student evaluations depends less on their actual validity than it does on theories that teachers hold of how changes they can make in their teaching would affect their evaluations.  Because student evaluations are used as input to tenure, promotion, and salary decisions, teachers will act in what they believe to be their self-interest, so they will make changes that they think will get higher ratings. Therefore, I surveyed our faculty to find out what changes teachers believe would improve or hurt their ratings.
Survey of Faculty
	During the last two weeks of the Spring Semester, 1998, I sent messages by email to teaching faculty inviting them to participate in a survey on student evaluations.  Within one month, 208 members of the faculty of CSUF completed the survey via the Internet.  (The email was also forwarded to faculty at other universities around the world; those data are still coming in and will be the subject of a separate analysis).
	In the CSUF sample were 105 Full professors with tenure, 31 Associate Professors with tenure, 30 untenured Assistant and Associate professors, and 38 lecturers.  There were 76 faculty members with less than 12 years experience teaching, 66 with 12 to 24 years, and 64 with more than 24 years of experience.
Results of the Faculty Survey
Faculty Theories of Student Evaluations.
If you were to RAISE standards for grades in your class, would it affect your student evaluations?  Nearly two-thirds of those surveyed (65.4%* or 136) reported that higher standards would result in lower evaluations against only 3.4% (7) who thought the opposite would occur, and the others stated it would make no difference.  
	 (*Throughout this paper, asterisks will be used to indicate that the split is statistically significant, with  = .05, by a two-tailed, binomial sign test against the hypothesis that the probability of the response and its opposite are equally likely.  The statistical result in the above case is equivalent to the probability of tossing 143 coins and finding that 136 are heads and 7 tails.  This event has a standard deviate of z = 10.79, which corresponds to a miniscule probability (|z| > 1.96 would be significant); therefore, we reject the hypothesis that this split occurred by chance).
	Two different theories have been proposed to explain why students may give higher evaluations to professors who give them higher grades (apart from any change in learning).  One theory is that students like to receive good grades and they like people who give them what they want.  The other is that students attribute their high grades to the success of the teacher in educating them.  According to this theory, students have no way to know if their knowledge of the material qualifies them to undertake the next course in the sequence; therefore, students mistakenly believe that high grades mean that they have mastered the content.  A third theory proposed in which higher standards might produce higher ratings is that high standards can frighten off students who don't really want to take the class, who enroll in other sections, leaving more motivated and positive students behind.
	If you were to INCREASE the amount of CONTENT (material) 
in your classes, would it affect student evaluations?  About two-thirds (65.9%*) responded that increasing content would decrease student evaluations, against 4.8% who thought the opposite would occur, and the rest thought it would make no difference.  One theory proposed to explain why evaluations are lower with more content is that with less content, the student (not knowing what the course should have covered) believes that the instructor was very successful in teaching the whole subject.  With greater content, students will have to struggle to learn the material, and will blame the teacher for doing such a poor job of organizing and presenting this material.  Because students do not know the amount of content that should have been included in the course, they will not know that important material has been left out of a low content class until later in life, long after the evaluations have been made.  The other theory proposed is that unmotivated students prefer less work to more work.
	Are student evaluations influenced by such variables as the teacher's personality, attractiveness, gender, race, dress, religion, ethnicity, sexual orientation, or disability status?  In response to this question, only 16.8% responded that student ratings are "unbiased;" 52,4%* responded that students are biased in favor of certain groups, and 26%* responded that students are biased against certain groups.  Most of these factors are beyond the control of the faculty member.  
	Before and during this research program, I have heard many anecdotes of various forms of bigotry expressed by students in anonymous evaluations.  One woman told me she quit teaching because a class of female students wrote personal attacks concerning her appearance; for example, one student wrote that she could improve her class by "getting a new face."  After that, she could not bear to stand before a class, knowing that students were thinking such thoughts.  Apparently, the use of anonymous evaluation forms encourages some students to express prejudices that they might be less likely to express on a signed form.  Apparently, some faculty have even seen terrorist threats on evaluations, threats of rape and of other personal attacks.  If the evaluations were not anonymous, perhaps fewer of these harassing evaluations would be made.  Gayle Vogt (1992) noted that if the evaluations were not anonymous, then a faculty member could sue for libel when false information is put in writing that might affect a teacher's career.
Theories of student learning
	The questionnaire defined student learning as "knowledge of the subject matter, as might be measured by objective, standardized exams...the sum of knowledge and skills that the student retains from the class and will be able to use in the future..."
	How would increasing the content covered in class and in assigned readings affect student learning?  45.2%* said that increasing content would increase student learning compared to 27.9% who thought that decreasing content would increase student learning. The most common theory mentioned was that by presenting more content, students will learn more.  An argument proposed for decreasing content is a variation of Gresham's law; namely, if one instructor has more content than others, students will switch sections to instructors who cover less content, resulting in lower student learning.  
	How would raising standards for grading affect student learning?  57.2%* responded that raising standards would increase student learning against only 7.7% who indicated the opposite effect.  The theory most often expressed was that students will work to achieve a certain grade.  If less is required of students to pass, then students ease off in their studies, so they will learn and retain less in the future.
The Incentive Structure and Student Evaluations
	Does the current system of promotion and tenure give incentives to RAISE standards for grading?  A surprisingly high 92.3%* stated "no" compared to only 5.8% who said "yes."
	Does the current system of promotion and tenure encourage faculty to LOWER their standards?  70.2%* said "yes" against 28.8% who said "no."
	Does the use of student evaluations encourage faculty to "WATER DOWN" content in their courses?  72.1%* said "yes" against 26.9% who said "no." 
	Thus, the majority opinion of the faculty is that the incentive system for tenure and promotion causes faculty to lower standards and water down courses, which most faculty members believe will decrease student learning.  Apparently, the majority of faculty believe that the incentive system has the opposite effect of what a citizen in favor of quality education would support.
Trends over Time
	Over the years, have you changed the amount of material presented in your classes?  48.6%* say that they now present less material against 14.9% who say that they present more material, and the rest indicate no change.
	Over the years, have you changed the standards required to get a passing grade in your classes?  32.2%* say that they now use lower standards against 7.2% who say that they now use higher standards, and the rest report no change.
	Since the majority opinion is that reductions in content and standards are harmful to student learning, it seems sad that so many faculty concede, even on a confidential questionnaire, making these changes that they believe reduce the quality of education.  It would be interesting to ask faculty if making these changes has injured their morale.
Student Preparation and Competence of Graduates
	The questionnaire asked, Please assess the preparation of students who are now enrolled in your college or university, compared to previous years.  The majority (67.3%* or 140*) reported that students are not as well prepared now compared to previous years, compared to only 2.4% (5) who judged students to be better prepared.  When asked what percentage of lower division students possess the study skills one should expect of the top 1/3 of high school graduates, the median response was 40%, with 85 responses below 30% and 134 (64%*) less than or equal to 50%.  Apparently, about two-thirds of the faculty think that half or more of our students do not qualify under the state's concept for admission to our university.
	One theory held by faculty is that declining qualifications and increasing grade inflation among recent new teachers is a cause of this problem.  Based on data published each semester at CSUF, one finds that students who plan to be teachers (e.g., majors in Child Development) have some of the highest grade point averages (GPAs) on the campus.  When asked if students with the highest GPAs are indeed the best students, only 12.7%* thought these "future teachers" are our best students; about twice as many rated these students as below average on the campus, and 55.8% judged them average.  When asked, what percentage of undergraduates who want to be teachers do you think should become teachers, nearly two thirds (63%*) of respondents said that half or fewer than half should become teachers.  Perhaps students who received inflated grades in college give inflated grades when they become teachers, leading to further declines in standards.
	When asked, what percentage of graduates in your department possess the general education, specific skills, and knowledge base that should be required of a graduate...," the median response was 60%.  Thus, the average faculty member believes that two out of every five of our graduates are not qualified to receive the degrees we confer upon them!  The requirements for receiving a degree do not ensure that the graduate is competent, in the opinion of the faculty. Apparently, incentives to lower grading standards result in the "production" of unqualified graduates who don't deserve their degrees.  It would be interesting to ask faculty, should degrees awarded in this system be accredited?
A Summary of Comments and Opinions
	I received many comments and messages by email, some with suggestions for revisions and extensions of the questionnaire for future studies.  I received many criticisms of student evaluations and their misuse in promotion and tenure decisions.  One person informed me of a multiple section experiment conducted over several years in Engineering. This study showed that a teacher who consistently got the lowest exam performance (students in all sections took the same exams) consistently got the highest student evaluations. 
A representative sample of comments, grouped roughly by topic, follows:
	"Student evaluations are a form of blackmail of untenured faculty." "Student evaluations are very biased as they are used to 'get back' at teachers." "Good teaching has little to do with good evals. I try to achieve both, but it's a stretch.""I think the student evaluations need serious reevaluation and revamping." "Do students evaluate how much they 'like' the course or how much they 'learn?'" "Evals measure 'satisfaction' not 'performance'; measures not validated." "Students are not qualified to assess a teachers ability until they have completed their work."  "Most students don't take them seriously.  They rush through them."
	"I take evals seriously. Sometimes get good suggestions." "Most students provide fail (sic) and reasonable evaluations of teaching."
	"Based on my experience, by far the biggest of these (biases) is personality. None (are unbiased)" " Small class size and interactive methods are most important!"  
	" higher standards=lower enrollments but raised evaluations; FTE pressure is the problem." "Raising standards makes more students drop out, resulting in decreased learning for them."  "I think more content makes more people drop the class." "(I'm) retiring partly because of lower standards."
	"Students today tend to be consumers, not producers, and expect to be given their educ." "Student ability to understand basic concepts has definitely decreased over the years." "Students in all areas are less prepared.  I don't believe this problem is unique to Teacher Educ."
	"Student evals are given too much weight & not enough to other types of teaching assessment" "Student evaluations should be optional and not used for promotion/tenure decisions." "Student opinion surveys do not give the teacher feedback on how to improve, or on student learning." "I object to the anonymity of student evaluations as well."
"Student evals tend to be one of the major obj. (sic) criteria used to evaluate teachers."
"I think the system of evaluation and tenure at CSUF 'actively 'encourages dumming down in courses.  " "Mid Semester Evals returned to faculty only would help quality of the teaching...Overall evals r dum." "Student evals at our shop have very poor psychometric properties but get heavy use for merit and t&p"
	"Student evaluations lower faculty morale, as faculty must cater to student expectations."
	"the 'survey' is a little too obvious."  "I think I see where your are going with this and I fully agree with what appears your direction." "While basic, this is a good first start!"
A Survey of Students
	In order to investigate the nonadditive, nonlinear, and multidimensional nature of student evaluation policies, I conducted a survey similar to one that David Perkins did several years ago.  I asked 142 lower division students to evaluate 89 hypothetical classes, based on factorial variations of three variables: the instructor's personality and individual characteristics, standards for grading in the course, and the amount of content the instructor presents (Birnbaum, 1998).  The 142 students represented 29 different majors from Accountancy, Art, and Biochemistry to Sociology, T.V. Broadcasting, and Theater Arts; there were also 26 with undeclared majors.  I anticipated that this heterogeneous mix of students would hold a variety of different views of what would be the optimal class.  
	However, to my surprise, the students were remarkably homogeneous in their evaluations of courses:
	• 94.4% (134* of 142) gave higher evaluations to an "attractive, well-dressed, 36 year old female with a nice personality" than to a "62 year old male with a slight tremor (due to a previous stroke) who doesn't smile in class."
	• 92.3% (131*) gave higher ratings to a class with "light" content (less than 100 pages to read in a semester, and nothing else to do outside of class) than to a course with "heavy" content (800 pages to read and homework assignments); only 9 gave highest ratings to courses with the most content.  Only 16.9% (24) rated a "medium" level of content as better than the "light" level, although the "medium" course was described as having "300 pages of medium level reading" to do in the semester, and the course might require 'some study' to master the material."  Surprisingly, students were not hesitant to say that they prefer "lite" education.
	• 97.9% (139* of 142) gave higher ratings to a course with "very easy" standards than to a course with "very hard" standards.  Only 14 (9.8%) students gave their highest ratings to a course with "medium-easy" or "medium-hard" standards.  
	The "very easy" standards course was described as follows: "This instructor gives most students As and Bs, even those who are struggling with the material or who have not been diligent in attendance and study.  Only the most clueless student will get a C in this class.  If a person has half a brain and attends some of the time, they will pass with an A or a B." "Medium-easy" was described as a course in which most students get As and Bs, there are a few Cs, and an occasional D or F.  "Medium-hard" was a class with 30% As and Bs, 50% Cs, and 20% Ds and Fs.  The "very hard" level was described as a couse in which 7% of the students get As, 13% Bs, 40% Cs, 25% Ds, and 15% Fail.  
	Apparently, students are not hesitant to say they will give higher evaluations to courses with lower standards and less content.  Students gave the highest ratings to the course in which the teacher is attractive and has a good personality, where the standards for grading are lowest, and where the content is least.  Apparently, the majority of faculty are correct in describing how to improve their student evaluations.  Yet this change also lowers student performance, according to the faculty.  Thus, according to the faculty, we have created a system in which changes destructive to education are encouraged.
Conclusion: Student Evaluations May Harm Education
	The evidence reviewed here indicates that many faculty believe that the incentive system (using student evaluations for promotion and tenure decisions) puts them in a conflict of interest between making changes that would improve student learning and making changes that would improve student evaluations.  When teachers are put in a conflict between the good of the people and self-interest, they should not be blamed by those who created the conflict if they protect their self-interests. 
	It has been pointed out to me that an assumption implicit in collecting student evaluations is that the average student is more likely right than the professor.  This assumption may not be warranted, and the very idea that a professor should redesign a course on the basis of anonymous student comments may be demoralizing to those who reject the notion that students who have not yet finished a class should decide its proper content or how it should be taught.  Should a professor follow suggestions of students, made anonymously during the stressful week before the final exam?  Should we revise courses based on opinions of students who have not completed the first course in a subject?  Can students know what the content of a course should have been before they have completed the next course in the sequence? Wouldn't students' evaluations be more accurate after they have completed their degrees and worked in society for a few years?
	I have run into some of our former students who now work as waitresses or at service counters at copy shops.  These people spontaneously tell me that they now realize that they did not understand what was important when they were in school.  They now have different attitudes and values, and they now wish they'd devoted their time in college to learning something.  Which attitudes should govern the university, the attitudes of freshmen, or the attitudes of those who have finished their degrees and tested their skills in the real world?
	Many students are inaccurate in describing what the teacher said in class when they are motivated to be as accurate as possible (when taking exams); therefore, is it reasonable to assume that these same students are accurate when they give evaluative descriptions of the class on an anonymous form when there is no incentive to be accurate and no penalty for libel? 
	The potential injury to morale produced by student evaluations should be measured in future surveys.  If professors feel that they are being forced to "give in" and do mediocre work, then it will likely injure their morale.  Furthermore, those who refuse to lower their standards may have their morale further injured by losing their jobs or being refused promotions because they resisted lowering their standards.  
	Those who do not care about students may have higher morale in this system because it is easier to teach less content and give undeserved high grades than it is to uphold high standards and teach students.  Those who can adopt faith in the doctrine that "the student is always right" may also be able to retain high morale, since they can say that they are doing is consistent with their faith.  These different moral philosophies and their correlations with morale should be studied in future surveys. 
	The finding that the average member of our faculty thinks that only 60% of our graduates have the education indicative of their degrees is a sign that our institution is in trouble.  Our incentive system and shrinking budget have apparently combined to produce a decline in standards that has produced inferior education.   At present, students are motivated to get good grades, and faculty are motivated to get good evaluations.  Unfortunately, these two interests can be satisfied by reductions in content and grading standards, which diminish the quality of education.  
	If students were given honest feedback concerning their performance, they would be concerned.  If those 40% who are unqualified to receive their degrees were denied their degrees, then the people of the state would notice that there is a problem in the educational system.  However, by giving grades that are convenient rather than honest, we prevent corrections from being made to a sick system.  We should begin to study how our incentive system can be changed to align the interests of students, faculty, and the people of the state.
References
	d'Apollonia, S., & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist, 52(11), 1198-1208.
	Birnbaum, M. H. (1974). Using contextual effects to derive psychophysical scales. Perception & Psychophysics, 15, 89-96.
	Birnbaum, M. H. (1998).  The effects of course content, grading standards, and instructor's personality on student evaluations.  Unpublished report.
	Birnbaum, M. H. (1999). How to show that 9 > 221: Collect judgments in a between-subjects design. Psychological Methods, 4(3), 243-249.
	Greenwald, A. G. (1997). Validity concerns and usefulness of student ratings of instruction. American Psychologist, 52(11), 1182-1186.
	Greenwald, A. G., & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52(11), 1209-1217.
	Marsh, H. W., & Roche, L. A. (1997). Making students' evaluations of teaching effectiveness effective. American Psychologist, 52, 1187-1197.
	McKeachie, W. J. (1997). Student ratings: The validity of use. American Psychologist, 52(11), 1218-1225.
	Parducci, A. (1968). The relativism of absolute judgment. Scientific American, 219, 84-90.
	Parducci, A. (1995). Happiness, pleasure, and judgment. Mahwah, NJ: Lawrence Erlbaum Associates.  
	Vogt, G. H. (1992).  The mask of anonymity.  Senate Forum, 7 (1 & 2),
24-26.
Copyright Michael H. Birnbaum, August, 1998, All rights reserved.