Dr Charles Dziuban | Student Ratings: Skin in the Game and the Three-Body Problem

An ongoing concern in higher education is how to include the student voice in teaching. Dr Charles Dziuban has dedicated much of his career to examining student and faculty outcomes as well as gauging the impact of online, blended and lecture capture courses in universities in an effort to improve the educational experience for students. On the basis of his work, he now presents a model for effective teaching and evaluation based on capturing the student voice.

The Importance of Hearing the Student Voice

Most professional educators agree that including the student voice in education will improve effectiveness, better accommodate our diverse student population, and demonstrate how universities can effectively respond to rapid societal changes.

But how do we ‘hear’ the student voice? Traditionally, students have provided feedback about their learning experience at the end of their courses in a process formalised by the university administration aiming to obtain formative feedback for instructors and summative information for faculty evaluation and also lend credibility to the student voice. However, in recent years, students have reported an absence of tangible impacts arising from rating courses. As such, they have no ‘skin in the game’ because they perceive that their opinions do not impact change in the instructional practice.

This led to the evolution of a second channel for the student voice: an alternative, informal, uncontrolled, and virtual student evaluation of their courses and instructors. Students make their opinions available through social media, meaning that faculty reputations are created in the alternative evaluation universe and spread widely. This channel for student feedback challenges the formal systems developed by universities as it is further reaching than the traditional on-campus approach.

Dr Charles Dziuban and his colleagues use the term ‘skin in the game’ to convey how students have no real investment in end-of-course ratings. In higher education, students assume more responsibility when they are actively engaged in their learning process, knowing that their efforts directly impact their futures. Meanwhile, educators who are committed to student success will make every effort to provide quality education and create a nurturing and supportive network that results in prepared and motivated graduates.

From a student’s perspective, the time and effort taken to complete course evaluations do not affect the course or the professor, compounded by an absence of psychological contracts between faculty and students about how an evaluation system will function. All parties concerned are suspect of the metrics provided by these data, and university administrators are skittish about high-stake decisions based on the evaluations.

The Three-body Problem

Another issue relates to student ratings in the context of the three-body problem: predicting the motion of three bodies under common gravitational forces. Although this is a physics problem, the issue clarifies the understanding of students’ evaluation because it parallels the complex dynamics of effectiveness in higher education. The challenge for both physics and education lies in their mutual complexity and the difficulty of obtaining exact solutions because of uncertainty and unpredictability.

The three fundamental issues that underlie the three-body problem are:

1) Interaction complexity: The culture of higher education involves complex interactions among students, instructors, curriculum, and course content.

2) Inherent unpredictability: The interaction of student ratings with such things as teaching style, student engagement, overall experience, and individual student dispositions typifies a complex system. Addressing this unpredictability is key to understanding the student voice.

3) Positive feedback loops: Student ratings experienced a sustained positive feedback loop reinforcing the system.

The three-body problem analogy to student ratings presents an open-ended challenge – no general solution exists because initial starting points are best guesses. The challenge taken on by Dr Dziuban and colleagues is to devise entrepreneurial approaches that lead to satisfactory solutions. This requires innovation, creativity, critical thinking, and trial and error. Embracing this uncertainty, ambiguity, and ambivalence can result in a sustainable and effective system for the assessment of teaching and learning from the student’s perspective.

A Possible Explanation

Dr Dziuban and colleagues used the end-of-course Student Perception of Instruction at the University of Central Florida, which contained 2,171,565 observations. The results were analysed with the objective of ascertaining the differential impact of course modality, college, department, course level, class decile, and pre-, during, and post-COVID timeframes.

The student rating process on university campuses is a good example of a complex system – for example, one can never predict how an intervention such as moving the rating system to online administration may have broader impacts. The first step was to calculate the total scores – and this is when an important anomaly was found. It appeared from the data that patterns of zero variance had emerged. Zero variance means that all the values in a data set are the same or there is no difference between the data values. Dr Dziuban and his team called this phenomenon ‘straight lining’ and followed up by checking the additional total scores. This side effect atomised the focus of the study by creating an emergence encountered in complex systems where the interactions are more meaningful than the individual components.

In the subsequent analysis, the researchers found that two-thirds of students were not engaged meaningfully in the evaluation of their courses – in other words, they demonstrated that they have no skin in the game with the straight-line response pattern. In focus groups, students reinforced that they did not see the impact of their responses. And while students expressed their feelings on social media, they were reluctant to express them in the formalised university-driven system.

An Idealised Cognitive Teaching Evaluation Model

Dr Dziuban and his colleagues present their concept of an effective and supportive teaching evaluation system in contemporary universities. This represents a seismic shift in the culture of higher education whereby change can emerge through the following:

Critically, the researchers propose that the interplay of these elements will establish a caring and supportive teaching network, fostering an educational community of practice that emphasises cooperation and the promotion of an environment for personal and professional growth. In such a university, a supportive teaching network would flourish, uniting faculty, students, and administration in a shared vision for academic excellence.