Using Natural Language Processing to gain additional insights from Course Comments
Course Evaluations are easily biased
Before going farther, I have to acknowledge that course evaluations are easily biased.
Things that influence course rating include
What grade the student got (Felton et al. 2004)
the perceived race of the professor (Reid 2010)
How attractive the professor is (Rosen 2017)
Whether cookies were available during class (Hessler et al. 2018)
The weather the day the student completed the evaluation (Braga et al. 2011)
Additionally, all data I am going to show is aggregated at the course level before comparison, so lots of averages, just a limitation of how I got the data
Given that, What can we learn from course evaluations?
I collected 1300+ course evaluaiotns
Spread out over the last 5 years across 8 different departments at Northwestern University
includes over 28,000 unique course comments
Overall rating of course
Overall rating of instruction
How much [the student] learned
the reason for taking the course
interest in the course at the start
Some things made sense
As the instructor rating improved, so did the course rating (r-squared = 0.74)
As the amount the student learned increases, so does course rating (r-squared = 0.64)
Same with student interest going into the course (r-squared = 0.29)
Surprisingly, Course requirement does not affect interest
Everyone likes to complain about be required to take a course, however the data does not support this, it doesn't really matter (r2 = 0.01)
Note that the points are aggregated at the course level, so it is the percent of students who were required to take the class on the x axis
What about the comments?
We can use Natural Language Processing (NLP) to algorithmically analyze course comments and to try to gain additional insights in how to improve our teaching
Once programmed, you can easily expand and repeat the same processing pipeline
You can process a large volume of data quickly (for instance the 28,000+ comments analyzed here)
Because the program is more agnostic of human assumptions, it can lead to unexpected conculsions
Easier to correct bias
While not free of bias, it can be easier to correct bias in an algorithm compared to changing the bias of a human being
Benefits of NLP
Comment Polarity captures how positive or negative a comment is
Polarity is calculated based off of movie ratings on a scale of -1 (super negative) to +1 (super positive)
We can algorithmically create comment topics using LDA, which can be compared to see if people talking about a certain topic is a good or bad sign.
The algorithm came out with 6 topics listed below
Metrics of NLP
So what are the topics?
The first topic consists of talking about the work associated with the course. This topic is dominated by words like exam, homework, material, and quiz.
The second topic is about the subject matter, words like course, learn, and the department or subject name (dept_name)
Professor - Personal
This topic is defined by the "prof_name" tag which means the comment mentioned the professor by name.
Professor - Formal
This topic is defined by refering to the professor and the course, but in a more general, more formal sense.
This topic is defined by talking about how difficult the course is. Importantly, this topic includes comments about the course being easy and hard.
This topic is defined by comments on how interesting or enjoyable the course is.
What can comments tell us?
Polarity aligns with course rating
As expected, the more positive the average comment on a course is, the higher the course rating is.
(R-squared = 0.58)
Remember that this is aggregated at the course level so the comment polarity is the mean polarity of all comments for a given course
The most common topic relates to course rating
When grouped by what comment topic was most common, some clear trends emerge
Mentioning the professor's name is the best option
Talking about the professor and the subject are also good
In comparison, if the comments focus on the difficulty of the course, that's a bad sign
So what does it all mean?
Whether or not a class is required does not affect students interest in the class
When designing a major, think about what students need to learn and don't worry about making those classes required
Comment polarity nicely correlates with course rating
Not surprising but confirms that comments and ratings align
Comments that mention a professor by name are the best, so try to establish a personal connection.
Similarly think about the difficulty of the course, too easy or too hard will lead to a lower course rating.
This project was completed as part of the Searle Teaching as Research program
Special thanks to Rob Linsenmeier and Lauren Woods for advising me during this project
All citations are linked above