scott-graham-5fNmWej4tAA-unsplash.jpg

COURSE EVALUATIONS

Using Natural Language Processing to gain additional insights from Course Comments

Scroll Down

Screenshot 2020-12-01 165155.png

Prefer to have me explain the study?

I presented this work as part of the Searle Teaching as Research program, a recording of which you can see on the left

red_signal_light.png

WARNING

Course Evaluations are easily biased

Before going farther, I have to acknowledge that course evaluations are easily biased.

Things that influence course rating include

Additionally, all data I am going to show is aggregated at the course level before comparison, so lots of averages, just a limitation of how I got the data

red_signal_light.png

Given that, What can we learn from course evaluations?

star pie chart.png

I collected 1300+ course evaluaiotns

  • Spread out over the last 5 years across 8 different departments at Northwestern University

  • includes over 28,000 unique course comments

  • Ratings include:

    • Overall rating of course

    • Overall rating of instruction​

    • How much [the student] learned

    • the reason for taking the course

    • interest in the course at the start

Some things made sense

  • As the instructor rating improved, so did the course rating                          (r-squared = 0.74)

  • As the amount the student learned increases, so does course rating     (r-squared = 0.64)

  • Same with student interest going into the course                                     (r-squared = 0.29)

Very Low

Very High

interest vs majmin.png

Surprisingly, Course requirement does not affect interest

Everyone likes to complain about be required to take a course, however the data does not support this, it doesn't really matter (r2 = 0.01)

Note that the points are aggregated at the course level, so it is the percent of students who were required to take the class on the x axis

What about the comments?

We can use Natural Language Processing (NLP) to algorithmically analyze course comments and to try to gain additional insights in how to improve our teaching

Repeatable

Once programmed, you can easily expand and repeat the same processing pipeline

Volume

You can process a large volume of data quickly (for instance the 28,000+ comments analyzed here)

unexpected ideas

Because the program is more agnostic of human assumptions, it can lead to unexpected conculsions

Easier to correct bias

While not free of bias, it can be easier to correct bias in an algorithm compared to changing the bias of a human being

Benefits of NLP

Comment Polarity

Comment Polarity captures how positive or negative a comment is

Polarity is calculated based off of movie ratings on a scale of -1 (super negative) to +1 (super positive)

Comment Topic

We can algorithmically create comment topics using LDA, which can be compared to see if people talking about a certain topic is a good or bad sign.

The algorithm came out with 6 topics listed below

Metrics of NLP

So what are the topics?

coursework.png

Coursework

The first topic consists of talking about the work associated with the course.  This topic is dominated by words like exam, homework, material, and quiz. 

subject.png

Subject

The second topic is about the subject matter, words like course, learn, and the department or subject name (dept_name) 

prof personal.png

Professor - Personal

This topic is defined by the "prof_name" tag which means the comment mentioned the professor by name. 

prof formal.png

Professor - Formal

This topic is defined by refering to the professor and the course, but in a more general, more formal sense. 

dificulty.png

Dificulty

This topic is defined by talking about how difficult the course is.  Importantly, this topic includes comments about the course being easy and hard. 

interest.png

Interest

This topic is defined by comments on how interesting or enjoyable the course is. 

What can comments tell us?

Polarity aligns with course rating

As expected, the more positive the average comment on a course is, the higher the course rating is.

(R-squared = 0.58)

Remember that this is aggregated at the course level so the comment polarity is the mean polarity of all comments for a given course

The most common topic relates to course rating

  • When grouped by what comment topic was most common, some clear trends emerge

    • Mentioning the professor's name is the best option​

    • Talking about the professor and the subject are also good

    • In comparison, if the comments focus on the difficulty of the course, that's a bad sign

topc vs rating.png

So what does it all mean?

 
interest vs requirement.png

Requirement

Whether or not a class is required does not affect students interest in the class

When designing a major, think about what students need to learn and don't worry about making those classes required

polarity vs rating.png

Polarity

Comment polarity nicely correlates with course rating

Not surprising but confirms that comments and ratings align

primary topic vs rating.png

Topics

Comments that mention a professor by name are the best, so try to establish a personal connection. 

 

Similarly think about the difficulty of the course, too easy or too hard will lead to a lower course rating.

This project was completed as part of the Searle Teaching as Research program

Special thanks to Rob Linsenmeier and Lauren Woods for advising me during this project

All citations are linked above