Do Notifications Get Noticed? New Study Finds That Embedded Alerts to Students Promote Action at High Rates

This is a joint post by John Whitmer, Daniel Nasiatka and Timothy Harfield.

What kind of effects do embedded nods and nudges have on student behavior?

Providing students with information about how they are performing in a course, and alerting them in advance if they are at risk of not passing a course, has been suggested as a powerful way that learning analytics can be used to improve educational outcomes (Dahlstrom, Brooks & Bichsel, 2014).  Student-facing dashboards are increasingly being built into educational technology applications, providing timely and actionable interventions directly to the people who need them most.  However, there is little empirical research in this area, and the research that has been conducted has been conducted with relatively small numbers of courses/students and has reported mixed results (Aguilar, 2016; Teasley, 2017).

An innovative feature in the Ultra experience of Blackboard Learn is the ability to automate notifications to students on the basis of their performance and activity. These rule-based alerts are built into the LMS experience.  They encourage students to improve when they are struggling, and acknowledge students when they are doing well.  Instead of appearing outside the learning context, they are embedded directly within the course.  When a notification is clicked, a student receives more detailed information as well as a suggested action.

From research with a simulation of this feature previously conducted with researchers from the University of Michigan, we found that low performers self-reported that they were more likely to benefit from this kind of feature compared to students from a higher GPA background.  Now that this feature is available to customers running Blackboard Learn in a SaaS (Software as a Service) environment, we have begun to investigate how students actually respond to these kinds of nudges in authentic learning contexts.

Notification Rules

Looking at an anonymized dataset that included 22,227 notifications displayed to 3,679 students in 141 courses, Blackboard data scientists Daniel Nasiatka and John Whitmer found the following:

1. Students Interact With LMS Notifications at High Rates

An automated alert is like a tree falling in the forest.  If no one notices, then it’s as if nothing happened at all.  As it turns out, students not only see performance notifications in Blackboard Learn Ultra, but they also engage with them at very high rates.

Click rates by Learn Ultra notification type

2. Students Are Interested in Their Performance Relative to Others

Knowing that students are acknowledging alerts is an important first step.  But what kinds of notifications are most likely to be noticed and acted upon?  In their research, Nasiatka and Whitmer found that click rates for notifications about activity relative to others (i.e, you are in the top 10% of the class, or the bottom 5%) were ten points higher (39%) than those for notifications about performance relative to self (i.e., your grades improved, or your grades have fallen).

This makes sense when we consider that information about activity relative to others is not something that students would otherwise have access to (in contrast to personal grade trends, which students can easily monitor themselves).  This finding is also exciting when we consider that students who check their activity relative to others have been found to be 1.5 times more likely to earn a grade of C or higher in research conducted by the University of Maryland, Baltimore County.

3. Students Like Hearing That They Are Doing Well

It doesn’t matter how many times a student hears they are doing well, the novelty never wears off.  Surprisingly, this even holds true for students who are consistently high performers.

Notifying students when they are doing well relative to others serves as an important reward.  It makes us feel good when we do well, and knowing that we are doing well by objective standards encourages us to continue to do well.  In education, we are trained to identify and correct mistakes.  It would be easy to develop a notification engine that focused exclusively on error-correction.  However, there is power in recognizing excellence and rewarding achievement.  By using data to create meaningful rewards, we have the opportunity to foster self-regulated learning and a love for education in a way that can promote student success and lifelong learning at scale.

4. Student Performance Has Momentum

When Nasiatka and Whitmer clustered students by notification type, five basic groups emerged.  Not surprisingly, four of the five groups of students tend to receive mostly one notification type and that remains consistent throughout the semester.  All else being equal, high performers remain high performers, low performers remain low performers, improvers continue to improve, and decliners continue to decline.

Frequency of Student Clusters

This finding has two important implications.  First, knowing that students with declining grades are likely to continue the pattern, a sudden grade decrease early in a semester is an important signal for an early outreach opportunity for instructors and academic advisors.  Second, the ‘Malleable Middle’ contains students that receive a more distributed mix of notifications throughout the semester.  Although they are regularly in the top 10% of the class, they are also frequently in the lowest 5%.  From week to week, they are about twice as likely to see a grade decrease as they are to see an increase.  Unlike other clusters, which see strong momentum in favor of receiving one specific kind of notification, this group sees significantly less consistency in their performance patterns.  Students in this category are not what institutions generally consider to be ‘at risk,’ but their lack of momentum means that they are the most likely to benefit from targeted intervention.  In light of the fact that this is the second largest student cluster, representing 23% of students in the sample, this represents a significant opportunity for institutions to increase their rates of student success.

Proportions of notification types received by student cluster

Read the full research report here: http://www.blackboard.com/resources/pdf/report_ultratriggers.pdf.