Internet Learning Volume 3, Number 2, Fall 2014 | Page 61

Many Shades of MOOC's
Results
questions were administered using
Learning Catalytics and consisted of
a mix of constructed-response and
multiple-choice questions.
The predictive models are shown in
Table 1, along with the R2 value and
the root mean squared error (RMSE)
for each; this latter value gives roughly the
“expected error” from using the model to
predict final exam score given the predictors.
Model 1 demonstrates that just
knowing students’ conceptual understanding
at the beginning of the semester is surprisingly
predictive of their final course
grades, with 29% of variance explained and
a RMSE of 6.9. Adding in knowledge of students’
self-efficacy at the beginning of the
semester (Model 2) adds significantly to the
model, raising R2 to 34%. The coefficient
for CSEM score is (unsurprisingly) positive
in Model 1 but remains positive in Model
2, indicating that conceptual understanding
at the beginning of the course is positively
associated with final grade even among students
with the same level of self-efficacy.
Model 3 indicates that Peer Instruction
self-efficacy does not add to the predictive
quality of the model above and beyond
CSEM score and general self-efficacy.
(Surprisingly, Peer Instruction self-efficacy
did not correlate at all with final grade; r
= 0.13, p > 0.05.) However, Models 4 and
5 demonstrate that by adding early indicators
of student performance it is possible to
substantially increase the predictive quality
of the model. Model 4 adds as an indicator
the number of Learning Catalytics questions
(ConcepTests) answered correctly in
the first three weeks of instruction, while
Model 5 replaces that with students’ average
scores on their first two problem sets, which
also occur within the first three weeks of instruction.
Since Model 5 is a stronger predictor
of final grades than Model 4, early problem
set scores are retained in the later models.
Models 6-8 add in successive scores on the
three midterms. Not surprisingly—at least
in part because midterm scores are a significant
part of students’ final grades—the
addition of each midterm to the model substantially
increases the model’s predictive
quality. We include these last three models
in part because of the impact on the coefficient
for self-efficacy: it decreases upon addition
of each midterm exam score to the
model, eventually becoming non-significant.
This suggests that over the course of
the semester, students’ self-efficacy—which
begins the semester simply as a thought
process—starts to crystallize into better
or worse performance; students’ midterm
grades essentially are likely accounting for
students’ prior self-efficacy. A similar pattern
is evident with students’ CSEM scores,
which may be the result of the same sort of
process: students’ background knowledge
about the subject domain starts to show up
strongly in their exam performance.
Finally, Table 2 shows two models
that regress final grades on gender and (in
the second model) self-efficacy at the start
of the course. These analyses show that male
students had course grades that were on average
almost 5 points higher than those of
female students, but that the difference becomes
statistically insignificant when controlling
for self-efficacy.
Discussion
Our first set of analyses demonstrate
that it is possible to use a simple
set of early measures, content and
non-content related—accessible within the
first three weeks of the semester—to predict
60