Internet Learning Volume 3, Number 2, Fall 2014 | Page 60

Using Early Warning Signs to Predict Academic Risk in Interactive, Blended Teaching Environments
by one of the authors) that permits the instructor
to pose non-multiple-choice ConcepTests
to students (e.g., sketch a graph)
and then can use student responses to automatically
group students for discussion.
Peer Instruction Self-Efficacy Instrument
To measure self-efficacy, both in general
and in a Peer Instruction environment,
we developed a set of
25 Likert-scale items aimed at measuring
various qualities related to self-efficacy,
including qualities that we believed would
be unique to a Peer Instruction environment.
These items were based on Fencl
and Scheel’s (2003) Sources of Self-efficacy
in Science COurses (SOSESC). The
statements, such as “When I come across a
tough physics problem, I work at it until I
solve it” were designed to gather data about
students’ self-reported beliefs their abilities
in physics and in a Peer Instruction environment.
As this was the first time the instrument
was used, this study simultaneously
served as an opportunity to use measurements
from this instrument as covariates
as well as an opportunity to gather some
initial validation data from the study. We
later extracted two subscales that we used
as variables in the study. The first subscale
was a seven-item set that conceptually covered
general self-efficacy; Cronbach’s coefficient
alpha reliability for this subscale was
0.85 when the scale was administered at the
beginning of the semester (pretest) and 0.83
when it was administered at the end of the
semester (posttest). The second subscale
was a six-item set that conceptualized our
notion of “Peer Instruction self-efficacy.”
Unsurprisingly, since the notion of self-efficacy
in a Peer Instruction environment is
a new concept, this subscale proved to be
somewhat less reliable, with coefficient alpha
values of 0.53 for the pretest and 0.68
for the posttest. The fifteen items used in
these subscales (as well as the other ten
items that were ultimately not used in this
analysis) appear in Appendix A.
Data Set
Our data set included all performance
data for students over the
semester, including:
• Summative assessment data collected
over the semester, including scores on
problem sets (eight over the course of
the semester), three midterm exams,
and the final exam.
• Pre and post-test scores on the Conceptual
Survey of Electricity and
Magnetism (Maloney, O’Kuma,
Hieggelke, & Van Heuvelen, 2001), a
conceptual inventory measuring understanding
of fundamental concepts
in electricity and magnetism.
• Pretest and posttest data from a noncognitive
assessment, developed by
the authors, to measure students’
self-efficacy in a Peer Instruction
environment as well as attitudes towards
science and education. Seven
items measured general self-efficacy;
Cronbach’s coefficient alpha reliability
for this subscale was 0.85 for the
pretest and 0.83 for the posttest. Eight
items measured Peer Instruction
self-efficacy; this subscale proved to
be somewhat less reliable, with coefficient
alpha values of 0.66 for the
pretest and 0.73 for the posttest.
• Formative assessment data consisting
of student responses to ConcepTests
asked by the instructor in class. These
59