Internet Learning Volume 3, Number 1, Spring 2014 | Page 39

Internet Learning
Data Coding and Analysis
To satisfy the first and second objectives
of this study, data were collected from
the three online courses and analyzed separately
with Winsteps--a Windows-based
software that assists with several Rasch
model applications--particularly in the areas
of educational testing, attitude surveys
and rating scale analysis (Linacre, J. M.,
2009).
To address the third objective of
this project the resulting data were treated.
Students’ results were converted into a
measure that is comparable to the reviewers’
rating. Student responses of to a great
extent “4” or to a very great extent “5” are
used as at or above 85% level and coded as
“1”. Student responses of to a moderate extent
“3”, to some extent “2” and to little or no
extent “1” are used as below 85% level and
coded as “0”. According to the majority
rule principle, if 2/3 of the students selects
to a great extent “4” or to a very great extent
“5” for an item in the survey then it is
determined that the course meets that specific
standard from a student’s perspective.
See Tables 1, 2, and 3.
Three QM certified peer reviewers
reviewed the three courses according to
QM standards and input their scores into a
spreadsheet. If a standard was met, “1” was
recorded for the standard. If a standard was
not met, “0” was recorded for the standard.
If two (2/3) of the peer reviewers assigned a
score to a specific standard, then it was determined
that the course met the standard
from a peer reviewer’s perspective. See Tables
1, 2, and 3.
The data were treated in a spreadsheet
and analyzed with SPSS. A nonparametric
Mann-Whitney U test (2 independent
samples) was used to evaluate median
differences between the two groups (students
and peer reviewers). Although the
peer reviewers were asked to take a student’s
point of view, the two groups were
independent.
Results
Course A
Thirty-five out of the 44 students
completed the course design evaluation
survey with a response rate of 79.55%. The
person reliability was 0.83 and the item reliability
was 0.48.
The item statistics indicate that
Item 1 (MNSQ = 3.31) will need to be revised
and Item 16 (MNSQ = 3.13) will need
to be revised or dropped if the instrument
is used in the future.
The Item Map (Fig. 1) indicates
that there are three tiers regarding course
quality from a student’s persepctive. Tier
I contains the items that students strongly
agreed with, thus indicating the course
met Standards 2.1 (Item 4), 3.1 (Item 14),
3.2 (Item 15), 6.1 (Item 20), 3.3 (Item 7) to
a great extent. Tier II contains items that
students agreed with, which indicates that
course met those standatds to a moderate
extent. Tier III contains items that students
agreed with to some extent, thus indicating
that the course did not meet Standards 7.1
(Item 24), 5.1 (Item 13), and 8.1 (Item 25).
Course B
Eighteen out of the 38 students
completed the course design evaluation
survey with a response rate of 47.37%. The
person reliability was 0.95 and the item reliability
is 0.63.
The item statistics indicate that
Item 14 (MNSQ = 2.29) needs to be revised
if the instrument is used in the future.
The Item Map (Fig. 2) indicates
that there are three tiers regarding course
38