Internet Learning Volume 3, Number 1, Spring 2014 | Page 38

Measuring Online Course Design
Statement of Problem
Research indicates that there are
many factors that can affect online course
quality. Some of these factors include
course design, course delivery, infrastructures,
learning management systems, faculty
readiness, student readiness, etc. Course
design is one of the critical pieces in the
quality control process as it affects course
delivery and the overall success of online
programs. Quality Matters (QM) is a process
and a tool to continuously improve
online course quality (Shattuck, 2012). The
2011-2013 edition of the QM Rubric standards
for higher education includes eight
general categories with 41 specific standards
addressing different aspects of online
course design. Each of the standards is
supported by rigorous independent online/
distance research and designed by a team
of experts in the field of online and blended
learning. A team of three certified QM peer
reviewers review online courses according
to QM annotated standards and provide
constructive feedback to course developers.
Although QM peer reviewers are asked
to assume a student’s point of view when
reviewing online courses there exists the
potential for differing perspectives. Therefore,
it is necessary to collect feedback from
students about the course design.
This study attempts to achieve three
objectives. First, it attempts to validate the
instrument design based on QM Standards
to measure online course design. Second,
it attempts to analyze the data and understand
to what degree the selected courses
meet QM standards from a student’s perspective.
Third, it attempts to identify existing
gaps between a student’s perspective
and QM certified reviewers’ perspectives
about QM essential standards.
Method
Instrument
Based upon the QM standards,
an instructional design team developed
a questionnaire that included 27 Likertitems
questions (to little or no extent 1-5 to
a great extent) and three open-ended questions.
Feedback was also obtained from a
professor in the field of research and measurement.
The instrument, simply referred
to as the Online Course Design Evaluation
Tool, specifically focuses on the design aspect
of online courses.
Data Collection
Student Data
Since fall 2011, the Online Course
Design Evaluation Tool has been used at
the university to collect feedback from students
about design aspects of online courses.
The project team identified three online
courses for this project. One course was offered
in fall 2011 and 35 students completed
the survey and two courses were offered
in spring 2012 whereby 18 students completed
the survey in the first course and 20
students in the second course.
Reviewer Data
Three QM certified reviewers who
were trained to review online courses from
a student’s point of view collected data
and provided reports on each of the three
courses; however, because this particular
review was not an official review, none of
the reviewers were subject matter experts
in the field of study of these courses.
37