Learning Objects Assessment Plan
Given the expense involved in the production of learning objects, it is critical that we take time to evaluate the benefits that this teaching method brings to the classroom learning experience. Plasnning of the assessment is aided by following an Evaluation Plan Template.
Our assessments are based on five potential sets of data: student background information provided by students, student usage logs, student performance on exams, student and faculty surveys, and targeted focus groups of students and faculty. Just a few or all of them may be used to evaluate the instance of use of any one learning object.
Student Usage Logs
Usage of learning objects is tracked with WebTrends, an http logging and reporting application. WebTrends analyzes the Apache server logs and produces reports that provide detailed information about when users are on the sites, how often they return, how much time they spend with a learning object, where they are coming from, and what type of system/software they are using to access the websites. It also has the ability to compare and analyze reports against each other.
Student and Faculty Surveys
Surveys of students and faculty are administered at a point in the semester after the learning object has been used and student learning evaluated by the instructor, but before the end of the semester. The surveys are based on the nationally-recognized Teaching, Learning, and Technology Group's Flashlight protocols. We ask participants to rate the usefulness of the object, and what they liked and disliked about it. Here's a generic student survey example.
Faculty surveys allow faculty to rate the learning object in terms of what they think it added to their class, and also collect information about how the learning object was used by the faculty member. Successful outcomes are likely based not only on student learning styles, but also on faculty teaching styles. In particular, how well the learning object was integrated into the course and the extent to which its use was justified to the students as a necessary activity or resource both play an important role in student acceptance and the degree to which they credit the material as beneficial.
Targeted Interviews and Focus Groups
Analysis of the exam and survey results allow us to identify particularly successful as well as unsuccessful implementations of the learning objects. These courses can then be targeted for further study using focus groups of students and interviews with faculty to identify what factors contributed (or did not contribute) to success. Again, we rely on TLTGroup's protocols for interviews and focus groups.
This evaluation results help us both in improving the design of individual objects, and also in identifying trends to guide future development efforts. What types of objects tend to work best? In what instructional context does this method of teaching produce the most improvements? Where should we concentrate our development efforts? How can we document the successes in order to encourage adoption of improved educational practices?
We recognize that the assessment plan varies by learning object and course. Our assessment plan is designed so that it can be tailored to the needs of each specific instance of use.