Instructional Design – Evaluation Tools (John Stormes)

Subscriber price: $160.00, Non-subscriber price: $200.00
Estimated total study time: 6 hours 10 minutes [Enroll now]

When evaluation tools are used properly for quality assurance and continuous improvement of training programs, they provide a powerful resource for keeping the programs linked to evolving organizational goals. This course shows how to develop and use evaluation tools for a variety of purposes during development and implementation of any kind of instructional program. Practice in preparing an evaluation plan and using evaluation instruments is included.

The following text is recommended for supplemental reading: Donald J. Ford, Bottom-Line Training. Houston: Gulf Publishing,1999.

1. Introduction to Measurement & Organizational Framework

Evaluation is an often undervalued process of instructional design. The purpose of this first lesson is to help you see the integral role evaluation plays in bringing about the most appropriate training for your purposes. Evaluation linked to the goals, development, and implementation of your training delivers dividends by reducing overall cost, improving instructional effectiveness, and extending the value received from your design efforts. This lesson provides the perspective of these roles of evaluation through a review of evaluation, evaluation goals in an organizational framework, and why evaluation of training is important to the organization.

(Estimated study time: 49 minutes)

2. Evaluation Instruments, Data, and Use

This lesson introduces sources of data and methods for collecting evaluation data, leading to an explanation of evaluation planning with examples of evaluation plans.

An important part of conducting successful evaluations is to have a clear idea of both the information you need and where you are going to find it. This lesson introduces you to the most commonly used sources of information and the kinds of instruments you will need to collect reliable and valid information. The lesson concludes with practice in sketching out an evaluation plan you can use in your own work.

(Estimated study time: 1 hour 0 minutes)

3. Writing a Questionnaire

This and the following lesson show how to develop a questionnaire designed to collect primarily qualitative data, the principles of collecting data with a questionnaire, and an exercise in identifying applications for questionnaires in evaluation.

(Estimated study time: 28 minutes)

4. Data Collection Practices

Like preparing a questionnaire, deciding how to collect data for your needs assessment or evaluations, requires some careful planning. Just as the questionnaire needs to be worded so the respondent clearly understands and does not feel pressure to give one reply rather than another, so the way you select people to be interviewed or tested and who you select must be done carefully to obtain reliable and valid results. In this lesson, you will learn some of the concepts involved in setting up a data collection plan and some recommendations both for sample selection and standard evaluation questionnaires.

(Estimated study time: 1 hour 10 minutes)

5. Developing Fair Criterion-Referenced Tests

Some people do not want to admit it, but testing is a pivotal part of training. The instructional designer must be very knowledgeable about tests, both in how to prepare them and how to use them for the learning level of evaluation (level 2). This module helps you achieve the following objectives:
  1. Be able to develop test items that are valid and appropriate for both the test taker and for evaluation purposes.
  2. Be able to create a performance test.
  3. Be able to explain the technical considerations involved in developing and administering tests.
An important issue explained in the following lessons is the difference between tests used for measuring the test taker's ability to perform the work and tests for comparing the abilities of test takers with each other.

(Estimated study time: 39 minutes)

6. Test Validation Concerns & Methodology

Tests of performance and tests of knowledge for performance-based training are only as good as their ability to distinguish correctly between learners who have mastered the knowledge and skills being taught and those who have not. Therefore, we need to have some ways to demonstrate that the tests we create actually achieve that goal.

In this lesson, you will be introduced to important principles and shown examples of techniques used to make certain a test measures what it was intended to measure.

(Estimated study time: 39 minutes)

7. Data Analysis & Interpretation for Evaluation

The remaining lessons of Achieving Instructional Goals with Evaluation Tools introduce you to the kinds of measurement employed in evaluation data collection, how they affect data analysis, and how the standards of statistics apply. The major concepts involved include the kinds of numbers encountered in the behavioral sciences, statistics used to describe the data, and statistical methods involved in analyzing the data.

(Estimated study time: 33 minutes)

8. Descriptive Statistics

This lesson describes the statistics for different kinds of measurement: ratio and interval data in the first section and ordinal and nominal data in the second. These descriptive statistics include: the mean (average), standard deviation, standard error, proportion, the median, and the mode.

(Estimated study time: 31 minutes)

9. Inferential Statistics Overview

The statistics used to describe a collection of data, such as a mean, standard deviation, etc., give us a numerical way to compare evaluation results from one treatment to another. They do not by themselves tell us anything about whether differences from one treatment to another are significant or important.

For that reason, if you conduct evaluation studies, you need to learn the main concepts in inferential statistics and how they help you stake out possible meaning for the data you have so carefully collected.

(Estimated study time: 21 minutes)