Evaluation of Effective Interventions to Solve
the Dropout Problem in Adult Distance Education

Dr. Yonnie Chyung
Dr. Donald Winiecki
Ms. Jo Ann Fenner
Instructional and Performance Technology
College of Engineering
Boise State University
1910 University Dr.
Boise, ID 83725
ychyung@boisestate.edu

IPT Homepage: http://ipt.boisestate.edu/

Abstract: This paper describes an evaluation case study conducted in the Instructional & Performance Technology (IPT) department at Boise State University (BSU). The IPT department offers a Master of Science degree both traditionally through on-campus coursework, and totally through asynchronous, computer-mediated distance education (DE). This case study describes interventions that the department used to reduce the drop out rate of students in its DE program. The dropout rate during 1989 and 1996 was 44%. The IPT DE program designed and implemented interventions to reduce the dropout rate during the three semesters in 1997. Within a year, a significantly positive result was obtained as a result of their interventions. The purpose of this paper is to help the audience understand how to analyze, design, implement, and evaluate interventions to reduce high dropout rates in asynchronous, computer-mediated distance education. Several instructional design models and an evaluation model such as the ARCS model, the Organizational Element Model (OEM), and Kirkpatrick's training program evaluation are discussed in the paper.

1. Introduction

1.1 Adult Education

The U.S. Department of Education defines adult education as the teaching of adults via any education activities, except full-time enrollment in higher education credential programs. According to the Digest of Education Statistics published by the U.S. Department of Education (1997), the number of adult education participants among 117,826,000 employed persons during 1996-1997 was 59,734,000.

What motivates adults to be involved in continuous formal education? Houle (1971) conducted a qualitative study in which he identified three types of adult learners: goal-oriented participants, activity-oriented participants, and learning-oriented participants (cited in McCreary, 1990). Verduin and Clark (1991) categorize three main types of adult education programs: adult basic education (ABE) programs (to acquire ABE), leisure and enrichment education programs (to increase enrichment in adult life), and career education programs (to prepare or upgrade their job-related knowledge and skills). Examples of adult education activities include part-time college attendance, classes or seminars given by employers, classes taken for adult literacy purposes, adult basic education or English as a second language, or courses for recreation and enjoyment.

1.2 Adult Distance Education

Distance education is defined as "any formal approach to learning in which a majority of the instruction occurs while educator and learner are at a distance from one another" (Verduin & Clark, 1991, p.8). Distance education, due to its time and geographic flexibility, has appealed to adult learners who work full-time yet want seek continuous education. Many adult learners attempt to achieve their goal of adult learning via distance learning options. Distance education institutions use various distant learning technologies such as audio and video conferencing devices, the Internet, or computer-mediated communication systems.

According to a survey conducted by the National Center for Education Statistics in 1995, out of about 14.3 million students enrolled in higher education institutions in fall 1994, about 758,640 adult students formally enrolled in distance education courses in the academic year 1994-95. Eighty one percent of institutions reported that they offered courses designed for undergraduate students; thirty four percent for graduate students; and thirteen percent for professional continuing education. Among distance education institutions, 39 percent of them targeted professionals who were seeking recertification, and 49 percent targeted other workers who looked for skill-updating or retraining.

1.3 Adult Distance Education in Boise State University

The Instructional & Performance Technology (IPT) Department at Boise State University offers a Master's degree program via distance education (DE). The IPT master's program is intended to prepare adult learners for careers in the areas of instructional design, job performance improvement, human resources, organizational redesign, training, and training management. The majority of students attend the IPT program not only to earn a master's degree in IPT but also to upgrade their professional knowledge and skills. As a result, the majority of the adult distance learners who enroll in the IPT-DE program may be classified as goal-oriented and learning-oriented learners. The asynchronous computer-mediated communication (CMC) environment that the IPT-DE program utilizes to deliver its DE classes permits interactive and dynamic discussions among the participants and facilitates higher education through collaborative learning experiences.

2. A Problem in Adult Distance Education

Although distance education appeals to adult students who work full-time and seek continuous education, a problem in DE exists as a high attrition rate of students. Retention of DE students is usually lower than retention of on-campus students (Verduin & Clark, 1991). Reasons for dropouts are found to be various. Adult DE students tend to drop out of DE programs when they perceive that their interests and course structure are not matched (Fenner, 1998), that they are not confident enough in learning processes (Chacon-Duque, 1987), and/or that they have achieved what they wanted (Holmberg, 1989 cited in Verduin & Clark, 1991).

The dropout problem in DE programs existed in the IPT-DE program at Boise State University as well. Between the fall semester of 1989 and the fall semester of 1996, 44% of DE students dropped out of the IPT-DE program by their third course (Fenner, 1998).

3. Systematic and Systematic Interventions

In order to solve the dropout problem, it is important to proceed in a systematic and systemic way. First, it is critical to conduct a careful analysis of the causes to the problem, and then to seek the most appropriate solutions to the problem, implement the solutions, and finally to evaluate the effectiveness of the solutions. The ARCS model, the Organizational Element Model (OEM), and Kirkpatrick's evaluation model guide such processes.

First, according to Keller's ARCS model (Keller, 1987), there are four factors that influence the degree of learners' motivation to learn: Attention, relevance, confidence, and satisfaction. Learners lose their motivation to learn and quit learning, especially when they do not perceive the instruction as interesting or relevant to their goal. They also lose motivation to learn when they are not confident in learning processes, and/or they are not satisfied with the instructional processes. Therefore, it is critical to design DE instruction based on the ARCS model in order to prevent DE students from losing motivation to learn, which contributes to the dropout rate.

According to the OEM (Kaufman & Thiagarajan, 1987), there are five elements in a results chain: Organizational inputs, processes, products, outputs, and social outcomes.   Effective instructional inputs and processes, possibly designed based on the ARCS model and delivered by an educational organization, will result in successful learning outcomes, which is one of the educational organization's products. Learners who experience successful learning outcomes will more likely continue to be motivated to learn.  That is, there will be a high retention rate in enrollment, which is one of the educational organization's positive outputs. Outcomes are effects in and for society, which are beyond the organizational efforts and results.

In order to evaluate the effectiveness of the educational organization's inputs and processes, products, and outputs, Kirkpatrick's model of evaluating training programs can be used (Kirkpatrick, 1996). In this model, level 1 evaluation measures learners' reactions to the educational organization's instructional inputs and processes. Since the instruction was designed based on the ARCS model, it is appropriate to evaluate learners' reactions to the instruction according to the four ARCS elements. We measured learners' perceptions of their attention levels during the instruction, relevancy of the instruction to their interests, confidence levels in learning, and satisfaction levels toward the instruction as well as the instructor. Level 2 evaluation measures the educational organization's product: i.e., the learning outcomes. Level 3 evaluation measures the educational organization's output: i.e., whether the dropout rate has decreased or not. A summary of the systematic and systemic interventions and evaluations of the effectiveness of the interventions conducted by the IPT-DE program at BSU is illustrated in Figure 1.

 Figure 1. Systematic and Systemic Interventions and Evaluations

4. Cause Analysis

In order to find out the causes of the dropout problem, the IPT-DE associate program developer conducted interviews with the students who dropped out of the program as well as those who were continuing the program between 1989 and 1996. From the interviews, it was concluded that their satisfaction levels during the first or second courses were the major factor that determined their decisions to continue or not to continue the program. Forty two percent of the students who dropped out of the program expressed "dissatisfaction with the learning environment" as one reason for their dropout. More specific reasons given for dropping out included:

• discrepancies between their professional or personal interests and the course structure
• low confidence levels in distance learning
• doubts in their online communication abilities
• lack of competence in using the DE software as an effective learning tool
• feelings of being overwhelmed or overloaded by advanced knowledge and information
• de-personalized learning environment.

5. Intervention Plan and Implementation

The cause analysis revealed that in order to reduce the dropout rate, it was very critical to help new distance students be satisfied with the instruction and the learning environment, and to guide them to improve their academic performance as well as their confidence levels, especially during the entry-level DE course.  During the three semesters (spring, summer, and fall) in 1997, the instructor of the entry-level DE course constantly evaluated and redesigned the existing instruction (a) to be more attractive to the new DE students, (b) to be more relevant to their professional interests, and (c) to increase individual students' confidence levels in learning as well as (d) their satisfaction levels toward the instruction. The ARCS model and the OEM were used to guide the instructional design and developmental processes. Especially important in accomplishing these goals was the instructor's efforts to help her students develop self-regulated learning skills in the DE environment. In accomplishing this, she used various instructional strategies to help her students become "metacognitively,   motivationally, and  behaviorally active, and  active participants in their own learning process" (Zimmerman, 1989, p.4). The instructional strategies that she used included:

• to administer a preknowledge assessment that measured students' previous knowledge levels at the first week of the semester
• to break down the instruction into small weekly modules and help them master one module at a time
• to inform students of the goals and objectives of the weekly instruction and encourage them to self-evaluate their weekly learning process as well as weekly learning outcomes.
• to provide students with clear criteria of expected performance levels such as regular participation in class discussions
• to deliver instruction via multiple media such as PowerPoint slide shows, the WWW, and electronic bulletin boards when appropriate.
• to provide meaningful examples and analogies to help students learn new concepts
• to modify instruction based on students' background when it is possible and appropriate in order to help them see the instruction as relevant to their professional and personal interests.
• to provide specific help and attention to the students who have low preknowledge levels
• to monitor individual students' performance and provide immediate, frequent, and personalized feedback as well as regular (weekly) feedback on their learning process
• to encourage students to monitor their own learning process
• to give positive reinforcement to students when they collaborated toward achieving the instructional objectives

In order to prevent the DE students from taking advanced courses too early and having unnecessary experiences of failure, the DE associate program developer and the faculty members provided advice to individual DE students, especially regarding the most appropriate sequence of courses for them to take.

6. Evaluation of Reaction, Learning Outcomes, and Dropout Rate

6.1. Reaction (Level 1)

At the end of each of the three semesters in 1997, a course evaluation questionnaire was administered to measure students' reaction to the quality of the entry level DE course and the instructor. The response scale used was A: Outstanding, B: Very Good, C: OK, D: Improvement Needed, and E: Unsatisfactory. As coding their responses, the number one was assigned to a response to 'A'; 2 to 'B'; 3 to 'C'; 4 to 'D'; and 5 to 'E'. The data reflected by average scores revealed that the revised instruction became more attractive to the students, and helped them see more relevance between the academic learning and their professional interests. Students became more confident and satisfied with the quality of the course as well as the quality of the instructor due to the revised instruction across the three semesters (see Table 1).

Table 1. Level 1 (Reaction) Evaluation: Attention, Relevance, Confidence, and Satisfaction Levels

Semester	Attention	Relevance	Confidence	Satisfaction toward Course & Instructor
Spring, 1997	M = 1.41	M = 1.92	M = 1.83	M = 1.92 & 1.42
Summer, 1997	M = 1.17	M = 1.63	M = 1.75	M = 1.50 & 1.00
Fall, 1997	M = 1.25	M = 1.47	M = 1.56	M = 1.31 & 1.06

6.2. Learning Outcomes (Level 2)

In each of the three semesters in 1997, a pretest and a posttest were administered to measure and compare the difference between the pre-knowledge levels and the learning outcomes. T-test analyses revealed that, in each semester, the average posttest score was significantly different from the average pretest score (see Table 2).  It indicates that the entry-level DE students not only had positive reactions toward the course but also learned the subject from the course at a significant level.

Table 2. Average Pretest and Posttest Scores and t-test Results

Semester	Means (Learning Outcomes)	t-test results
Spring, 1997	M (pre) = 24.58, M (post) = 36.83	t (11) = -20.61, p < .01
Summer, 1997	M (pre) = 25.59, M (post) = 34.78	t (16) = -7.64, p < .01
Fall, 1997	M (pre) = 25.38, M (post) = 34.81	t (15) = -9.25, p < .01

6.3. Dropout Rate (Level 3 and 4)

Between the fall semester of 1989 and the fall semester of 1996, the average dropout rate was 44%. At the end of 1997, a year after the interventions were implemented, the dropout rate was cut in half, to 22% (Fenner, 1998). Reducing the dropout rate from 44% to 22% is a significant decrease.

Among these 22% who dropped out of the program in 1997, three students dropped out after the first week of the semester due to hardware and software compatibility problems. Six students cited that they decided not to continue the program because their professional goals and the degree program did not match. Other reasons cited were a health problem and time constraints. No students reported other previous reasons such as low confidence levels in distance learning, doubts in their online communication abilities, lack of competence in using the DE software as an effective learning tool, feelings of being overwhelmed or overloaded by advanced knowledge and information, and de-personalized learning environment. Therefore, it appears that the reduced dropout rate was due to our organization's efforts of eliminating the previous causes.

7. Conclusions and Educational Implications

This evaluation case study reveals how the ARCS model, the OEM, and Kirkpatrick's evaluation model successfully guided the process reducing the dropout problem in the IPT distance education program at Boise State University. The impact was significantly positive in terms of the benefits to the educational organization itself as well as to the learners themselves. In particular:

1. Due to the educational organization's efforts in designing and implementing effective interventions based on the ARCS model and the OEM framework, the DE students perceived the delivered instruction as more interesting and relevant to their professional and personal goals. They were more confident in learning and highly satisfied with the distance learning environment.

2. Due to the effective instructional inputs and processes, DE students achieved significant learning outcomes. Due to the positive reactions to the instructional inputs, learning processes, and successful learning outcomes, they more likely decided to continue to learn via distance education.

3. As a result of the effective interventions described above, the educational organization significantly decreased its dropout rate.

It is more cost-efficient to decrease dropout rates to maintain the number of enrollment instead of trying to recruit new students. This case study revealed the positive effects of a systematic and systemic analysis, design, development, implementation, and evaluation of distance education programs based on effective instructional design models and an evaluation model.

8. References