Mastering Computing Technology: A New Approach for Non-Computer Science Majors

Mark Urban-Lurain (urban@cse.msu.edu)
Donald J. Weinshank (weinshan@cse.msu.edu)
Department of Computer Science & Engineering
Michigan State University

Presented April 20, 1999 at the Annual Meeting of the American Educational Association, Division C, Section 7

Purpose

Designing instruction to provide "computing literacy" for non-Computer Science students is a challenge faced by computer science departments in many colleges and universities. (Joyce, 1998; Kolesar & Allan, 1995; Townsend, 1998) To meet this challenge, we conducted a series of interviews with the Chairs of 67 departments on our campus to determine the needs of their students. Some themes we identified are:

Literacy is context-dependent. Literacy differs for K-12 teachers, business majors and biology researchers.

Literacy does not require computer programming. As computers and applications have become more powerful, programming knowledge has become less important for non-computer science majors.

Literacy requires performance-based assessment. Literacy does not consist of abstract knowledge alone. The literate user is able to accomplish various tasks in some context. Thus, the best assessment requires the learner to demonstrate literacy in an authentic manner.

Literacy requires adaptability. Regardless of the context, merely learning to perform a fixed set of tasks using specific software packages does not constitute literacy. Furthermore, rapid changes in computing environments, incoming student experiences, and the needs and expectations of client departments mandate continuous revision of the curriculum.

These themes led us to create a fixed course structure with continuously changing course content. The structure provides feedback for assessing students and revising the course content to meet the changing demands of client departments, changing student experience, and changing hardware and software environments.

To meet the needs of the different client departments, we designed the course with "core" materials for the first half followed by "tracks" appropriate to various majors. However, computing concepts (e.g., data representation and abstraction) provide the foundation for the diverse application content.

This paper focuses on the key component of this system: performance-based assessment. We summarize how our assessments are designed and how the data from the assessments are used to improve student learning and revise the course.

Perspective and Method

A major limitation of many introductory courses is that assessment and evaluation depend on surrogate measures, such as exams and quizzes that test static recall. To address these problems, our course:

Is entirely lab-based (Herrmann & Popyack, 1995; Kay, 1998). It consists of lab exercises that build on concepts from the readings and homework.

Replaces competitive grading with a collaborative learning model (Johnson, Johnson, & Holubec, 1990; Johnson, Johnson, & Smith, 1991). This constructivist approach to learning reflects the way that technology is used in most workplace settings.

Replaces "points" with a series of mastery learning (Block, Efthim, & Burns, 1989; Osin & Lesgold, 1996) bridge tasks (BTs.) At regularly scheduled intervals, each student takes a bridge task (BT) in which s/he must synthesize and apply the concepts and competencies to solve a set of authentic problems.

Bridge tasks (BTs) are evaluated on a mastery pass/fail basis. If a student demonstrates mastery on the first bridge task, he or she "locks in" a minimum grade of 1.0 in the course. For each subsequent BT passed, the student's course grade is incremented by 0.5 until she or he has passed the 3.0 BT. In the strict sense, this is a modified mastery model in that students must complete the five BTs within the confines of a semester (about 12 BT attempts), rather than having an unlimited amount of time.

Once a student passes the 3.0 BT, s/he may choose a semester project to attempt increase the course grade to 3.5 or 4.0.

This assessment model

provides greater opportunity for learning than traditional multiple choice examinations. The students' focus shifts from accumulating points to mastering concepts. Since students repeat failed bridge tasks until they pass, these repeats provide additional opportunities for learning.

actually indicates the concepts and competencies that the student has mastered. In most courses, a grade of 2.0 means that a student has accumulated points from a variety of exams and homework assignments. It does not indicate what knowledge the student has. A 2.0 in this course means that the student has demonstrated mastery of windowing skills, electronic mail, searching the Web, manipulating files and directory structures, and sufficient proficiency with word-processing to write a term paper.

is non-competitive. Since students' grades are criterion-referenced rather than norm-referenced and are based on individual performance rather than group grading, the model facilitates collaborative learning and fosters cooperation and peer-teaching in the classroom.

Data Sources

Because of large enrollments (1800 students per semester) and resource constraints (two full time faculty plus Teaching Assistants) cost-effectiveness is a major design consideration. Performance-based assessments are usually more labor-intensive to create, administer and evaluate than traditional multiple choice or computer-based-training exams (Schoenfeld, 1994). To adapt performance-based assessments to a large enrollment course and use resources most effectively we considered several factors:

The BTs must test appropriate concepts.
The BTs must be individualized so that different students receive different BTs and repeated BTs differ.
The evaluation criteria must ensure a high degree of consistency among graders and facilitate efficient grading.
We must track and analyze student performance to verify that the instruction is meeting our design goals.

Each BT consists of multiple dimensions, each of which assesses certain concepts. Within each dimension, there are multiple instances, specific tasks to test the students’ abilities to apply the concepts. Furthermore, each BT contains one or more extension tasks, materials not explicitly covered in the course but to which the students must apply the concepts. These extension tasks test transfer by having students apply the concepts to the solution of new problems.

Each student’s BT is created by randomly selecting a single instance for each dimension. For each instance, there is a detailed set of criteria used to determine mastery of that dimension. Mastery on a BT is determined by combining mandatory and optional measures of mastery on the various dimensions. The BT contents are summarized in Table 1.

Table 1: Bridge Task Contents

Bridge Task	Concepts and Competencies
1.0	E-mail; Web; Distributed files; Help
1.5	Boolean searches of databases; Word-processing; Creating Web pages
2.0	Hardware; Software; More Word-processing
2.5	Spreadsheets (functions, charts)
3.0 Track A	Object embedding; Designing Web sites; Network communication tools
3.0 Track C	Advanced spreadsheets; Importing and transforming data; Data analysis; Add-on tools
3.0 Track D	Advanced spreadsheets; Fiscal analysis and modeling; Add-on tools

Results

This model provides a rich database of student performance that we use to

improve the instructional design of the course. Reviewing BTs from fall 1997, we discovered that searching databases with Boolean operators had the highest failure rate. The instructional revisions we did for the spring course dramatically improved student performance on these dimensions.

refine the bridge tasks. We initially structured the first few days of the course to accommodate a wide variability of incoming computing experiences. However, we discovered that this pacing lulled the students into a false sense of security. We changed both the content and timing of the 1.0 BT to address this problem.

help individual students better understand concepts. Our software gives the students, TAs and instructors access to all student BTs. It displays the formatted text of any BT, the evaluation criteria, the graders’ comments, and the student’s results. This allows the students to identify their areas of weakness when preparing for subsequent BTs.

Table 2 shows two measures of the BT repeat rates for Spring 98. For each BT, the table shows the number of attempts to pass that BT. The data are presented both for the entire class and, separately, as a percentage of the students who eventually passed that BT. Since some students never pass all BTs, the Percent of Class Passing decreases on each successive BT.

For each BT, about 50% of the students pass on their first attempt, about 33% pass on the second and about 10% pass on the third.

There is no significant difference on the Percent of students who passed BT across BTs by single factor ANOVA across BTs (N=1770 students.) We therefore conclude that the BTs are of comparable difficulty.

Table 2: Bridge Task Attempt Rates

	One attempt		Two attempts		Three attempts		Four or more attempts
Bridge Task	Percent of class passing	Percent of students who passed BT	Percent of class passing	Percent of students who passed BT	Percent of class passing	Percent of students who passed BT	Percent of class passing	Percent of students who passed BT
1.0	68%	71%	20%	21%	6%	7%	2%	2%
1.5	40%	43%	36%	39%	12%	13%	5%	5%
2.0	45%	52%	27%	30%	11%	13%	4%	5%
2.5	42%	53%	26%	33%	9%	12%	2%	2%
3.0	28%	46%	24%	40%	9%	14%	0%	0%

Table 3 shows the final course grade distribution from Spring 98 in tabular form. Grades of 3.5 and 4.0 are obtained by doing a semester project only upon successful completion of the 3.0 BT.

Table 3: Course Grades

Final Grade	0.0	1.0	1.5	2.0	2.5	3.0	3.5	4.0
Percent of students	1.5%	3.9%	4.2%	9.3%	18.0%	19.5%	13.8%	29.9%

To those accustomed to a normal distribution, these results may be startling. However, as Bloom, Madaus and Hastings (1981) point out:

If we are effective in our instruction, the distribution of achievement should be very different from the normal curve. In fact, we may even insist that our educational efforts have been unsuccessful to the extent that the distribution of achievement approximates the normal distribution. (p. 52)

We use BT data to revise the instruction and assessment. Based on the analyses of the BT performance statistics from fall 1997 semester, we made extensive revisions of the instruction and BTs before spring 1998. While the previous data show the internal consistency of these changes, student ratings provide an external measure of these changes across semesters.

Table 4 shows some of the Student Instructional Rating System (SIRS) data for the first two semesters we offered this course. Students are asked to respond to each statement on a scale of 1=strongly agree, 2 = agree, 3 = neither agree or disagree, 4 = disagree, 5 = strongly disagree.

Table 4: Student Evaluations

Question	Mean Fall 97	Mean Spring 98
I usually did my homework before coming to class.	3.47	2.23
I learned a lot in the group exercises in class.	3.79	3.15
The Bridge Tasks were a fair test of the material I learned in class.	3.50	2.29
I felt that the "extension tasks" on the Bridge Task, which asked me to do something I had not previously done, were reasonably connected to what I had already learned.	3.68	2.32
The grading was fair for the Bridge Tasks.	4.29	2.98
The grader's comments explained what I did wrong on the Bridge Tasks.	3.51	2.55
I feel that my course grade will reflect my understanding of computer concepts.	3.88	2.80
I would recommend this course to my friends.	3.70	2.48

On every question, the student ratings improved dramatically. The average overall improvement is 1.13 (on a scale of 1 to 5). All results are very highly significant by two-tailed T-tests. We attribute these improvements to the extensive revisions of the course and BTs we made from fall to spring.

Educational Importance

This course has been very well received by our client departments. As the first year’s cohort progresses through the University, we expect that student grades in the course will successfully predict preparedness for subsequent courses in our client departments. This will allow our client departments to concentrate on using computing technology in their domains.

These results should be of interest to educators who are

teaching technology literacy.
teaching introductory computer science for non-computer science students.
applying collaborative learning in large-enrollment courses.
implementing mastery-learning, performance-based assessments in a fixed-semester environment.