The Development and Evaluation of an Adaptable Computer Aided Instruction (CAI) Program
for Acquiring Problem Solving Skills in Biochemistry on the WWW: The "BioChem Thinker"
B. Hershkovitz
Hadassah School of Medicine, The Hebrew University, Jerusalem, Israel. Published on JAMIA (Journal of American Medical Informatics Association) Oct 1997 AMIA page 708.
3. DESCRIPTION OF THE PROGRAM.
5. STAGES IN THE DEVELOPMENT OF THE BIOCHEM THINKER.
6. THE INTEGRATION OF THE PROGRAM IN THE CURRICULUM.
7. METHODS FOR EVALUATION OF THE PROGRAM.
9. DISCUSSION AND CONCLUSIONS.
BioChem Thinker is a CAI program that was developed to enhance problem solving skills and ability to integrate knowledge in biochemistry for medical and dental students. The program runs on a WWW browser. BioChem Thinker is adaptable, it enables the teacher to create a new problem solving assignment, or edit existing assignments without in-depth knowledge of computer programming. This provides teachers with greater independence and flexibility so as to be able to adapt the program to their own course requirements. The program was implemented and evaluated in the 3rd year biochemistry course of The Hebrew University-Hadassah Medical School. The tool used to develop Biochem Thinker can be utilized to develop similar CAI in other biomedical areas.
The study of biochemistry in a traditional medical school curriculum is achieved in two phases. Initially the student studies detailed biochemical pathways most often in a large group lecture format. Small group learning experiences are provided, a variety of educational techniques may be utilized, most frequently clinical case presentations serve as the mechanism by which integration of the knowledge provided in first phase of the course is accomplished. This step is essential if students are to understand the complex interactions, disorders and be capable of solving clinical problems.
Often course coordinators are confronted with situations which may limit the effectiveness of the small group session, lack of resources or the variation in the teaching effectiveness of the small group tutors may negatively affect the educational experience. This may result in a situation where students are at risk of not learning how to apply the knowledge they have acquired. Failure to develop this important skill can have serious repercussions for students both in terms of the course work that follows, and ultimately in their ability to be solve problems in the clinical setting.
The Biochem Thinker (1) was developed in order to overcome the resource related problems, and to provide students with self-directed learning opportunities that enable them to acquire the skills necessary for the integration and improve their understanding and ability to solve biochemical clinical problems.
A number of studies have been reported in the literature which have suggested that Computer Aided Instruction (CAI) programs frequently do not live up to their expectations(2),(3). The problems most frequently identified in with regard to CAI that guided the development of BioChem Thinker were: 1) programs that are not simulations or problem oriented clinical programs are not cost effective. 2) CAI programs lack flexibility and as such are limited in value when it comes to dealing with the uncertainties of clinical practice. Data in the programs can not be updated, or create new tasks or simulations independently on the developers of the program. 3) CAI programs have generally not been integrated into the medical school curriculum and examinations. 4) There are compatibility problems between the advanced programs and the "old" non upgraded computer laboratories. 5) CAI programs tend to be costly to develop and/or purchase. 6) Problems in the evaluation of the direct benefit of the CAI program are frequently difficult to document. In order to overcome some of these disadvantages CAI developers have attempted to design programs that have a distinctive advantage over more traditional teaching methods, in terms of both their academic and financial benefits (4),(5),(6),(7).
The program Biochem Thinker is a CAI which aims to assist students understand and integrate knowledge of biochemistry, and improve their ability to solve clinically oriented problems. The program has a number of different simulations, the task of the student is to define the status of the different biochemistry path-ways in the different organs according to the presented clinical case. At the end of each simulation the student is provided with immediate feedback relating to his/her decisions and explanations. The program is an open learning environment, it enables the teacher to update existing assignments or create new assignments without in depth knowledge of computer programming. The program also provides students with the ability to create their own assignments to enhance further their knowledge and problem solving skills. Students can practice the simulations directly through the internet by using a standard Internet browser such as NetScape or Microsoft Explorer (see example at http://www.md.huji.ac.il/BiochemThinker) or from the hard disk of the computer with the Biochem Thinker program.
Biochem Thinker was subjected to a rigorous pilot test, information about the effectiveness of the program, how it could be best integrated into the Biochemistry course and the degree to which students found the program user friendly was sought. Students were requested to answer a questionnaire about their personal experience with the program. A major objective was to obtain information that would help make the program more user friendly. Based on the information acquired modifications were made to the program and it was included as a component of the Biochemistry course given to students the following year.
The program was subjected to evaluation by students who used it on a formal basis for the first time. The major purpose was to determine the degree to which the program contributed to the ability of students to understand, integrate and solve biochemistry clinical problems.
The Biochem Thinker is a program that helps the student in thinking, integration and understanding biochemistry processes in different clinical conditions.
The program has two components: 1) Run Mode. 2) Edit Mode.
1) The Run Mode.
In the Run Mode the student chooses a clinical case such as diabetes, long fasting, short fasting, well fed state, glycogen storage disease etc'. Students are presented with the details of a clinical case they are the requested to define the amount of activity of each biochemistry path-way in the different organs (Liver, Blood, Adiposite and Muscle) and the level of the metabolites in the blood relevant to the clinical case presented (Figure 1).
Students may request feedback relating to their decisions (i.e. settings) at any point in the exercise. The program provides the student with feedback automatically, if the decisions made after the first two evaluations are incorrect. If the student continues to make incorrect decisions after the third evaluation the program automatically marks the wrong settings. The student is then able to request feedback concerning his/her decisions with regard to the settings selected.
Students can practice assignment through the internet with one
of the standard web browsers NetScape or Microsoft Explorer with a
Macromedia's ShockWave Plug-In(8) that is sharable,
or from a hard disk using the Biochem Thinker application.
2) The Edit Mode (Assignment Generator).
The fact that the most of the biochemistry pathways and connections in the different organs are represented schematically in one screen enables the teacher to create an infinite number of new assignments. The teacher can edit existing assignments and create new ones without any knowledge of computer programming (Figure 2). To create a new assignment the teacher has to describe the clinical case, for example, a diabetic person with glycosuria and keto-acidosis after a meal that contained sugar. This is followed by setting the activity status of each biochemistry path-way in the different organs and the level of the metabolites in the blood according to the clinical case in the path-way's specific "ID card". The teacher inputs the feedback that will be provided to the student in the event that incorrect settings are selected, and the information that will be provided as a part of the evaluation process. The assignments are saved and can be exported as simple text files. The teacher can continue the assignment editing with a simple word processor (Figure 3). The text file can be imported to the program to update the assignments. These text files can be transferred between universities in order to expand the assignments pool by sharing the assignments between the universities. Learning assignments from other university can be re-edited and imported to the Biochem Thinker program.
To create an assignment that runs on the web, the text file is imported to a Macromedia Director file that contains the run mode abilities of the Biochem Thinker and resaved in a "dcr format". The file is loaded on the server and the simulation can run from the web.
The Assignment Editor was written using SuperCard and can run on any Macintosh computer using system 7.1 or later. The Assignments are saved as simple Text files. The "Run Mode" program can run on the internet by a web browser such as NetScape or Microsoft Explorer using Macromedia's shared ShockWave Plug-In .
STAGES IN THE DEVELOPMENT OF THE BIOCHEM THINKER
The program was developed in three steps: 1) Development of a general tool, the Thinking Tool, that will be the basis for creation of new programs in biomedicine subjects that can be presented as a flow chart (flow and connections of path ways or processes). A set of switches and bars that represent the different path-ways and level of metabolites can be used to determine the amount of activity of each path-way and the level of each metabolite. 2) Implementation of the Thinking Tool on the Biochem Thinker program. 3) Assignments were written by the course coordinator, a senior scientist in the medical school, using the "Assignment Editor".
THE INTEGRATION OF THE PROGRAM IN THE CURRICULUM
The objective of the program is to provide students with a self-directed learning experience improve their ability to solve clinical problems in biochemistry. The program is best used once the student has acquired basic knowledge of the biochemistry pathways. Students were given the opportunity to use the program towards the end of the course. Prior to using the program students were provided with a series of guiding questions and reading in order to review their knowledge about the biochemistry pathways. Students were also given a detailed orientation and explanation about how to use the program .
The students had two guided sessions, each one for 3 hours once a week. At the end of each session there was a discussion about the experience with a biochemistry instructor.
The end of course examination was an open book exam consisting of 11 open-ended questions. Screen prints from the program with integrative questions about them where included. The skills required to answer the examination questions where similar to those that were needed to solve the clinical problems in the program.
METHODS FOR EVALUATION OF THE PROGRAM
The evaluation process was in two steps. The first step evaluated the program after the use of the beta version in the first year (pilot test). Students responded to a questionnaire which evaluated their experience with the program. The second step in the evaluation program was the evaluation of the degree to which BioChem Thinker contributed to the students ability to understand, integrate and solve biochemistry clinical problems. A pre-post design was used to accomplish this.
Student Evaluation of the Program.
A total of 106 students out of. 175 (60.5%) responded to the questionnaire which contained 50 items. Each question was graded on a 1 to 5 point Likert scale. The questions covered the following topics: 1) Prior experience of the students with computers. 2) The degree of agreement between students perceptions of the objectives of the program and the formaly stated program objectives. 3) Students views of how the program is best used. 4) The human engineering and design of the program. 5) The general opinion of students about the program. The information obtained was used to improve the program.
Objective evaluation.
A 175 3rd year students in the faculty of medicine where divided into two parallel groups
(groups "A" & "B").
Students were given a short examination prior the use of the program, and again after two sessions of using the program (six hours of practice). The questions in the pre and post-tests were equivalent with regard to degree of difficulty and the content examined. The test items were designed to evaluate the ability of the student to integrate and solve clinical problems in biochemistry. Each test consisted of 4 clinical cases with a total 10 A-type multiple choice questions to be answered in 10 minutes. The students where also asked to indicate for each questions whether they knew or guessed the answer.
Students were informed that participation in the experiment was voluntary and that their grade on the test would not affect their final mark in the course.
Students who took the pre and post-tests were randomly allocated to one of two groups.
Group "A", took tests numbered 1 and 2 . A student who took test number 1 as a pre-test, took test number 2 as a post-test and the opposite for test 2. Group "B", answered exams number 3 and 4 and the same process was followed as in Group "A".
The first step in the evaluation of the tests was to assure their equivalence. First, Students t-test (non paired) was used to test the equivalence of the tests with in groups (for example test 1 versus test 2 in both pre and post evaluation). Second, grades achieved by students in the pre and post-tests were compared using paired Students t-test for matched groups . Non paired t-tests were used to determine if there was a significant difference in scores for each tests (i.e. test 1 was administered as a pretest to one group of students and as a post-test to another).
Table 1 shows the average grades on tests
numbered 1 and 2 and 3 and 4 before and after the use of the program.
|
Before |
After | ||||||
Avrg Grd |
SD |
n |
Avrg Grd |
SD |
n | ||
|
Group A |
Test 1 |
64.2 |
14.8 |
38 |
82.7 |
14.6 |
29 |
|
Test 2 |
60.3 |
14.2 |
32 |
79.0 |
13.7 |
31 | |
|
Group B |
Test 3 |
67.3 |
16.1 |
37 |
85.1 |
12.5 |
33 |
|
Test 4 |
66.3 |
13.6 |
33 |
84.5 |
13.5 |
35 | |
Table 1
Table 2 shows the non paired t-test for groups "A" and "B" before and after exposure to the program. There was no significant difference in grades between students who took test 1 and 2 in Group "A" or tests 3 and 4 in Group B. These results show that exams 1 and 2 were equivalent (P>0.1), and 3 and 4 were equivalent (P>0.1).
|
Non Paired T test |
Before P Value |
After P Value | |
|
Group A |
Test 1 and 2 |
0.27 |
0.31 |
|
Group B |
Test 3 and 4 |
0.80 |
0.86 |
Table 2
Table 3 shows the results of the paired Students t-test for the pre and post-test for both Groups "A" and "B" (after 6 hours of experience with the program). A significant improvement in performance of students was noted for both Groups (P<0.01).
|
Paired T test |
Before |
After |
P Value |
|
Group A |
Test 1 |
Test 2 |
9.88E-05 |
|
Test 2 |
Test 1 |
2.97E-08 | |
|
Group B |
Test 3 |
Test 4 |
2.38E-13 |
|
Test 4 |
Test 3 |
1.02E-07 |
Table 3
Table 4 shows the results of the non-paired Student t-test for the pre and post tests for each test. A significant improvement in performance was noted for each test (P<0.01).
|
Non Paired T test |
Before |
After |
P Value |
|
Group A |
Test 1 |
Test 1 |
3.09E-06 |
|
Test 2 |
Test 2 |
1.67E-06 | |
|
Group B |
Test 3 |
Test 3 |
2.58E-06 |
|
Test 4 |
Test 4 |
6.33E-07 |
Table 4
The results of the objective evaluation show that there is a significance improvement with the performance of the students after 6 hours of practice with the Biochem Thinker. Feedback given by both the students and faculty supported the objective results of the experiment. The advantages of the Biochem Thinker are: 1) The program improves the integration process and the ability to solve clinical problems in biochemistry. The use of the program was perceived by students to be a very valuable learning experience. In the end of course debriefing students stated that the BioChem Thinker program was "outstanding" and that its use should be expanded. Students commented on the fact that they had actively sought out the program as an aid to preparing for the end of course examination. 2) The program is an open and flexible learning environment . The program enables the teacher to create new assignments or edit the existing assignments without any knowledge in computer programming or dependence on the manufacture. In the world of science and medicine, which is dynamic and changes very fast, the education must be updated frequently. With an instruction tool such as the Biochem Thinker the teacher has maximum flexibility. 3) The program can become a creative tool for the students. They can write assignments by themselves. 4) The assignments can be transferred between universities easily. This contributes to cooperation between universities and expands the pool of the clinical cases. 5) The program is not limited to a specific platform or software. It can run on a PC's or Macs by the standard WWW navigation browsers such as Netscape and Microsoft Explorer. 6) The program is user friendly and simple to operate. 7) Lack of experience in using of computers does not limit the use of the program. 8) The programming skeleton of the Thinking Tool can be used to develop new programs in different subjects. This will lower the development prices and will raise the cost effect of the program. Based on the experience derived with the Biochem Thinker program in biochemistry similar programs are being planned in clinical nephrology, the Nephro Thinker.
I would like to acknowledge the contribution of Professor Oded Meyuhas, the course coordinator for writing the assignments used in BioChem Thinker, Dr. Robert Cohen, Chairman of the Center for Medical Education and Professor Rami Rahamimoff, MD for their help in preparing the paper.
1. URL: http://www.md.huji.ac.il/BiochemThinker
2. Richard Bruce Friedman, MD. Top Ten Reasons the World Wide Web May Fail to Change Medical Education. Acad. Med. 1996;71;979-981.
3. Hooper J, O'Connor J, Cheesmar R, Price C P. Tutorial Software for Clinical Chemistry Incorporating Interactive Multimedia Clinical Cases. Yearbook of Medical Informatics 1966;538-541.
4. Cornelius Rosse, MD, Dsc. The potential of Computerized Representations of Anatomy in the Training of Health Care Providers. Acad Med 1995; 70;499-505.
5. Patel V L. Recent Advances in Computer Technologies and Medical Education. Yearbook of Medical Informatics 1966;521-524.
6. B. Collis, D. Parisi, M.B. Liorio, Adaption of Courses for trans-European tele-learning. J.of Comp. Assis Learn 1996; 12;47-62
7. I Neilson. R Thomas. Designing educatinal software as a re-usable resource. J of Comp Assis Learn 1996; 12; 114-126.
8. URL: http://www.macromedia.com
|
Research and Simulation Programming by Barak Hershkovitz
For more information and feedbacks contact Prof. Oded Meyuhas
|