THE ADOPTION OF COMPUTER ADAPTIVE TESTING IN EDUCATIONAL ASSESSMENT
ABSTRACT
The purpose of this study was to establish whether Computer Adaptive Testing can be successfully adopted in educational assessment in Kenya by focusing on factors likely to affect its adoption. The study objectives were to determine a) the extent to which adoption of CAT is affected by students and teachers perceptions b) the extent to which adoption of CAT is affected by the existing technological infrastructure; and c) the extent to which adoption of CAT is affected by levels of computer literacy.
The study used a sample selected from public secondary schools in Makueni County through simple random sampling technique. Questionnaires with both open and closed questions were used to collect primary data from the respondents while secondary data was obtained from the Ministry of Education, Makueni County Office. The data was analyzed using the Statistical Package for Social Sciences (SPSS) where Descriptive statistics such as frequency distributions, percentages, means and standard deviations as well as inferential statistics such as Pearson Correlation and ANOVA tests were utilized.
The findings of the study indicate that teachers and students perceptions are positively related with adoption of CAT (0.085) although the relationship is weak; there is significant relationship between ICT infrastructure and adoption of CAT (0.336) and the relationship between teachers and students level of computer literacy and adoption of CAT is negligible (0.049).
TABLE OF CONTENTS DEDICATION iv
ACKNOWLEDGMENT v
TABLE OF CONTENTS vi
LIST OF FIGURES x
LIST OF TABLES xii
LIST OF ACRONYMS xiii
ABSTRACT xiv
CHAPTER ONE 1
INTRODUCTION1
Background1
Statement of the Problem4
Research Questions5
Objectives of the Study6
Research Hypotheses6
Justification of the Study6
Scope and Limitation7
Operational Definitions8
CHAPTER TWO 9
LITERATURE REVIEW AND CONCEPTUAL FRAMEWORK9
Introduction9
The Review9
Theoretical Literature9
Types of Adaptive Testing11
Definition of Computer Adaptive Testing12
Global Perception of Computer Adaptive Testing13
Implementation of e-Assessment in Institutions14
University of Central Florida, USA14
University of Ilorin, Nigeria14
Aga Khan Academy, Mombasa, Kenya15
Advantages of Computerized Adaptive Testing15
Why Migrate to Computer Adaptive Testing?17
Empirical Literature21
Theoretical Framework29
The Theory of Computerized Adaptive Testing29
Item Response Theory (IRT)30
Conceptual Framework32
CHAPTER THREE 33
METHODOLOGY33
Research Design33
Data Types and Sources34
Data Collection34
Pilot Survey34
Target Population and Sample Size35
Data Collection Instruments36
Sampling Procedure39
Data Processing and Analysis40
Data Processing40
Data Analysis Techniques40
CHAPTER FOUR 42
RESULTS AND DISCUSSIONS42
The concept of Adoption42
Background Information42
Computer Adaptive Testing and Teachers’ and Students’ Perception44
I believe I can do a Computer Adaptive Test well45
Computer Adaptive Tests can be fair to all students46
I can take a test using a computer with confidence47
I would like to take a test that gives immediate results48
Taking an examination using a computer can be very interesting49
How do you rate your ability to use computers?50
Do you think your students are capable of doing an examination via the computer?51
Do you think the use of Computer Adaptive Testing in assessment can offer a better .............................................................................................52_Toc50104884
Do you think your students can enjoy taking a Computer Adaptive Test53
examination? 53
Computer Adaptive Testing and Technological Infrastructure54
Lack of reliable educational support software has contributed to slow rate
of ICT integration in this school 54
The poor state of ICT interconnectivity affects its adoption rate in schools55
Low connectivity speed has contributed to slow rate of ICT adoption in this school56
Low internet access has contributed to slow rate of ICT adoption in this school57
How reliable is electrical power for educational support in this school58
How do you rate the speed of internet connectivity in meeting the
teaching learning needs for educational purposes in this school? 59
Criteria used for placing ICT Infrastructure for Educational Activities60
Computer Adaptive Testing and Computer Literacy61
How would you rate your own computer literacy?61
How would you rate your own Internet literacy?62
How would you rate your current typing skills?64
How do you rate the ability of your students to use a computer?65
How Easy or Complex is use of Computer Applications among your students.66
How would you rate the adequacy of computer literacy for your students in67
supporting their learning activities? 67
Pace of CAT Adoption68
CHAPTER FIVE 74
SUMMARY, CONCLUSION AND RECOMMENDATIONS74
Summary74
Conclusion76
Recommendations77
REFERENCES 79
APPENDIX I: LETTER OF INTRODUCTION TO THE RESPONDENTS 82
APPENDIX II: STUDENTS’ QUESTIONNAIRE 83
APPENDIX III: TEACHERS’ QUESTIONNAIRE 86
APPENDIX IV: INTRODUCTION LETTER FROM THE UNIVERSITY 90
APPENDIX V:PERMISSION TO COLLECT DATA 91
APPENDIX VI: RESEARCH PERMIT 93
CHAPTER ONE
INTRODUCTION
Background
Computer Adaptive Testing (CAT) has been defined by Chuesathuchon (2008) as a type of Computer Based Testing (CBT) where a student takes a “customized” or tailored test where after each item is administered, a student’s competence is assessed and the next question is chosen based on the examinees estimated competency level. Weiss & Kingsbury (1984) argue that in Computer Adaptive Testing questions are chosen and delivered to the student via the computer depending on the answers supplied to earlier delivered questions.
Computer Adaptive Testing is now a common mode of test administration and it is slowly becoming a mainstay in the large scale examination in modern society (Tian et.al, 2007). According to Embreston & Reise (2000) &Weiss (2004) Computer Adaptive Testing consists of a set of test items given to an examinee which give optimal information about his/her abilities. In Computer Adaptive Testing, test takers do not have to see exactly the same questions as any other examinee and they don’t have to answer equal number of test items (Weiss & Kingsbury, 1984). The objective of a Computer Adaptive Test is to choose for every student only those questions that most effectively and efficiently measure an examinees capability (Van der Linden et al., 2000).
The widespread adoption of Computer Adaptive Testing in educational assessment has been made practical by two major developments: the use of computers in testing (with improved processing abilities and speed) and Item Response Theory (IRT) procedures and this has led to the increased movement towards Computerized Testing in general and specifically towards the use of Computer Adaptive Testing.
Adaptive Testing is not completely a new idea because its principle can be traced from the work of Alfred Binet in the development of Binet IQ test used in psychological measurement (Davey & Stone, 2011; Weiss, 2004). In a Computer Adaptive Test, a test question is chosen, delivered, the answers scored and the test takers ability determined and the next question is chosen based on the determine ability level of the test taker. If the student gets the question right his/her competence level is increased and it is assumed that the examinee is also able to answer a more difficult question. If the item is wrongly answered, the examinee ability is decreased and an easier question is then chosen and presented (Gathitu, 2010).
Computer Adaptive Testing is seen as a more reliable mode of test administration than the usual Paper and Pencil Testing because it’s only those questions whose level of difficult matches the ability of the student are delivered to the student (Chuesathuchon, 2008). Computer Adaptive Testing aims at assessing the capabilities of every student accurately while at the same time solving the problems encountered under the paper and pencil tests because it is only those questions considered appropriate are selected and administered to the test takers (Da Silva et al., 2017).
Research conducted in the area of Computer Adaptive Testing has been limited, however, those that have been done have stressed that compared to Paper Based Testing, Computer Adaptive Testing is more effective because the test items are adapted to the test takers ability level. Besides its general accessibility and basic statistical analysis of data, the advantages of Computer Adaptive Testing include reliability, validity, fairness and feasibility (Chuesathuchon, 2008). According to Karnjanawasri (2002), Computerized Adaptive Testing improves test reliability, improves test security, it greatly minimizes and controls cases of cheating in tests and greatly cuts on the cost of printing and shipping tests. CAT is also known to be convenient and flexible in scheduling the test, testing can be done anytime, it offers immediate scoring and reporting of test results, reduces test supervision and fewer test items can be used to arrive at a more accurate estimate of test-taker proficiency levels.
According to Ndume et al., (2014), the use of computers in assessment has continued to provide a powerful tool that can be used to design assessment that are able to measure student capabilities that goes beyond the possibilities offered by the traditional assessment methods. The adoption of Information and Communication Technologies has therefore called for the reconsideration, rethinking and remodification of traditional examinations and out of this call; the idea of adopting Computer Adaptive Testing in assessment was born. The widespread use of computers in educational assessment has led to massive improvement in test administration and with the invention of the internet examination bodies and organizations can comfortably administer examinations worldwide with least problems. With the availability of computers in all schools,
Computer Adaptive Testing would present better methods of assessment and solve the problems associated with paper and pencil examinations (Al A’ali, 2007).
Statement of the Problem
The aim of this study was to establish whether Computer Adaptive Testing can be successfully adopted in educational assessment in Kenya by focusing on factors likely to affect its adoption.
The Government of Kenya has an ICT policy that aims at improving the lives of her citizens by making sure that ICT services across the country are accessible, efficient, reliable and affordable. Although technology has penetrated many sectors of the economy such as banks, transport and communication and in the medical field, the Kenyan education sector has not kept the same pace in adopting technology and has therefore lagged behind. According to a study conducted by Kandiri (2012), the use of computers is still an idea that is new and has not fully permeated in Kenyan schools and the study found that the perceptions and experiences of teachers and administrators plays a critical role in the use of computers and general applications of technology in Kenyan classrooms.
Although Kenya has an ICT policy for basic education, nothing much has been done to adopt ICT as an important tool in learning, teaching and assessment. According to UNDP (2015) use of ICT has a positive mark in fostering the goals of education if and only if it is effectively employed. ICT remains the greatest driver for development in both the
economic and technological fronts in the 21st Century, although despite this knowledge, a great deal of Kenyan school curriculum and assessment are delivered manually.
The problem facing the education sector in Kenya is how best the curriculum, instruction and assessment can be transformed to meet the ever rising call to employ ICT to teach and assess learners in the 21st century. In Kenya, the adoption of ICT in the education system is far from getting to the optimal required level and there is no empirical study that has fully addressed the level of integration of ICT required.
Research Questions
The study focused on the following questions:
i) To what extent is adoption of Computer Adaptive Testing in educational assessment affected by students and teachers perceptions?
ii) To what extent is adoption of Computer Adaptive Testing in educational assessment affected by the existing technological infrastructure?
iii) To what extent is adoption of Computer Adaptive Testing in educational assessment affected by the existing computer literacy?
Objectives of the Study
The objectives of the study were to determine:
i) The extent to which adoption of Computer Adaptive Testing is affected by students’ and teachers’ perception.
ii) The extent to which adoption of Computer Adaptive Testing is affected by the existing technological infrastructure.
iii) The extent to which adoption of Computer Adaptive Testing is affected by computer literacy.
Research Hypotheses
The study hypotheses were:
i) Adoption of Computer Adaptive Testing in educational assessment is not affected by students and teachers perceptions.
ii) Adoption of Computer Adaptive Testing in educational assessment is not affected by the existing technological infrastructure.
iii) Adoption of Computer Adaptive Testing in educational assessment is not affected by the level of computer literacy.
All hypotheses were tested at α 0.05 (95% confidence level)
Justification of the Study
Research on Computer Adaptive Testing in Kenya has been limited and therefore, this study hopes to provide awareness on the potentials of using computers in educational assessment. The findings of this study, it is hoped, they will provide feedback to
educational policy-makers, particularly the Ministry of Education regarding Computer Adaptive Testing policy regulation, formulations and implementation in educational assessment.
The study will also be useful to the Kenya National Examination Council and students in preparing them to keep abreast with emerging trends in educational assessment. Future scholars and researchers who might be interested to pursue this area further, it is hoped, that they will find this study important.
Scope and Limitation
This research project aimed at assessing the possibility of shifting the mode of test administration from Paper Based Testing to Computer Adaptive Testing. The study did not attempt to develop a Computer Adaptive Test but it purely relied on responses concerning the perception of students’ and teachers’ on the adoption of CAT. Since students have never been tested using CAT before, the findings of the study were based purely on how the students and teachers would feel if CAT was to be adopted in educational assessment.
This study was conducted only in those schools that offer Computer Studies as an examinable subject in Makueni County. Makueni County was chosen because the inhabitants are not only known to be very receptive to new ideas but also their willingness to retain those new ideas.
Operational Definitions
Computer Adaptive Test: an exam administered on a computer that adapts the difficult of each item to the competency level of the student. The next question selected depends on the response to the previous item.
Adaptive Testing: the process of test administration where the selection of the test items for administration depends on the examinees responses to earlier administered items (Weiss & Kingsbury, 1984).
E-Assessment: an electronic assessment process where technology is employed to administer, score, record and report assessment activities.
Computer Based Testing: method of test administration in which the questions are delivered via the computer.
Educational Assessment: the process of measuring and documenting what students have learnt.
Assessment: an educational process of gathering, analyzing and interpreting information related to students’ performance. (Ghaicha, 2016)
Perception: the way in which something is regarded, understood or interpreted. How students and teachers view adoption of Computer Adaptive Testing in educational assessments
Adoption: the readiness to use computers in educational assessment. The readiness to adopt Computer Adaptive Testing in educational assessment was established by determining the coefficient scores for variables such as the ratio of students to operational computers, power reliability, speed of the internet, ICT Infrastructure and students ease of use of computers.
.