European Journal of Educational Research A Combination of Context Input Process Product and Kirkpatrick Evaluation Model to Determine the Effectiveness of E-Training for Principals during COVID-19 Pandemic

: COVID-19 pandemic forces training for principals to be conducted online. This study aims to evaluate the context, input, process, response, study, behavior, and outcome to determine the effectiveness and make recommendations for training. The approach used in this study was a mixed method with a concurrent embedded design and a qualitative method as the main method. The subjects of this study were the principals, committees, and instructors. Data were collected through interviews, observations, and questionnaires and analyzed using Miles and Huberman's model, descriptive analysis, and a t-test. The results show that e-training is effective in context, input, and process. Response, learning, and attitude assessments prove that knowledge, skills, and attitude have improved. Participants will be able to implement the experience gained and impact school quality improvement. This study contributes to the combination of the two evaluation models proven to produce a complete result. The study for the e-training recommends needed assessment before the training, the activity before the training to acquire the skills in using the learning management system, and the monitoring and evaluation after the training. A combination of context input process product and kirkpatrick evaluation model to determine the effectiveness of e-training for principals during COVID-19 input training impact, are CIPP evaluation

The situation of the COVID-19 pandemic leads to training being conducted online. E-training is a remote training process that uses the internet or intranet so that training participants can acquire the knowledge they need (Ben Amara & Atia, 2016). E-training effectively develops human resources (Al hila et al., 2017). E-training can solve the problems of traditional training that have limitations in implementation (Wolor et al., 2020). It reduces the cost of travel, accommodation, and trainers. It also provides easy access and flexible materials and can support the learning process well (Belaya, 2018). E-training can also improve performance (Alkali & Mansor, 2017;Garg & Sharma, 2020;Kamal et al., 2016;Zainab et al., 2017). However, on the other hand, e-training has some weaknesses in that it leads to limitations in communication, lack of motivation, frustration due to login problems and the system not working, and an additional workload (Dhull & Sakshi, 2017;Hussein et al., 2020;Lemay et al., 2021), technical problems with internet access due to poor connection (Baczek et al., 2021;Gumede & Badriparsad, 2021;Hussein et al., 2020), and the participants' lack of digital skills (Baczek et al., 2021;Dhull & Sakshi, 2017). The CIPP evaluation model is one of the legitimate evaluation approaches (Stufflebeam & Zhang, 2017). It takes into account the CIPP. Contextual assessment is the attempt to examine the needs assessment, problems, potentials, and opportunities to determine goals and priorities (Karimnia & Kay, 2015;Neyazi et al., 2016;Stufflebeam & Zhang, 2017). Input evaluation refers to the alternative approach, planning, and budget adequacy to select the plan that maximizes program effectiveness (Divayana et al., 2017;Shih & Yuan, 2019). Process evaluation focuses on scoring the implementation of program planning. It provides feedback on the extent to which the program is being applied (Esgaiar & Foster, 2019;Shi, 2018;Sopha & Nanni, 2019). Product evaluation assesses the attempt to measure the program's performance and evaluate its outcome (Al-Shanawani, 2019;Finney, 2019;Gokmenoglu et al., 2021;Mohmmed et al., 2020).
Meanwhile, the Kirkpatrick model is popular because it is straightforward and more practical in evaluating training programs (Maudsley & Taylor, 2020). The Kirkpatrick model is an evaluation model that consists of four levels of evaluation. These are evaluations of reaction, learning process, attitude, and result. On the level of reaction, it is about participants' responses and reactions to the training, i.e., satisfaction and positive attitude (Florea et al., 2016;Jain et al., 2021;Kirkpatrick & Kirkpatrick, 2007;Ragsdale et al., 2020;Steele et al., 2016). The learning process level measures cognitive change, skills, attitudes, and a consequence of the training program (Bernardino & Curado, 2020;Dorri et al., 2016;Sahni, 2020). The third level is the evaluation of attitude. It aims to see how participants apply what they learned during the training when they return to work Zahro & Wu, 2016). The fourth level is outcome evaluation. It focuses on the organization-level evaluation and refers to the long-term outcome of whether the organization's goal can be achieved after participating in the training and has a significant impact on its service to customers (Abdulghani et al., 2014;Agarwal et al., 2014;Calvo et al., 2019;Y. Park & Jo, 2019).

Research Design
This evaluation study combines the modified CIPP with the Kirkpatrick evaluation model. The approach is based on a mixed method with the Concurrent Embedded Model. This model was chosen because the qualitative method is the primary method, which has greater value in evaluating context, input, process, setting, and outcome. Meanwhile, quantitative data is the second method to assess the response and learning process. The data obtained qualitatively is expected to provide reliable and intensive information. The subjects for this research are the school principals who participate in the in-service training for school principals. This training was attended by 165 school principals from kindergartens, elementary schools, junior high schools, and senior high schools in Indonesia. Participants must have an academic degree no lower than a bachelor's degree, be actively working as a school principal, have been appointed before 2018, and be no older than 60 (sixty) years old. The other participants were a head of the training institution, a secretary, three committees, and six instructors. The convener who passed the selection and followed the training and technical guidance of the Ministry of Education and Culture was conducting the training.

Phases and Instruments
The research phases are based on the combined evaluation model of CIPP and Kirkpatrick. The aspects of CIPP evaluation are context, input, and process. The aspects of Kirkpatrick's evaluation are reaction, learning process, attitude, and result. The evaluation tools used to collect data are tests, interviews, observation, tasks, and document studies. The data sources are training decision-makers, instructors, participants, and committees. The data source of the decision maker was the head of the training institution assigned by the Ministry of Education and Culture of the Republic of Indonesia to manage the training. The data source of the head of the training institution was the training context, government policy regarding the training's goal, curriculum, material, and objective, which were obtained through interviews and documentation studies. The data source from the participants was the school principals who participated in the training until it was completed. The data collected from the participants are about the training's input, process, response, learning, behavior, and outcome. The data were obtained through interviews, observation, documentation, questionnaire, and task. The data source of the trainer was the aspect of input, process, and learning. The data was obtained from the interview, documentation study, observation, questionnaire, test, and task. The data source of the committee was the aspect of the training process, which was obtained through observation, document study, and questionnaire. The explanation of the phases and instruments for this research can be found in Table 1. The interaction occurred in the training 5 Participants of the training can follow the entire parts of the training 6 Supervisors fulfill the supervising duty during the training 7 The committee fulfills the duties in the facility in the process of training  Improvement in the quality of the school program 3 Improvement in quality of the school The procedure of e-training was conducted in three phases. The phases and materials for the e-training for the principals can be observed in Table 2 below.

Data Collection
Qualitative data were obtained through interviews and documentation studies. An example of the interview questions was: Is the policy of training for principals relevant to the need to improve the competence of principals? How is the accuracy of the conduct of training to the predetermined schedule? Do the participants successfully implement the knowledge and skill gained in work? Does the training have any impact on the improvement of the professionalism of teachers and the quality of the school? The data of the documentation study were obtained to study the document or information related to the training, which was the government policy about the conduct, curriculum, material, mark, assignment, and presence list of the training.
Meanwhile, the quantitative data were obtained to learn about the aspect of reaction, and the reaction evaluation was done through questionnaires. The option of answers used a Likert scale interval of 5 (very agree), 4 (agree), 3 (neutral), 2 (disagree), and 1 (very disagree). Examples of the statements of the questionnaire were: the training is conducted professionally, the training materials are particularly useful, the satisfaction with the service, the trainers master the materials, the varied training methods, and the ease of accessing the facilities. The evaluation of the aspect of learning was obtained from the test result and the assignment of the participants.

Validity Test on the Instrument
Three validators validated interview and observation instruments. The evaluation result showed suitability between the questions and the indicators. The validators suggested that several sentences in the questions must be revised and two more questions suitable for the indicators. The validity and reliability test of the questionnaires was done through an experiment on 20 principals. The questionnaires were valid and reliable, with the score of the validity test in the range of 3.1 to 4.4 and the score of the reliability test is 8.9.

Analyzing of Data
Quantitative data were processed descriptively by calculating the percentage and creating the categorization. Meanwhile, to test the difference between the pre-test and post-test data, a t-test difference test was used. The analysis technique for the qualitative data was done in three phases: (1) data reduction, (2) data presentation, and (3) conclusion drawing (Miles et al., 2014). In the data reduction phase, the researchers conducted the coding on the data gained from the field. The code was meant to choose which data are disposed of and which pattern can be a finding. The next phase of the data analysis was to conduct the data presentation. Then, the researcher conducted data categorization, or data grouping, into several classifications after the data was gained and after it was given coding on the data. The final phase in analyzing the data was to draw a conclusion and verify. In this phase, the researcher discovered each symptom's meaning and obtained data. Then, the data became valid and robust data for each conclusion.

Results
Implementing the CIPP Evaluation Model is done on context, input, and process evaluation. Meanwhile, the Kirkpatrick model evaluates the reaction, learning, behavior, and result. The response evaluation explains the participants' satisfaction with the training and the behavior. The result supports the evaluation of the product in finding out whether the participants implement the experience obtained in work and the impact on the institution.

Context Evaluation
The context of the training indicates that the training is based on governmental policy to increase the competence of the principal. Curriculum, objectives, and materials are developed based on the standard of competence of the principal. Still, it is not based on the result of the need analysis of the institution and each principal's needs. The training design uses On-the-Job Training 1 (OJT 1), In-Service Training (IST), and On-the-Job Training 2 (OJT 2). The target of the training is that the principals can master the theory and practice in analyzing the school's condition, challenges, and opportunities for development on the school so that it will impact the improvement of the school's quality.
As stated by the head of the training institution, "the curriculum is accurate to the needs of the principals to be able to handle the issue, fulfilling the duty as principal, improving the managerial skill, supervision and the entrepreneurship. Yet, there is no need for analysis in planning the practice for the participant candidate".

Input Evaluation
The input evaluation shows the selection process of the participants, trainers, supervisors, and administrators has fulfilled the term and agreement. This finding is relevant to the statements of management for the training that "the recruitment of the trainers has fulfilled the requirement, it has proven that the trainers' ability and knowledge are relevant with the content taught. The selection of the participants is made according to the terms and agreement such as the administrative requirement and that every participant is at least working for four years".
However, the information from the administrators stated that "there was no knowledge prerequisite that the participants must acquire as well as the skill in using the information and communication technology so that the initial skill of the participants are very diverse". Trainers number 3 and 6 mentioned that "the administrator and supervisor master information technology and can manage the LMS well".
The LMS facility is designed to facilitate the learning process, the interaction between the trainers and the participants, and among participants, as well as to monitor and evaluate the participants' work. As stated by the participant of the training, "LMS contains the facilities of forums for discussion, chatting, and the access to the materials and the assignments uploaded".

Process Evaluation
The training is done according to the schedule and steps planned, 1 st OJT, IST, and 2 nd OJT. This result is stated by trainers number 2, 4, and 5 that "in the OJT 1, the participants conducted the material exploration and the problem identification on the school. The assisting and the guidance from the trainers are done through the LMS, which is for problem identification, good practice, and material exploration". Then, trainers 1 and 3 mentioned that "the 1ST activities are the initial test, problem-solving, various types of good practice, drafting the development plan and the final test". The trainers have guided well through the LMS, WhatsApp group chat, and Zoom meetings. The participants' motivation to join the Zoom meetings is higher than those held on LMS. The participants mentioned that "the OJT 2 is done well. The trainers guided the participants in preparing the SDP by making the schedule, socialization, coordination, committee building, guideline drafting and proposal making, as well as the monitoring instrument analysis and evaluation".
The OJT 2 is done well, as mentioned by the participants that "the trainers used the correct media. The assistance by the administrator is done professionally. The committee managed the activity correctly and facilitated the participants and the trainers well".
The weakness in the conduct of the 1 st OJT is that there was no virtual meeting. The activities are done entirely through LMS. The participants could still not use the LMS well, so the discussion and interaction were not optimal. The participants experienced issues in the virtual meeting through the Zoom application. As mentioned by one of the participants, "I am disturbed by the bad internet connection, the Chatting menu of the LMS is seen to be less effective by the participants and the trainers, and not all the participants are actively discussing in the forum".
During the training process, the participants did not like the training through LMS. They prefered virtual meetings. As stated by one of the participants, "It is more exciting to join the training in a meeting, although it is done virtually instead of learning using the LMS because each meeting is more interactive".

Reaction Evaluation
The participants' reactions to the training can be seen in the questionnaires given by the committees to the participants. The reaction evaluation is done upon the satisfaction of the service (convener, committees, admins), trainers, the content of the materials, methodology used, media, facilities, ease of access, and the assignment. The questionnaires given use the following Likert interval scale. The satisfaction of the training participants is seen from the eight indicators. It shows a tendency that falls to the excellent category with the highest score percentage on the indicator of trainers. On the other hand, based on the answer to the statement in the questionnaires, the motivation of the participants is low. The participants prefer a direct meeting with the trainers and the other participants.

Learning Evaluation
Learning evaluation is one to determine the improvement in the learning result on the aspect of knowledge and skill acquired by the participants through the scoring of the working demo, as well as the scoring on the participants' attitude through observation during the training process. The post-test result on the knowledge aspect shows that most participants earned a score in the category of 'satisfactory'. To determine an enhancement in knowledge, the difference test of pre-test and post-test scores was conducted using the t-test. The result is as follows:  The result of the data analysis on the pre-test and post-test shows an improvement in the mean from the pre-test to the post-test of 14.1256, with a correlation score of .568 and a significance level of .000. Based on the result of the analysis, it can be drawn that there is a difference between the pre-test and post-test scores with a significance level of .00, which means that the improvement is significant between the pre-test and post-test scores conducted by the training participants.
The score of attitudes during the training is analyzed using descriptive data analysis by categorizing the score attitudes. The following results from the score of attitudes are in Table 5. The findings show that the attitude score of all the participants scored high, which falls in the category of 'very satisfactory'. That includes an independent attitude such as innovative, strong principles, and brave in expressing their opinion, as well as a cooperative attitude such as the ability to do discussion, cooperation, consolidation, and solidarity. The scoring in terms of skills (working demo) is the scoring in designing, conducting, and applying SDP during the training. The SDP is a top program that every participant determines. The result of the working demo can be observed in Table 6. The program of SDP made are the development of the teaching media, workshops on the PowerPoint presentation, drafting and conducting academic supervision, entrepreneurship training for the teachers and students, training in making teaching videos, training on utilizing information technology, training on using online teaching applications such as Google Classroom, Google Form, training on library management, and the training in making tech-based teachers' administration.

Attitude Evaluation
Attitude evaluation is related to the changes in the attitude of the participants after joining the principal's proliferation program. The evaluation is indicated to be successful if the principals teach experiences gained during the training to the staff and teachers and can apply principles of management, supervision, and entrepreneurship in the form of a top program of SDP. The result of the response to the changes in attitude is explained in Table 7. The ability to teach the acquired knowledge and skill in the workplace 85.4% Good 2 The ability to apply the acquired knowledge and skill in the workplace 84.8% Good 3 Improvement of knowledge 92% Excellent 4 Improvement of skill 89.8% Excellent 5 The skill to design, conduct and evaluate the SDP 91.4% Excellent 6 Skill in solving work-related issues 77.6% Good 7 Improvement of motivation 90.8% Excellent 8 Changes of attitude 89% Excellent Rata-rata 87.6% Excellent The changes in the participants' attitudes show the tendency to score that falls in the excellent category with the highest percentage on improving knowledge, and skill in developing the SDP.

Result Evaluation
Result evaluation was done one year after the training. This step aims to measure quality, productivity, and performance improvement. Based on the interview with the principals, the SDP's conduct impacts the teachers. That was the improvement in creativity, skill in using innovative teaching methods, classroom management, ability to finish school administrative works, and the improvisation in media usage of the teachers. The impact on the students was that their learning motivation was improved. The effect on the school was that the programs were more directed and planned according to the SDP that had been constructed. One of the participants said that "after applying the development program for teachers, the students are more motivated in joining the lesson. That is proven from the questionnaires of motivation that the teachers distribute to the students".

Discussion
Research findings suggest that the curriculum should be developed based on the competencies of school leaders rather than the needs assessment of participants. The training impacts improving competencies, and the objectives of the training are needs-based, specific, measurable, and relevant (Aziz et al., 2018;Liao & Hsu, 2019;Mohebbi et al., 2011). The 1st OJT -IST -2nd OJT training model allows participants to analyze the state of the school and observe the challenges and opportunities for school development. This model is consistent with Stufflebeam and Zhang's (2017) idea that context is used to assess the needs and state of the educational environment.
The research findings show that the LMS's curriculum management, facility, and platform are already consistent with the process and conditions. This finding also applies to the selection process of the participants, the recruitment of the teachers, the recruitment of the LMS administrator, and the recruitment of the supervisors who have met their qualifications and competencies. The appropriate teaching materials and curricula encourage participants to follow the teaching and learning activities to improve their knowledge and skills (Hakan & Seval, 2011;Lippe & Carter, 2018;Tuna & Başdal, 2021). The quality of competent teachers in their subject areas enables participants to understand the material. These findings are relevant to the theory of Danton, who mentions that the instructor must meet the quality requirements as a teacher, the proportionality of the number of teachers, and the facilities used (Al-Shanawani, 2019).
However, the problem in practice is that some participants are not proficient in information and communication technology, which prevents them from performing the learning activities through the LMS, and their motivation is low. Ismail et al. (2020) concluded that the ability to use e-learning motivates participants to learn. Another conclusion of this research is that the initial knowledge of the participants is very different, as no analysis was required before the training. This information is contrary to the statement that training requires an examination for efficiency and effectiveness (Cotes & Ugarte, 2021;Garg & Sharma, 2020;Ludwikowska, 2018;Sunita & Ajeya, 2010). Input evaluation is a way to build the support system, solution strategy, and process design for future program implementation and helps determine the changes needed for successful work (Esgaiar & Foster, 2019).
The process evaluation indicates that the 1st OJT, IST, and 2nd OJT were well managed according to the plan. Good management actively encourages participation to influence the learning outcome (J. . The training was conducted according to the established curriculum and under the guidance of the instructors via LMS, Zoom meetings, and WhatsApp group chat. Implementing the curriculum and the online learning method positively contributes to the success of the training (Amin et al., 2021;König et al., 2020;Li et al., 2021;Nácher et al., 2021;Smith & Rains, 2020).
There is no monitoring and evaluation after participants return to work. Monitoring and evaluating the training helps to find out the impact of the training (Hanaysha & Tahir, 2016;Mpofu & Hlatywayo, 2015). The monitoring and evaluation of the training can help to discover the impact of the training, so it is important to conduct the monitoring and evaluation after the training (Altavilla, 2019;Foster et al., 2017;Schukro et al., 2019). The problem of the network, the limitation of interaction and communication, and the results of previous research show that not all participants adopt online training due to the difficulty of accessibility and psychological problems (Alavudeen et al., 2021;Maqableh & Alia, 2021). Participants' skills and abilities in using information technology (IT) are still limited, resulting in less effective training (Nasser & Ja'ashan, 2020;Welsh et al., 2003;Zalat et al., 2021).
Responses show that participants' satisfaction with training falls into the "good" category, but the motivation to learn is lower (Dhull & Sakshi, 2017;Lemay et al., 2021). This situation differs from Belaya's e-training, which can motivate participants to improve their knowledge and skills to perform better (Belaya, 2018). Evaluating the response to the training program is successful if it encourages participants and increases their interest. This result is consistent with Dewi and Kartowagiran's (2018) finding that response assessment is successful when participants feel happy and are motivated to learn. Participants' interest, attention, and motivation to follow the training process are indicators of program success. The online training model that provides the benefit will promote interest in learning (Chang et al., 2021;Hofmeister & Pilz, 2020;Sriprasertpap, 2015;Yu-Fong Chang et al., 2021). Participant resonance can be measured when participants respond positively to the instructional dimension, the use of instructional media, the method that promotes good participation, and the quality of the instructor (Embi et al., 2017). The indicators of good response are when participants provide positive feedback on the training (Paull et al., 2016).
Based on the result of the learning evaluation, participants' mastery of the knowledge aspect improves compared to the time before the training. Participants' skills and attitudes are in the "excellent" category. This result is consistent with the findings (Ismail et al., 2020;Jain et al., 2021) that e-training can improve the knowledge and skills of principal candidates. In mastering the knowledge and skills of management, entrepreneurship, and supervision, school leaders must perform their duties well. Principals need the training to improve their knowledge (Brauckmann et al., 2020) and the skills to enhance their competence and leadership quality (Al-Hamad et al., 2020;Ismail et al., 2020). The exclusive functions of e-training enable participants to learn optimally. The developed functions of e-training significantly impact acquiring knowledge and new skills (Ismail et al., 2020).
The attitudinal assessment shows that all participants have implemented the product resulting from the training in their school. The attitudinal assessment combines instruction and participants' experiences in their workplace. This finding is consistent with the statement mentioned by Kirkpatrick and Kayser that the attitudinal assessment is a synthesis between a person's understanding and reflection on attitudinal change in the workplace . The training can improve the participants' attitude regarding management skills, knowledge, and the necessary skills (Deodhar & Powdwal, 2017). During the training, participants have the talent to apply the knowledge and skills that can be used directly in their work, and the changes in attitude and awareness about the degree of change and continuity (Ruskanda, 2018) and the application of the training material in terms of knowledge and skills and attitude change (Embi et al., 2017;Heydari et al., 2019;Paull et al., 2016).
The evaluation of the result shows that the training impacted principals' performance improvement and school quality. As suggested in the work of Wolor et al. (2020), e-training can improve work performance. The training directly impacts teachers' professionalism, and school leaders must become teacher managers who inspire instruction (Wolor et al., 2020). Principals contribute to and make decisions about the school organization's success or failure. Their role is related to school effectiveness and student outcomes (Chalikias et al., 2020).

Conclusion
The difference between this research and the others is the combination of two evaluation models, the Kirkpatrick model and the CIPP model, which gives a complete result because the evaluation is performed in seven steps. Based on the explained research results, the following conclusions can be drawn: (a) the context of the training program has relevance to the improvement of competence and the goal of the training, although no analysis was required; (b) the input of the trainers, supervisors, and administrators is competent and recruited according to the procedure and with a sufficient LMS. Participants are selected according to the requirements, although they were not provided with the necessary skills to operate the LMS and Zoom sessions; (c) the training process in terms of time management, media, and setup is well conducted. The trainers, supervisors, and administrators worked according to their tasks. There were some issues with internet connectivity, and some participants had difficulty using the LMS. The conveners did not conduct sufficient monitoring and evaluation after the training; (d) the participants' response to the training shows a rating with the indicator of 'good', but is low in the evaluation of motivation; (e) the learning evaluation shows the improvement of knowledge, which can be seen from the result of the t-test with the significance of .00, and the skills and attitude are in the category of 'good'; (f) the participants' attitude after the training achieved the rating of 'excellent'; (g) the training has an impact on the quality improvement of the school. Overall, the training was effectively conducted for school leaders. However, improvements are still needed in various aspects.
This study contributes to the knowledge of the new evaluation model combining CIPP and Kirkpatrick. The combination of the two assessment models provides a complete result, and the training evaluation of CIPP can evaluate the context and input Kirkpatrick lacks. At the same time, the Kirkpatrick model can assess training response and impact, which are missing from the CIPP evaluation model.

Recommendations
The findings recommend that for online training of school leaders, a needs assessment of the institution and individuals must be conducted, participants must be prepared to operate the LMS and virtual meeting application, and a pre-test is required to classify participants based on their competency level. Post-training monitoring and evaluation must be conducted to see the participants' success in implementing the lessons learned in training and the impact of the training. For future research, it is expected that the topic will be expanded to other regions that are broader and more complex so that the results will be more diverse and from different perspectives.

Limitations
This study has one limitation: the target group is only from a specific area (Brebes Regency) and the education levels studied are primary and secondary schools.