INTRODUCTION
Interest in educational methods has increased recently, due to an emphasis on the learning process, and in the era of the Fourth Industrial Revolution, it is socially necessary to foster creative talents based on an information- and knowledge-based society. Problem-based learning (PBL) education is recognized as a method that is applicable to this situation (Hasegawa, 2019; Minamide & Takemata, 2019;Peramunugamage et al., 2019). The aim of the PBL method is to deal with practical problems using a learner-centered approach, in which students form a team and solve problems together (Barrows, 1985). This enables more immersive learning by finding realistic solutions to problems. Furthermore, because it includes team or group activities, various elements such as communication, teamwork, leadership) can be improved in the process of problem solving (Woods, 1996).
Therefore, educational methods, such as PBL, are realistic alternatives that allow students to experience the creative process through voluntary learning activities, actively solve existing problems, and gain new knowledge, before evaluating the PBL team performance using various methods, such as detailed evaluation, report evaluation, team portfolio evaluation, exhibition evaluation, team peer evaluation, and attendance. However, some cases involve more subjective judgment than the test evaluation conducted in general lectures. This highlights the need for students to create their own evaluation ratio, while explaining a clear evaluation method at the beginning of the class. Based on this need, in this study, the relative positions between PBL evaluation factors at the beginning of the class were calculated and evaluated directly by the students participating in the class using the matrix method. In other words, by applying the method of importance-performance matrix analysis (IPMA), an extension of importance-performance analysis, a method mainly used in business administration is achieved. Specifically, IPMA is a valid methodology for prioritizing evaluation factors in two dimensions (ie importance and performance). In other words, in the description of specific evaluation factors, the focus is mainly on the potential of factors of high importance but at the same time of relatively low performance (Fornell, et al., 1996; Gronholdt et al., 2000; Hock et al., 2010; Schloderer & Ringle, 2014). The ultimate purpose of this study was to examine how the evaluation factors of the class were recognized in each matrix by the students prior to participating in the class.
1. Background
In our prior experience, an instructor usually performed evaluation after the completion of PBL class. A complementary alternative to this general evaluation, a method in which students directly participate in class evaluation, such as the study by Kim (2019), could be considered. However, to apply this evaluation method more logically, it is necessary for each student to understand the shared perception of the evaluation factors of the class before proceeding with the class.
Therefore, in this study, as shown in the following figure, along with the evaluation method of Kim’s (2019) study, a procedure for determining the relative positions of relevant evaluation factors before the class was proposed. This approach intended to build an integrated model (importance-performance matrix-analytical hierarchy process; IPM-AHP) as an attempt to build on Kim’s (2019) study.
METHODS
At the beginning of the semester (at the beginning of the class), IPMA was used as the primary survey (Kim, 2019), and each evaluation factor, the average value of the latent variable, and the estimated path coefficient were used; therefore, the partial least squares-structural equation modeling (PLS-SEM) analysis method could be expanded. Specifically, PLS-SEM analysis provides information on the relative importance of each factor in explaining the cause and effect between each factor within the proposed model. In other words, it was possible to grasp the direction of the PBL class by analyzing the effectiveness of the evaluation items by using a simple questionnaire for students at the beginning of the class. In this regard, it was possible to apply not only PBL, but also highlight learning methods that included flipped learning and action learning. Specifically, in the first week of each semester, students were given a questionnaire (Appendix) and an explanation of how to evaluate the class, and after the questionnaires were completed, the analysis results were introduced in the following week. Students were able to recognize the evaluation direction of the final evaluation factors in the future while checking the relative positions between self-evaluation, peer evaluation within team members, and evaluation of other teams. IPMA used the average value of the latent variable values and the estimate of the path coefficient in a general PLS-SEM to extend the analysis method. More precisely, the IPMA process compared the total effect of the structural model on the final dependent factor (endogenous variable) with the mean value of the total latent variable of the antecedent independent factors (Fornell et al. 1996;Gronhold et al., 2000;Hock et al., 2010; Schloderer & Ringle, 2014), where the total effect showed the degree of influence on the final dependent factor (variable), indicating the level of importance or importance. The average value of the antecedent factors (independent variables) was expressed as a performance level or performance. The goal of this analysis was not only to recognize the antecedent independent variable with relatively high importance for the final dependent factor, but also to recognize the relatively low level of performance. In the end, among the factors evaluated, one can show potential areas of improvement that may receive more attention in the future.
This analysis focused on the results of the existing student satisfaction (performance) questionnaire and utilized information on the relative importance of the final dependent variable, the total effect, on the overall expectation of the class (Hair et al., 2016). In this study, SmartPLS 3.0 was used to calculate numerical values and the graph was created using SPSS 19.0.
1. Research Model
Using the IPMA method, this study analyzed the evaluation factors through the students' survey opinions at the beginning of the class, as shown in Figure 1 (research model): PLS-SEM based on IPMA. Various studies employing the IPMA method have used many measures other than performance for one axis of the matrix (Su & Cheng, 2019). The IPMA method is a technique for obtaining the average value, placing importance and performance on the Y-axis and X-axis, respectively, and for drawing a scatterplot across the four quadrants, that is, a two-by-two matrix (Figure 2). As it is a method for substituting and analyzing measurement values based on importance, performance, or other factors, it is a useful method for checking which areas or fields are lacking and which are not sufficient or overflowing.
This study utilized SmartPLS 3.0 software to conduct IPMA.
In previous studies, analysis was mainly conducted on only one aspect (importance or performance). However, this study considers importance and performance at the same time and analyzes the results with a matrix to consider the priority of evaluation factors (Al-Emarn & Mezhuyev, 2019; Sohaib et al., 2019). In the above matrix, as in the case of Figure 3, the instructor can inform the participating students in advance that the self-evaluation factor is the most effective class management method. In other words, students can become familiar with a clear evaluation direction at the beginning of the class.
RESULTS
1. Research Scope and Data Collection
The timeline of this study proceeded according to the research procedure, starting with the base year (2020), and the spatial scope of this study was for the PBL class conducted by this author. Analysis occurred prior to the class, to reach a consensus among students at the beginning of the semester. This study derived analysis results and applied them to actual PBL classes, so that they could be used to improve class evaluation through feedback on the next lecture. The data used in this study comprised involved feedback from a total of 30 persons, focusing on interested respondents in PBL-related classes, and the final data were analyzed. The questionnaire items presented in this study were extended and applied to Kim’s (2019) study based on the contents extracted from Kang’s (2003) study (Table 1). The first criteria presented in Figure 2 were classified into three categories: Self-evaluation, team member evaluation, and team evaluation. Four criteria were established as secondary criteria.
2. Empirical Results
It was possible to create an IPM using the previously calculated IPMA data analysis results. That is, the X-axis represents the total effect of each non-standardized independent variable on the final dependent variable as importance and the Y-axis represents the performance level, which is a numerical value obtained by recalculating the average value of each independent variable.
One can see by looking at the following IPM analysis (Figure 4), this study placed the importance and satisfaction of each variable on a two-dimensional line, where the Y-axis represents the level of satisfaction with export activation and the level of satisfaction increases as it goes to the right, and the X-axis shows the importance of export-activating factors. The average values of importance and achievement were calculated using the X-and Y-axes as the central axes.
Accordingly, the most meaningful result in the expressed IMPA is the interpretation of variables that are elevated in the fourth quadrant (located under the right foot). In other words, a high level of importance (influence) is shown for the final goal, the dependent variable (here, the expected satisfaction level for the total class), but a low level of performance (here, the satisfaction of each factor) is shown, so it is potentially much improved. It is interpreted as having high potential for improvement if a variable with a low level of importance compared to other variables in the IPM has a lower priority in terms of performance improvement. The results of this analysis show that it is meaningful, in terms of lecture efficiency, to prioritize the evaluation factors of S2, S4, T4, and T2. That is, it is a meaningful factor for students to prioritize the performance improvement of variables that are highly important for the final target variable (e.g., expectation) as a lecture evaluation factor.
CONCLUSION
In the case of student-led classes, such as PBL, the instructor often evaluates without an exam. In other words, it is necessary for students to self-assess each student's contribution to the group process, and to encourage them to establish aspects that are most relevant to self-directed assessments. Therefore, it is difficult to determine how to conduct final evaluation and grade. In this case, it is possible to supplement the weights between evaluation factors through a method in which students directly participate and decide (Kim, 2019). As an approach that complements this method, it can be a very useful for instructors to conduct PBL classes by figuring out the areas or factors that students see as relevant prior to the class. Hence, this study is meaningful in that it suggests the applicability of the methodology rather than focusing on the results of this study itself.
Specifically, contrary to the evaluation method of Kim’s (2019) study, which was an approach taken when all classes were completed, the method of this study was conducted at the beginning of the semester when classes started. Therefore, the instructor was able to increase immersion in the lecture by recognizing in advance which evaluation factors the students wanted to focus on. Ultimately, this study intends to apply the approach to try the integrated model (IPM-AHP) in the future, as this approach is a continuation of the previous research (Kim, 2019) proposed by the proponent.
Therefore, this study is regarded as an exploratory study, in that it is possible to grasp the evaluation direction in advance at the beginning of class by introducing the PBL education procedure and evaluation method. In the end, this approach can be used by the instructor to recognize the evaluation factors of class participants in advance and to seek a more objective evaluation method. It is hoped that such an exploratory study will eventually provide a way to increase students' immersion and satisfaction with the class.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Download Citation
Print
