Warning: mkdir(): Permission denied in /home/virtual/lib/view_data.php on line 93 Warning: chmod() expects exactly 2 parameters, 3 given in /home/virtual/lib/view_data.php on line 94 Warning: fopen(/home/virtual/ejpbl/journal/upload/ip_log/ip_log_2024-07.txt): failed to open stream: No such file or directory in /home/virtual/lib/view_data.php on line 100 Warning: fwrite() expects parameter 1 to be resource, boolean given in /home/virtual/lib/view_data.php on line 101 An illuminative evaluation: Student experience of flipped learning using online contents
J Probl Based Learn Search


J Probl Based Learn > Volume 4(1); 2017 > Article
Yoon, Park, and McMillan: An illuminative evaluation: Student experience of flipped learning using online contents



Flipped learning is a pedagogical approach that supports the notion of active learning by putting emphasis on accessibility to advanced technologies such as Massive Online Open Online Courses (MOOCs) to support a blended learning approach. The growing trend of encouraging connected learning among students reflects the need for teachers to participate in online course activities.


An illuminative evaluation approach incorporating focus groups was used to study the program; how it operated, how it is influenced by various school situations, what those directly concerned regarded as its advantages and disadvantages and also how students’ intellectual tasks were impacted.


The data on perceptions of outcomes formed 3 main themes: Challenges to self-regulated learning; Transcending learning concepts - Repeat, apply and debate; and Value of global content. Students have an interest in and skill with information technology and are responding positively to their experiences with Flipped Learning, blended learning activities and the use of MOOCs.


While there is no single model for an approach to the flipped classroom, those involved in this project are aware of how important it is to consider the extent of achievement of integration of the learning activities. There is also need to be cognizant of the educational design and its consistency with our ambitions for graduates who can perform in the real world of practice.


Flipped learning is a pedagogical approach that supports the notion of active learning by putting emphasis on accessibility to advanced technologies to support a blended learning approach. A primary rationale for the flipped model is that learning at the lower levels of Bloom's Taxonomy (acquiring/remembering information and understanding the concepts) is something students can easily accomplish on their own; learning to apply these foundations is best accomplished with the guidance of an instructor (Khanova, McLaughlin, Rhoney, Roth & Harris, 2015). Students benefit from the outside classroom events because they can allocate their time and pace their online learning to meet their individual levels of comprehension (Gaughan, 2014; Gilboy, Heinerichs & Pazzaglia, 2015; Lowell, Utah, Verleger & Beach, 2013). In face-to-face classroom sessions, students have the opportunity to become more active and interactive through group activities rather than passively listening to lectures. Teachers in turn, are able to commit more in-class time to monitoring student performance and providing adaptive and instant feedback to an individual or group of students (Kim, Kim, Khera & Getman, 2014).
In the higher education sector, great emphasis has been placed on the student-centered learning environments in which students are strongly engaged in higher-order tasks and taking charge of their own learning (Shea & Bidjerano, 2010). The growing trend of encouraging connected learning among students reflects the need for teachers to participate in an online course activity, especially, those teachers who are passionate about participating in online forums and in particular joining Massive Open Online Courses (MOOCs).
MOOCs have brought about a revolution in education (Pappano, 2012). During the last year, the MOOC has been gaining attention among academics from higher education institutions that are launching courses in various areas. There are Platforms like Coursera, edX, MiríadaX, FutureLearn and K-MOOC of Korea providing open courses to global audiences mostly with free to access or with minimal access fees. Most of these platforms offer learning content as a form of short video lectures with activities that lead to outcomes evident in formative and summative assessment (Belenger & Thornton, 2013; Kolowich, 2013). This global educational revolution embarks upon new blended learning opportunities in universities in which the main ideas of MOOCs are exploited to enhance teaching and learning in the form of successful flipped classrooms (Presti, 2016). Students watch videos containing the theoretical concepts from home and practice the application of these concepts with automatic correction exercises. They later attend classrooms to solve problems with teachers present; eventually the aim is for deeper learning on the topic (Tucker, 2012). As growing numbers of colleges and universities experimented with MOOC integration into learning events, diverse approaches regarding MOOCs adjustment to context, integration across disciplines, and the use of research were employed.
The purposes of this study were (a) to design a new undergraduate course named ‘Emergency and Disaster Nursing’ by integrating MOOC content and flipped classroom practices, and (b) to explore the effectiveness of this instructional approach through an illuminative evaluation involving focus groups that sought to appraise students’ experiences and perceptions.


‘Emergency and Disaster Nursing’ was a newly developed course offered to 3rd year students as one of the nursing electives within in the 4-year Bachelor of Nursing Program. This course was designed to run 3-hours per week for 15 weeks, involving a 1 hour online lecture, 1 hour online discussion, followed by 1 hour classroom discussion each week. Online lecture clips related to each week’s learning topics were selected from content offered on MOOCs platforms. One of the reasons to use MOOCs content for this course was the lack of learning content available for Disaster Nursing in Korea. This was an area of practice still to be developed within the Nursing Discipline in Korea. Online lecture content served to introduce the global practice perspective to discussion within the Korean context.
An online discussion board was mainly used to pose a set of topic-specific questions for students to answer and discuss after listening to the lecture, aiming to link concepts between the online lectures and offline classroom activities. In the offline classes, small group discussions and various scaffolding activities were used to challenge students’ deeper learning and critical thinking skills. This course design included the 2 key components of flipped learning, as described by McLaughlin and Rhoney (2015): (a) Moving acquisition of foundational knowledge outside the classroom through the development of self-paced online learning format for review prior to class; and (b) using class time for active and applied learning.


Parlett and Hamilton (1974, p 4) described an illuminative evaluation approach as one that aimed to study the
…innovative program: how it operates; how it is influenced by the various school situations in which it is applied; what those directly concerned regard as its advantages and disadvantages; and also how student’s intellectual task and academic experiences are most affected…to document what it is like to be participating in the scheme, whether as a teacher or a pupil.
The methods chosen involved an initial literature review undertaken to investigate options for contemporary student centered learning that involved information literacy and computer assisted learning. This informed the design of the learning activity. The illuminative evaluation was meant to shed some light on the nursing students’ views on their experiences of the flipped learning event utilizing a MOOC. Focus Groups (FG) were therefore used. This involved an organized discussion with a selected group of individuals to gain information about their views and experiences of a topic. This approach is particularly suited for obtaining several perspectives about the same topic. Focus groups are widely used as a data collection technique. The value of using focus group is obtaining information of a qualitative nature from a predetermined and limited number of people (Greenbaum, 2000).
The data collection was conducted at the end of 15 weeks of implementation to, as far as possible, gain unbiased opinions from students. Among those who volunteered, a purposive sample of 12 students formed 2 groups of 6. Nine open-ended questions were posed to guide the focus group interview related to their experiences of flipped learning involving the use of the MOOC.
1. Introductory question:
  • Have you ever experienced the method of flipped learning utilizing MOOC before?

2. Main questions:
  • How did you feel about the method of flipped learning utilizing MOOC?

  • What were the most impressive/best things, in your opinion, about the flipped learning utilizing MOOC experience?

  • What were the most obstructive/worst things, in your opinion, about the flipped learning utilizing MOOC experience?

  • What aspect the flipped learning utilizing MOOC was most useful/ insightful for the learning event in which you were involved?

  • How different are your experiences with the traditional lecture methods and those involving the flipped learning utilizing MOOC?

  • Are there any strategies that might need to be made to foster learning outcomes arising from the flipped learning utilizing MOOC?

3. Wrap-up question:
  • Would you like tell me anything else about the flipped learning utilizing MOOC experience?

Data analysis

The methods used to code and categorize focus group data were adapted from approaches to qualitative content analysis discussed by Graneheim, and Lundman (2004) and focus group data analysis by Stewart and Shamdasani (2014). For coding purposes, it was necessary to go through the transcripts line by line and paragraph by paragraph, looking for significant statements and codes according to the purpose of the evaluation. The researcher compared the various codes based on the differences and similarities and distilled these into two categories and finally, the categories were formulated into 3 main themes: Challenges to self-regulated learning; Transcending learning concepts: Repeat, apply and debate, and Value of global content.


Theme 1: Challenges to self-regulated learning

E-learning demands self-regulated learning abilities from participants (Ps) who should autonomously control their learning schedules. In general, the advantage of online lectures and online discussion activities are that the engagement to learning is unrestricted in regards to time and space as long and there is a computer or mobile device with online access. Most students agreed with the value of this unrestricted access.
  • “I usually take notes while listening to lectures. It was helpful for me to double check some parts that I struggled to understand; it allowed me to repeat. It was especially helpful during the exam preparation…I was able to repeat entire lectures” (FG 1, P 1).

The unrestricted access, on the other hand, has challenged students who are usually not well disciplined with the time management. The attendance marks were automatically recorded when students streamed lecture clips and joined the online discussion. Students who received downgraded marks regretted their inability to self-regulate.
  • “In my case, I lacked self-discipline, I feel disappointed in myself for losing marks. I should have joined online lectures and discussion within given timeframes” (FG 2, P5)

It was interesting to note that students felt their concentration on online lectures lasted for shorter timelines than their experience with offline lectures.
  • "Normal lectures usually last for two hours. But (even though online lectures are shorter) since I usually listen to it in a comfortable and cozy environment, I tend to watch it at late night and even go through around the dawn" (FG 2, P 2).

  • "Most of online lectures are 45 to 60 minutes long. But after the 30 minutes of listening to it, it was hard to listen because I could not concentrate fully. Especially, when lectures last longer than 60 minutes, I felt overwhelmed and beginning to think ‘How long is it to the end'. I also found myself constantly checking times" (FG 1, P 5).

The less formal private environment was blamed for their failure to stay focused, but it is also important to acknowledge that in traditional classrooms students do not necessarily stay engaged at all times.

Theme 2: Transcending learning concepts: Repeat, apply and debate

In this course, each week’s learning concepts were taught, discussed and applied through three different forms of learning events: online lectures, online discussion and offline activities. Students saw the benefits of this type of flipped learning and told the evaluators that they were able to move beyond the level of understanding to fully engage in the in-depth debate, and to eventually explore innovative ways to solve practice problems. Moreover, gaining preliminary knowledge and developing ideas through online learning promoted the offline debate and discussion to be more meaningful.
  • "I might just forget the content of lectures easily if I just memorized and took an exam, but through the flipped learning, I could deal with the same topic again and again. I thought that was a great review (process) for me" (FG1, P 6).

  • "I was able to hear a lot of different opinions about one topic in detail, letting me think outside of boundaries and contemplate more. I felt in a similar way while I was watching other groups' debate during the offline classes" (FG 2, P 4).

  • "When I was participating in the debate, I saw students were able to give logical reasons against others’ opinion. That was meaningful to see" (FG 1, P 6).

Through the reverse-learning process, students were constantly reminded of the core learning issues. The idea of indirectly applying the elements of the case, through case-oriented discussion based on prior knowledge, promoted reasoning ability beyond the level of acquisition of knowledge through memorizing and lower order understanding.
  • “It's not just watching the lecture video but was letting me think through the scenarios in the discussion. I could think how other students should cope and think about the topic while imagining the actual situation and applying it immediately; the discussion class that I joined was impressive” (FG 2, P 1).

Theme 3: Value of Global contents

A selected list of video clips from MOOCs was provided for students to review as preparation for each week’s online lecture. Students valued listening to the global experts explaining the principles and knowledge related to the practice of emergency and disaster management. As most speakers were the field experts, embracing their interpretation of field experiences and sharing real-life examples added value into this course. Students reported that they felt better prepared to be global citizens with the potential to work in many settings or deal with people from diverse cultures.
  • “The foreign content definitely stimulated me to expand my perspective. In fact, it was hard for us to find this ourselves, so I was glad that I was able to know about the world that we have never encountered” (FG 1, P 4).

  • “The field of emergency disaster care is not a field that can be learned only through literature reviews, but it is a subject that should be practically applied. So the lectures from actual field experts sufficiently met the original goal of this course very well” (FG 2, P 2).

On the other hand, having different speakers to each topic of lecture caused some students to perceive the course was lacking coherence as a whole. Also, some reported that the style used by speakers felt awkward in comparison to their more familiar Korean lecturers who were mostly friendly, pointing out the core-concepts of learning.
  • “.... I felt that the flow was cut off. For example, if you have a lecture on the subject, you should be able to get to the next lecture. However, I felt as if I was just listening separate lectures from different specialists (FG 1, P 1).

  • “The teaching methods of foreign professors seemed to be difficult and unnatural at first because it was not the same way as Korean professors saying 'you have to know these things' or emphasizing important contents and adding detailed explanations” (FG 1, P 5).

TThe implications of teachers’ directions to students need to be carefully considered given consequences the potential to move away from the principles of student centered, adult learning techniques within the design of learning events.


The researchers realize that students are capitalizing on their interest in and skill with information technology. Students are responding positively to their experiences with flipped learning, blended learning activities and the use of MOOCs. The evaluation demonstrated that in Korea as elsewhere, there are some challenges with the management of learning platforms. However, there does appear to be greater student and teacher engagement in the processes of learning.
Although students’ satisfaction with the course was high and the flipped learning approach was well supported, there were a few suggestions for further improvement of the course. First, the main challenge lies with the lecturers to reasonably but continuously monitor and analyze the learner's online participation because the effectiveness of this flipped learning approach mostly depends on students’ active involvement on online prior to offline activities. Sending out reminders to prompt participation and requesting students writing weekly study logs can be considered as one option to deploy. It is important to establish the ground rules for student/teacher engagement and manage the expectations around feedback loops.
Another challenge of this approach was that the use of MOOCs content that are initially not specifically designed for this course. Lecturers had to search for video lecture clips through numerous platforms to match them to each week’s topic. Also, given most lecture clips are provided in English, the language barrier added some difficulties for lecturers and students. Considering the value of bringing the global expertise and real-life experiences into the class, the challenge still lies in managing resources for learning events, that is finding and utilizing contemporary resources that engage students and match a series of sessions designed to match the learning outcomes that derive from course learning objectives.
This study reported the findings from the development and evaluation of a course using flipped learning approach, which utilized online lecture clips from MOOCs. The learning pathway of online lecture, online discussion and offline discussion was well supported by students.


Belenger, Y., & Thornton, J. Bioelectricity: A Quantitative Approach, Duke University’s First MOOC. Duke University Libraries, (2013). 1–21.

Gaughan, J. E. (2014). The Flipped Classroom in World History. History Teacher, 47(2), 221–244). Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=tfh&AN=94677349&lang=es&site=eh ost-live.

Gilboy, M. B., Heinerichs, S., & Pazzaglia, G. (2015). Enhancing Student Engagement Using the Flipped Classroom. Journal of Nutrition Education and Behavior, 47(1), 109–114). http://doi.org/10.1016/j.jneb.2014.08.008.
crossref pmid
Graneheim, U. H., & Lundman, B. (2004). Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Education Today, 24(2), 105–112.
crossref pmid
Greenbaum, T. L. (2000). Moderating Focus Group Research. Sage, London.

Khanova, J., McLaughlin, J. E., Rhoney, D. H., Roth, M. T., & Harris, S. (2015). Student Perceptions of a Flipped Pharmacotherapy Course. American Journal of Pharmaceutical Education, 79(9), 140. http://doi.org/10.5688/ajpe799140.
crossref pmid pmc
Kim, M. K., Kim, S.M., Khera, O., & Getman, J. (2014). The experience of three flipped classrooms in an urban university: an exploration of design principles. Internet and Higher Education, 37–50.

Kolowich, S. The MOOC ‘Revolution’ May not Be as Disruptive as Some Had Imagined. The Chronicle of Higher Education (2013). http://chronicle.com/article/MOOCsMay-Not-Be-So-Disruptive/140965

Lowell, J., Utah, B., Verleger, M. a., & Beach, D. (2013). The Flipped Classroom : A Survey of the Research. In Proccedings of the Annual Conference of the American Society for Engineering Education (p. 6219). Retrieved from http://www.asee.org/public/conferences/20/papers/6219/view

McLaughlin, J. E., & Rhoney, D. H. (2015). Comparison of an interactive e-learning preparatory tool and a conventional downloadable handout used within a flipped neurologic pharmacotherapy lecture. Currents in pharmacy teaching and learning, 7(1), 12–19.
Parlett, M., & Hamilton, D. (1972). Evaluation as illumination: A new approach to the study of innovative programs. Centre for R, Occasional Paper 9, the University of Edinburghesearch in the Education Sciences.

Presti, C. R. (2016). The flipped learning approach in nursing education: A literature review. Journal of Nursing Education, 55(5), 252–257). doi:http://dx.doi.org/10.3928/01484834-20160414-03.
crossref pmid
Shea, P., & Bidjerano, T. (2010). Learning presence: Towards a theory of self-efficacy, self-regulation, and the development of a communities of inquiry in online and blended learning environments. Computers & Education,, 55(4), 1721–1731). http://doi.org/10.1016/j.compedu.2010.07.017.
Stewart, D. W., & Shamdasani, P. N. (2014). Focus groups: Theory and practice. (Vol. 20), Sage publications.

Tucker, B. (2012). The flipped classroom. Education next, 12(1).

Share :
Facebook Twitter Linked In Google+
METRICS Graph View
  • 5 Crossref
  • 17,675 View
  • 220 Download


Browse all articles >

Editorial Office
Halla·Newcastle PBL Education and Research Center, Cheju Halla University
38 Halladaehak-ro, Jeju-si, Jeju Special Self-Governing Province, 63092, Korea
Tel: +82-64-741-7430    Fax: +82-64-741-7431    E-mail: jpbleditor@gmail.com                

Copyright © 2024 by International Society for Problem-Based Learning.

Developed in M2PI

Close layer
prev next