INTRODUCTION
PBL and short-term projects
Creative sectors in the economy have been using strategies to entice people to come up with innovative ideas. A Hackathon is one of those strategies deployed as a platform that develops short-term projects (STP). Hackathons are projects that last less than 48 hours, aiming to encourage teams to develop creative solutions to real problems and deliver innovative products at the end. All necessary steps for the development of a project can be covered, even in a short time. Designed by IT companies, Hackathons aim to develop software and apps quickly. Later, investors used this model to identify new ideas for the market and support start-ups to create new products. The name hackathon combines two words: Hacker, the program developer, and Marathon, long-distance race symbolizing endurance and tenacity pursuing a goal. Nowadays, creative companies and industries incorporate this methodology; they regard innovation as a critical element for survival in the information society dynamics of competition and the rapid obsolescence of products.
Hackathons are educational tools that prepare students for future jobs given through experiences that elicit their creativity during such processes. Educational hackathons can be a subgroup of Project Problem-Based Learning (PBL) (Braga et al., 2021).
These short-term projects may pose a challenge for educators. The first question is whether the fundamentals of PBL concepts can provide consistent support for these new platforms. PBL is a methodology using real-world problems to students to activate their skills through project development (Graaf & Kolmos, 2007; Kokotsaki et al., 2016).
Objectives
This descriptive evaluative study aimed to compare researchers' observations with the students' perceptions about skills developed during a hackathon-type event called Expin48.
The questions were:
• Which skills in this short-term PBL did the participants use?
• Were there any new skills?
• Which skills need development?
• Which skills influenced the success or the failure of the project?
Theoretical Framework
Nowadays, professionals need to have a multidisciplinary set of skills to maintain employability (Lemaitre, 2006). Some authors defend the idea of learning based on hands-on practise to gain experience and understanding to develop problem-solving skills. Through projects, students perform tasks and walk through the cycles of conceiving, designing, and implementing (Edström & Kolmos, 2014). While understanding-based learning requires an examination of technical objects, processes, systems, and the production of scientific and mathematical references, action-based learning enables application as knowledge becomes contextualized. Therefore, it is necessary that understanding-based learning and hands-on practice be complementary for developing problem-solving skills (Pittich, 2020).
In education, the combination of understanding-based learning together with hands-on practice is referred to as experiential knowledge based on hands-on activity and not solely relying on theoretical academic understanding. Thus, these educational concepts change the definition of knowledge acquisition to development of competence (Lemaitre, 2006). Professionals need to integrate different skills to identify and solve complex problems of the current world regardless of the competence classification.
There are different points of view on how to determine which competence is the most important for today's world. For example, given the internationalization of society and commerce, students need to prepare to work in a global community, requiring preparation for practice in various contexts. Thus, language and communication skills are essential for those who will live in international environments. Likewise, diversified interpersonal skills, considering cultural and ethnic diversity, are crucial, including fostering creativity, since a diverse team has a better chance of recognizing opportunities (Lohmann, 2006). In 2014, the OECD had already pointed out that, among others, cooperation is a component that positively influences student performance at school and beyond, being as important as cognitive skills (OECD, 2014).
Although Lévy created "collective intelligence" in cyberspace, it is possible to use it in 'face space'. The collective intelligence to individual skills and competencies coordinated in networks that favor the collectivity (Lévy, 1994). The networks are a way through which information is built and shared. However, for this to occur, it is necessary to identify each one's knowledge that can be considered valuable for developing a given project, that is, identifying individual competencies that can provide mutual enrichment. In this way, knowledge acquisition is collaborative, forming collective intelligence. In addition, members will associate competencies with teamwork and communities, since through the social relations established in the group and the sharing of its members, they can manifest a collective competence—the competence created is more than the simple sum of individual competencies (Le Boterf, 1999).
The competence of innovation is central to the 21st-century engineer. However, this competence is complex, encompassing a set of others, such as creativity and leadership, which can act separately or overlap (Ovbiagbonhia, 2020). Studies show that companies struggle in developing innovative projects, mainly due to a lack of understanding of development processes (Millet et al., 2016). Schools need to develop in students the concept of innovative projects and assist with developing their inventive capacity. Among the key competencies required to address the needs of today's world, we can highlight leadership, teamwork, and creativity. Several authors describe the importance of leadership in the success of projects (Novo et al., 2017). Burke (2006) believes that, without dynamic leadership, design teams would move without a defined direction, like a boat without a rudder. Educators can also define leadership as the process of an individual influencing a team to achieve a specific goal (Gebczynska, 2019) successfully.
Amidst the variety of definitions, it is possible to identify a common element, the interaction with others aiming to accomplish an objective. Another common aspect of leadership among researchers is that leaders play a vital role in the success of projects. Thus, the lack of leadership or the lack of leadership skills may be directly associated with the failure of a project (Gadirajurrett et al., 2018). A meta-analysis study identified a correlation between leadership and project success. However, it is not a strong correlation because of various leadership styles (Damayanti et al., 2018).
Leadership by itself does not guarantee the success of a project. Teamwork is also essential. Team members work interdependently and adaptively on a common task, sharing responsibility for the results (Tarricone & Luca, 2002; Sanyal & Hissam, 2018). Therefore, a project can focus on a standard task. Teamwork can improve team members' performance rather than members working on the same job individually; team members skills become complementary when working as a team; this enhances their strengths while minimizing individual competence gaps. Other advantages of teamwork are sharing of workload, a boost in members' self-confidence, and promoting a creative work environment through an exciting, satisfying, and pleasant experience (Sharma, 2012). Teamwork can also foster team members skills development through exchanging opinions, experiences, and points of view. Studies have shown that teamwork is efficient, especially in situations that require creative resolution. Teamwork efficiency is related to the existence of common goals, establishing and understanding of individual roles by each team member and the general perception of participation in decision-making.
Developing Skills
Knowledge by itself is not enough to provide sound solutions to challenging problems. Instead, it is necessary to focus on deploying knowledge to generate new ideas, concepts, and artifacts, for problem-solving. Therefore, it is essential to include several hands-on and problem-solving skills in curricula as a priority.
The current teaching model is, however, often centered on the teacher. In this model, students are used to being passive recipients of information; Freire called this the "The banking concept of education" (Freire, 1996). According to Piaget (Piaget, 1971), students need to act and reflect on their actions to learn. The development of competence in reflection allows them to learn how to learn. A solution to the problems of traditional education is the use of more active methodologies.
PBL is an alternative option to the current model. After the teacher provides a broad theme through this methodology, students are organized in small teams and are encouraged to provide possible solutions through prototyping and elaboration. Students observe their community environment and identify problems related to that topic presented by the teacher. A learning stage Freire (Freire, 1996) referred to as "reality reading"; in the project's development, the teacher acts as a mediator and mentor, promoting teamwork and motivating interdisciplinary learning. Thus, students have incentives to learn and develop essential skills for current world problems (Graaf & Kolmos, 2007; Kokotsaki, Menzies & Wiggins, 2016).
Students have different backgrounds and prior knowledge, intelligence levels, skills, and competencies. Supervision by a mentor is essential to assemble teams with complementary competencies and skills and determine a common goal for the team members of each group. Mentor supervision also plays a role in recognizing learning cycles. During those cycles, team members notice what they need to learn to reach their goals. Knowledge is constructed collectively through the exchange between peers, practical activities, analysis, debates, and questions (Matthews et al., 2010). This collaborative experience is essential to obtain competencies and develop skills that can be invaluable qualities in future work environments (Hallinger & Bridges, 2016). PBL adds a variety of essential skills to future professionals (Lehmann et al., 2008), in addition to increasing students' interest in STEAM education (Blinkstein et al., 2017) and curiosity for the STEAM careers (Maiorca et al., 2021). Having recognized the importance of particular skills, the use of active methodologies of the STEAM type may assist in preparing students for working in teams, exercising leadership, and identifying leadership. In addition, this approach highlights the value of the idea of holding events in schools where students can develop socio-emotional, technological, and entrepreneurial skills.
Developing skills relies on incentivizing students to hypothesize, question, seek information, cooperate with the team, make decisions according to the information collected and, if necessary, reevaluate their initial points of view. In other words, there must be aware of the need to "learn to learn" (Saliceti, 2015). We consider these skills as paramount to teamwork and productivity. Furthermore, such skills are considered critical nowadays to employability. We intend to provide some evidence to the scientific and academic communities, a better understanding of students' contribution in their collaborative, dynamic learning situations; schools can deploy tools to access and measure the development of students' skills and competencies.
Expin48 (Experience in Projects of Innovation – 48hours)
Expin48 is a short-term PBL event developed by the Centre for Technological Education of Rio de Janeiro (CEFET/RJ) in Brazil. CEFET/RJ is an educational institution that differs from the traditional technology universities because it accommodates three academic levels; the Technological High School (THS), which prepares technicians in several areas of engineering, the Engineering undergraduate courses, and the Engineering graduate school. Expin48 is the first experience that includes students from those three educational levels to work together during the event for innovative solutions for real-world problems.
Expin48 is not part of the curriculum and is not related to any particular discipline, nor is it committed to summative assessment. Participation in Expin48 is voluntary, we assembled a selection of 30 students with the intent to have at least one student from each level in each team. A week before Expin48 begins, the event organizers present themes as part of that Expin48 edition. Throughout that week, students can elect at least three themes. Each team is composed of six students, gathered according to their thematic preference. Each team must choose a different theme. Assembling teams is a complex process that requires negotiation.
The event has four brief stages: a) Interaction - Students select themes and interact with each other forming teams; b) Design - The team should define a problem and present three possible solutions for mentors who have the mission to help them to choose the best one; c) Prototyping - The team develops a prototype for the chosen solution; d) Presentation - The team should prepare three kinds of presentations for the judges: Video ("Elevator Pitch" in 90 seconds), a white paper (a brief technical description), and a conference presentation.
During the first hour, students interact to assemble their teams. After that, they have three hours to elect three real-world problems and brainstorm a possible solution for each. At the first meeting with the mentors, all teams should present their ideas as problems and their solutions. During this meeting, they discuss with mentors the viability of their ideas. Then, they must choose which will be carried out (Clark & Wheelwright, 1992). Each stage has a deliverable established. It is crucial to maintain smaller goals throughout the process, preventing students from mismanaging their time. At the end of each step, there are meetings with mentors where they should present and check the project's status (Cooper, 1993). Then, each team presented the final product to a committee composed of six members from private companies and universities. The six members are part of the event Board which awards the winning team, and four other categories: technical quality, feasibility, innovation, and presentation.
METHODS
The first edition of Expin48 took place in October 2019, where proposals included global themes on cities of high-density populations: Topics for that edition covered urban mobility, waste processing, and senior life. In addition, specific themes focused on the reality of issues in developing countries such as floods, landslides, and disease.
Students named their teams, but they are de-identified as A, B, C, D, and E in this study.
We invited all 30 students to participate: 15 from technical high school (THS), ten from undergraduate and five from the graduate school. Out of the total, 19 students answered the questionnaire anonymously. However, it is not possible to determine the academic level of those who contributed. Team representation involved four students from each except Team E with three; we can infer that this represented all academic levels. Active observation by the researchers and focus groups centered on all students.
This study was submitted to the Oswaldo Cruz Foundation's Human Research Ethics Committee, Oswaldo Cruz Foundation, Rio de Janeiro, Brazil, and approved under the number 48109321.3.0000.5248.
Data Collection and Analysis
Evaluative data was collected using a triangulation of three different methodologies, all performed with student authorization.
1. Participant observation
The researchers participated as mentors, making it possible to study the students' behavior during all stages of the event. That observation enabled elaboration through a questionnaire to clarify some factors observed.
2. Questionnaire
Event participants responded to a questionnaire using open-ended and closed questions and a "Google Forms" platform, anonymously, individually, and voluntarily. The researchers formulated the questions to assess competence parameters within three categories, each divided into items: personal skills - curiosity, initiative, persistence, knowledge, flexibility, creativity, written communication, verbal communication; process skills - leadership, teamwork, organization, planning, operationalization of expertise, objectivity; technological skills - technological knowledge (tools).
The results were analyzed using a scale, where a score of one represents an unsatisfactory opinion for that item analyzed, two regulars, three indifferent, four good and five excellent. In addition, the questions were designed to score in a binary way, offered options of either one or five, minimizing the risks of subjective interpretations. To determine team scores, individual responses were considered, and calculations involved the simple average of individual responses. Thus, the intermediate grades were the result of these averages. Likewise, for the overall result, the average of the team's scores was used.
From the scores, maturity models adapted from Fisher (Fisher, 2004) were created, using the same scoring scale, where each item represents a vertex of the graph. To calculate the degree of maturation of each dimension, we assumed one as a weight for each of them because we understand that all are equally important for estimating the element. For data interpretation, we used an approach like that used by Senra & Braga (2020).
RESULTS
After the event ended, we evaluated the projects using an "ad hoc" panel comprised of academy members and innovative companies. The examining Board chose the winning team and awards in the following categories: Technical quality, feasibility, innovation, and presentation. Team B was the winner of the general category and technical quality winner, team C in the feasibility category, D in innovation, and E in the presentation. Team A did not receive an award in any category. Thus, we compared Team A with B to find differences that justify the failure and success of these teams based on their skills and competencies. We did take into consideration our observations related to the other teams.
In general, the members of the teams knew each other, except Team C, that was entirely randomly created. This information is relevant for assessing the synergies created between the components during teamwork. On the other hand, several of the participants of the other teams knew each other, as they mostly declared themselves as classmates, boyfriends, schoolmates. This familiarity can also reflect the moment of enlisting for the event when one encourages others to participate. However, when asked how they got together in teams, the majority (14 students) answered the criterion was associated with the theme.
Researchers evaluated questionnaire items according to the criteria described in the previous section, and the results were expressed in the form of a maturity model. According to this model, the "Competencies" element has 15 dimensions: Curiosity, initiative, persistence, knowledge, flexibility, creativity, written communication, verbal communication, leadership, teamwork, organization, planning, operationalization of knowledge, objectivity, and technological knowledge. Figure 1 shows the results.
It is essential to highlight that the notes on the responses to the questionnaires represent the averages of each student's perceptions of each element. However, our participant observation allowed further analysis, generating a new set of scores, as shown in Figure 2.
According to the self-assessment of team members, all teams have a reasonable degree of maturity in two elements, with an average of four. Only the leadership element had scores considered insufficient or regular in all teams. The skills with the highest scores were written communication, flexibility, teamwork, organization, operationalization of knowledge, and verbal communication, while the leadership scores were the worse. However, our observations show that there was clear leadership in at least two teams.
Team C members were not only unknown to each other but seemed not to have empathy for the theme. Thus, work began in a discouraging way and was refractory to the ideas of colleagues. However, along the way, a graduate student assumed the role of leader, managing not only to stimulate his colleagues but also to direct the work. Since he did not do this explicitly, the other members did not realize they were being led. Perhaps this is the reason they did not assume that there was leadership in their team. In a different way, in Team B, a graduate student clearly assumed the leadership role, determining the next steps to be taken and defining what was important. Team E worked erratically during the process, with rough discussions and members not giving up their opinions. From several interventions from the mentors, on the second day of the event, the team was able to find a course to proceed, still with no apparent leadership.
Perhaps due to the leadership exercised by a student in Team B, who highly organized the group, they had a strict division of labor, where each member was assigned a task to which she had better knowledge to perform it. The THS students of this team were not intimidated by their academically superior colleagues and performed complex functions, such as computer modelling, video editing, and prototyping. Team D also performed their tasks smoothly, on time, and without apparent tension. Members worked to develop their tasks, always presenting their productions to each other, reaching consensus. As previously reported, both Teams C and E worked most of the time in a disorganized, chaotic manner. However, from the second afternoon on, they achieved a level of organization that allowed the realization of a final product.
Some teams made a regular self-assessment of the items related to knowledge, initiative, planning, and persistence, and the researchers' observation was not so negative. The teams had good knowledge, worked according to a plan, had the initiative to act, and were persistent in searching for results. However, according to our observations, Team A showed a lack of planning and persistence during the process.
When analyzing this data in a graph, it is possible to verify that the dimensions are represented very similarly in all teams. Interestingly, when we superimpose the graph of the competence element of Teams A (lowest place in the event) and B (highest place in the event), it is impossible to identify significant differences. However, when we compare the data from our observation with those obtained through self-assessment, differences arise, showing that the winning team was superior in the dimensions of the studied skills (Figure 3).
When we compared the winning Team B chart with other teams, we noticed it stands out in all items and obtained lower scores in some, both by self-assessment and participant observation (Figure 4).
Through a self-analysis of the characteristics of their teams, the members were invited to score different aspects, one represents less important, and nine is most important. We observed that all teams, self-qualified with high scores for reasonable features such as friendship, interaction, and organization. While Team A scored itself highly on untimely features such as disorganization, not listening to each other, lack of focus, and competitiveness, they give low scores for opportune features such as friendship, knowledge, interaction, and organization (Figure 5).
DISCUSSION
The triangulation of the methods of capturing the information allowed for the removal of minor inconsistencies, guaranteeing significant credibility in the answers given by the students to the questionnaire. Therefore, when comparing the responses obtained by the self-assessment with the field observations, it was possible to compare. Furthermore, when we used the interview to clarify some doubts, the need for this approach became evident since, through questionnaires and participant observation, some participants' behaviors did not match.
When we analyze the leadership element, there is a substantial discrepancy between the responses to the questionnaire - the result of students' self-assessment - and the field observation. According to the majority of participants, there was no leadership in their teams during the event. As the questionnaire was designed to evaluate the elements from their answers to several different questions, it was possible to observe that the teams discussed each moment of the project elaboration and thus ruled out the possibility of leadership. In the students' conception, leadership is seen as a perjorative term and therefore cannot be related to their team. Furthermore, there is a natural confusion between the two words leadership and boss.
The results demonstrate the importance of this competence for successful project development. Leadership is closely related to teamwork. The leader motivates the team to walk ahead, promotes a cooperative environment, and inspires trust, encouraging collaboration and innovation (Anantatmula, 2010). Although the participants did not admit it, there was leadership in the teams that developed projects of technical quality. Some seemed to be unable to find a way, like Teams C and E, but thanks to the role of a leader, even discreetly, they found their path successfully.
On the other hand, one emotional disruption in Team A was produced by divergencies. This fact reduced cooperation between the members, leading them to perform below their capabilities. As a result, Team A failed to build leadership that sought balance at that time.
Since the 1980s, personal relationships between students have been a point of discussion. Johnson (1981) points out that student-student relationships can be more critical than student-teacher, as long as they are based on acceptance and supported by all the teammates. For him, during interaction through cooperative tasks, divergences or controversies are inevitable, and these conflicts can be constructive in promoting the performance and development of students. More recently, Wanders and collaborators (2020) state that the better the relationship between students, the more likely they are to feel safe and willing to participate and express their opinions. Unfortunately, this concept of better collaboration between friendly members did not seem to positively influence Team A's results as there was a misunderstanding between the team members. On the other hand, Team C shifted the behavior demonstrating a lack of initial interest, changed completely, and became united and competitive by the time of the final verdict. The leadership by a member of the team prompted this shift.
When analyzing the applied skills, the degree of maturity between the teams did not show significant differences, even between the winner team and the others. This outcome can be justified by the homogeneity of the teams formed by members of the same academic level from the same educational institution.
The highest degrees of maturity were those related to personal skills, such as written and oral communication and operationalization of knowledge. Other competencies, such as flexibility, teamwork, and organization, those related to processes, were also well evaluated.
According to Servin and De Brun (2005), knowledge acquisition management needs to be supported on three pillars: people, processes, and technology, in a multidisciplinary view, with intentional and systematic coordination between the three (Dalkir, 2013). That is, diverse knowledge is necessary for understanding and dealing with problems. When we evaluated the teams according to field observation, technological knowledge also appears to be considered excellent with a degree of maturity. Thus, we verified that the teams achieved good positions in the three fundamental pillars.
When we analyzed the element of creativity, again, there were no significant differences in the degrees of maturity of Teams A and B. On the other hand, Team E had higher scores in the degrees of maturity analyzed through the two approaches. Despite having a conflicting relationship between its members at the beginning, this team managed to present an innovative and creative solution, based on process, different from the other teams that found their solutions based on products.
Thus, having the teams present similar degrees of maturity in all elements, we have some chances to explain the failure of Team A in the event. Communication is a crucial factor for good team performance, together with cohesion, here understood as the members' desire to remain in the team. Another significant factor is a collaboration between members (Tarricone & Luca, 2002). These factors were lacking in Team A when a member exacerbated competitiveness, reduced communication and collaboration, and decreased cohesion. In 2002, Tarricone and Luca described a situation where a team that developed an educational activity had to be undone because the members complained about members who did not contribute. This situation generated resentments, leading to serious disagreements, which even the team's mentor could not resolve. The interesting thing is that before the activity, the researchers informed the teams of the attributes necessary for the success of teamwork. However, a significant number of students realized the importance of teamwork for the project's success. Asking what skills and/or competencies they developed at the event, "teamwork" was most cited.
From the responses to the questionnaire, it is possible that this team had minimum requirements for success, such as interdisciplinary technical skills and other critical personal skills, such as creativity and knowledge (Iqbal et al., 2017). According to our analysis, this team obtained a degree of maturity similar to the others, including the winner team. In addition to the lack of adequate interpersonal skills, the team detected other deficiencies, such as disorganization and lack of focus.
According to our observations interview, it appears that this team has entered a negative spiral, with one problem leading to another. The lack of leadership led to them not working as a team; this consequently brought disorganization, here understood as a lack of task rationalization. Members developed an activity and, when looking for an alignment among them, they realized there was none; this led to demotivation, which considerably decreased communication between participants, increasingly feeding this snowball effect and leading the entire team to fail.
In general, we believe that the Expin48 event was a success. Many of the difficulties encountered, evidenced by degrees of maturity classified as regular, are mainly due to the limited time and lack of intimacy with the methodology. On the other hand, the idea of gaining knowledge and innovation were remarkable; this denotes the importance of the role of students in the search for knowledge, as proposed by the PBL methodology.
The participants acknowledged that there was a gain, skills and competencies with the experience. However, teamwork was the gain most cited by the participants. Therefore, knowing how to work as a team is a crucial strategy in today's job market. Hence the importance of developing it as soon as possible.
CONCLUSIONS
The researchers sought to assess if specific competencies were essential for the successful development of an STP. No significant difference in the degree of maturity between the team that did not win any awards (Team A) with the winner (Team B) was observed. However, due to the triangulation of data capture methodologies, we realized that Team A did not obtain expressive leadership. Unlike other teams, they had difficulty working as a team and could not organize their path.
All teams, with no exception, recognized that teamwork was the primary competence needed to be acquired at the event. Unfortunately, Team A could not put this element of competence into practice, probably due to the lack of leadership. This outcome could have permitted the attitudes of one member to induce demotivation with a consequent loss of cohesion of the team.
We also suggest that leadership, teamwork, and organizations are key competencies for corporate jobs and educational settings, especially during project-based activities.
The researchers also verified that the PBL Expin48 event reached its goal; students were protagonists searching for the enrichment of knowledge, and they recognized their gain of skills and competencies with the experience. It was possible to notice that the participants did not distinguish the skills they already had from those they needed to develop. For example, although they recognized that organization, time pressure, and teamwork are their most significant difficulties, this was attributed to the project model and not to their need to develop these skills. To minimize this misunderstanding, we suggest that participants meet with the mentors at the end of the event, where these issues are raised and discussed.
Although the study contributed to the literature in this field, the researchers are aware that there are limitations. Given the sample size and participant characteristics (students being from the same campus as the institution, there were many similarities), the results should be considered in context and not generalized. Future studies can be carried out to correct these limitations, such as the extension to other institutions, a more significant number of participants, and heterogeneous teams. Further studies should also analyze the possible effects of the participants' performance over the years.
Our studies provide a starting point and important insights regarding the use and development of skills during a short-term PBL event.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Download Citation
Print
