SCIENCE COMPETITIONS: DO THEY FOSTER LEARNING?

Many studies have shown positive effects of gamification and serious games on teaching and learning. From 1989, we have been organizing National Science Competitions in Portugal, with participants from schools from all over the Country. In this work, we assess the impact of these competitions on student’s motivation and performance in the school. The study is mainly based on surveys done on the years 2015 and 2017. Both, students and their teachers, have participated in the surveys. We also present some data about usage of the computer system used in the competitions during the event and for practicing to the event. Based on teacher’s opinions we conclude the events have some positive influence on student’s motivation.

These questions are generated by Question Generator Models (QGM), which are highly parametrized templates described in [2]. Parameters in a QGM can be text, numbers, figures, tables, etc. thus allowing each QGM to generate thousands of different questions on the same topic but preserving the difficulty level. Fig. 1 presents an example of a question generated by a QGM used in a Mathematics competition. In the game presented in this figure the team is on the first level of twenty (nível 1/20), the time remaining is 29 minutes and 34 seconds (29:34) and two lives remain (on each level the student has two trials to answer correctly).

Figure 1: A question from PmatE
As a curiosity, some teams can complete the maths tests, which means providing 80 true/false correct questions, in less than 3 minutes. This seems impossible, since they take, on average, a little more than 2 seconds to answer each question, but it happens because they take an enormous amount of time practicing for the competitions.

METHODOLOGY
We are interested in the impact of competitions on learning in general and our aim is to contribute with the case of PmatE's science competitions. To investigate the effect of these competitions on the motivation and outcomes of students from several school levels in Portugal we used surveys and data from the PeA from practice and competitions.
Schools decide if they want to participate in the competitions and if so they select their students and take them to the University of Aveiro. On the arrival, teachers go to the registration desk and in the years 2015 and 2017 they filled a questionnaire about the impact of NSC on students' motivation and learning. We analyze the teachers' answers, in a Likert scale (1-5) for two questions: (QT1) "Do you think the competitions contribute to increase students' motivation for learning?" (QT2) "Do you think there is some relation between students' performance in the school and their results in the competitions?" The first question aims to confirm the perception we have that students get very motivated when they are answering questions from PmatE's tests and that it has influence on their behavior on the school. The second question contributes to investigate if PmatE's competitions have some effect on students' performance in the school. We value teachers' sensibility to these questions and, although it is just their opinion, it is somehow the best source we can get because they are in close contact with their students.
In 2015 we gathered 326 answers and in 2017, only 224. This difference lies in the fact that in 2015 teachers were surveyed from the 4 teaching cycles that participated in NSC and in 2017 only the teachers who attended the pupils of the 3 rd cycle and upper secondary education were surveyed. In addition, 1659 students in 2015 and 1902 in 2017, randomly chosen after competing, answered a questionnaire about the NSC, filled by staff members to avoid problems about readability and minimizing incomplete answers.
We analyze the data from 2 non-consecutive years, 2015 and 2017, to minimize the existence of the same students in the same school levels, and to get a broader representation.
In 2015, 3531 students from the 7 th to the 9 th grades (3 rd cycle of basic education) and 2030 from 10 th to 12 th grades (secondary education) participated in the NSC, so our sample is 29,8% of the participants. Regarding 2017, the participation was distributed as follows: 3818 students from 7 th to 9 th grades (3 rd cycle of basic education) and 1787 from the 10 th to the 12 th grades (secondary education), so our sample is 33,9% of the participants.
We cannot control the number of teachers that accompanied the students because not all of them went to the registration desk, however we do know the number of schools participating in the NSC. In 2015 there were 229 schools (including the ones of primary education and 2 nd cycle education) and 326 teachers answered the questionnaire. In 2017 there were 187 schools from the 3 rd cycle and secondary education and 224 teachers were inquired, so we may assume that a large percentage of schools answered the questionnaire.
As a curiosity, regarding the number of students that are part of the educational Portuguese system we gathered some data from [5] and [6].
In 2015, there were 315950 students in Portugal (excluding the archipelagos of Madeira and Azores) enrolled on the regular education system in the 3 rd cycle, which means that 1,12% of these students participated in the event NSC this year. On secondary education there were 196380 students enrolled on the corresponding courses, however only 106208 (54,1%) were enrolled on STEM areas, and as NSC for these grades are for these areas we can state that 1,9% of these students participated in the 2015 NSC edition. We get similar numbers for 2017.

RESULTS
In this section we will analyze teachers' and students' answers to questionnaires and also data extracted from the PeA from which we infer that students are highly enrolled on these kind of games.
Since the beginning of the school year, the PeA provides tests with the same format and on the same topics of the ones used in NSC, although using in each level several different QGMs preventing memorizing the type of answer. These tests are called "training tests" that can be used either for training for the NSC or to improve students' knowledge on some of the curricular topics they are studying.
PeA collects data from the interactions of each user along the time, which allows us to infer some conclusions.   level (the best time recorded was 2 minutes and 16 seconds) and 485 teams from the 9 th grade, from which only 11 got to the 20 th level (the best time recorded was 7 minutes and 3 seconds).
The 2015 NSC edition had 1514 participants from secondary education, 293 teams from the 10 th grade (17 teams got to the 20 th level registering 2 minutes and 51 seconds as the best time), 259 teams from the 11 th grade (with only 5 teams attaining the level 20 and a best time of 12 minutes and 15 seconds) and 205 teams for the 12 th grade (with 5 teams on the 20 th level and the winner registering a time of 3 minutes and 59 seconds).
The analysis of number of training tests done from January to May 2015, shows that a great number of students participating in the NSC are effectively engaged in the games. For instance, the total amount of training tests performed just for the competition Equamat for 7 th grade in 2015 was 29433.
The scenario repeats itself in 2017. The NSC 2017 received 3420 participants on Mathematics competition Equamat, distributed by the three grades, 7 th to 9 th : 575 teams on the 7 th grade, 79 of them getting to the 20 th level (the best time recorded was 6 minutes and 10 seconds), 539 teams from the 8 th grade, 15 getting to the 20 th level (the best time recorded was 2 minutes and 58 seconds) and 597 teams from the 9 th grade, where 21 of them got to the 20 th level and the best time recorded was 5 minutes and 50 seconds.
The 2017 NSC from secondary education edition had 1336 participants: 293 teams from the 10 th grade (26 teams got to the 20 th level registering 3 minutes and 4 seconds as the best time), 195 teams from the 11 th grade (with only 10 teams attaining the level 20 and a best time of 5 minutes and 39 seconds) and 230 teams for the 12 th grade (with 7 teams on the 20 th level and the winner registering a time of 2 minutes and 06 seconds an absolute record).
The number of students and schools accessing PeA to solve training tests is superior to the participants in the NSC (see Table 1), as there are many schools and students that use them for other purposes as testing their knowledge or even just because they like the challenge. These numbers support the belief that science competitions engage students in their process of learning, and are in accordance with the answers given by either teachers or students, as we will see in the next section. Since serious games can be more effective for learning than conventional instruction (see [4]) the students' enthusiasm can have an important positive impact on learning.

Competitions and motivation
In Table 2 we have the results of teachers' answers for question (QT1) on 2015 | 2017 competitions. The questionnaires were filled by 326 teachers in 2015 and by 224 teachers in 2017. The possible answers go from A1 (nothing) to A5 (a lot). In the first row the answers refer to NSC in general; in the second just for 3 rd cycle maths competitions (7 th to 9 th grades) and in the third row the numbers for secondary school maths competitions (school grades 10, 11 and 12). In Figure 4, with the percentages of answers, we can see that almost all teachers have answered that the competitions have "some" or "a lot" of influence on students' motivation. There are some differences on answers A4 and A5 between the years 2015 and 2017. These variations occur due to the differences in our samples of the analysed years. While in 2015 the teachers inquired were from primary (3 rd and 4 th grades) to secondary education, in 2017 only teachers from 3 rd cycle and secondary education were inquired.

Competitions and learning
We decided not to ask students if the competitions contributed to their motivation in school, because it is a difficult question for young people to answer, however we have included the following simple question: (QS1) "Do you think you learn with the competitions?".
In 2015 there were 4530 students participating in mathematics competitions and 1659 answered the questionnaire, what corresponds to 36,6%. In 2017 we collected 1902 from a population of 4756 students corresponding to 40%. Table 3 collects the answers to QS1 and Figure 5 presents the histograms for 2015 and 2017. The scale used varies from A1 (nothing) to A5 (a lot).