ON THE IMPORTANCE OF ASSESSMENT ON FLIPPED LEARNING

We have been using a combination of computer systems in Calculus for some years. The system MEGUA on Sagemath for authoring parameterized contents, system SIACUA for helping autonomous learning with the parameterized contents created, with Bayesian feedback and finally PmatE, a computer system used in Portuguese national competitions, for doing the assessment. Although this combined use of the systems accomplishes one of our goals, to motivate study during the whole semester instead of only before the main tests, it does not seem to be enough to convince students to work before the classes in an inverted learning environment. Students tend to use the systems only before the assessment tests. In the academic year 2018/2019, we have confirmed this thesis. Although our effort to provide to students the best possible conditions for working before the classes, they correspond to this motivation only it this work is assessed in the beginning of the class. Hence, giving the materials and assuming they work is, at least in our context, an ineffective approach. We compare two topics in flipped learning in a course with 99 students of Industrial Management Engineering (IME): a topic where assessment took place in the beginning of the class with another one for which, there was no assessment in the beginning of the class. Moreover, we compare the performance of these students in IME with others of a similar sample, from other engineering courses, on the same topic in flipped learning. The first ones knew they would be evaluated by a mini-test and the later knew that they would not be subject to evaluation. Our main conclusion is that, at least in our context, it is not safe to assume students work seriously before classes in flipped learning. In the absence of intrinsic motivation, using assessment in the beginning of the class is the best approach to convince students to do the previous work. Also, in some simple topics, providing appropriate learning materials, including short videos, this approach can improve learning outcomes. The data we use include, computer systems usage data, student’s marks and interviews of IME students with the best final marks in the Calculus course.


INTRODUCTION
Since about 25 years ago ( [1]), active learning pedagogies and new methods of delivering course material have being developed.Although active learning can be seen simply as integrating in-class activities together with traditional exposition, some teachers have been adopting the flipped classroom model by adding some instructional contents for working outside the classroom ( [2]).In preparation for class, students are required to view these contents.According to Tucker ([3]) students should use the time in class to work through problems, advance concepts, and engage in collaborative learning.Lage et al ([4]) gives a simple definition of flipped classroom: "Inverting the classroom means that events that have traditionally taken place inside the classroom now take place outside the classroom and vice versa".This definition, however, only suggests a reordering and redefinition of which activities are done in and out of the classroom, but some research in this subject shows that it is not enough.In some authors point of view ( [5]), as important as providing good study materials, textbooks, lecture notes, videos or other materials is to include an evaluation of the work done at the beginning of the class.Steed ([6]) suggests starting small and moving away from traditional classes to more active-learning methods one lesson at a time.Some other authors defend that we should flip a whole course and not only part of it.To keep students' attention, videos should be kept to around 5 minutes length and activities for previous independent study should not be repeated in the class.Class time is devoted purely to active and collaborative assignments, activities or quizzes.Frydenberg ([5]) suggests counting the quizzes toward the final grade to motivate students.We adopted an approach to flipped classroom as a learning methodology that consists of a direct computer based individual learning outside and before the classroom, and then learning activities in the classroom.It is part of our previous work, the creation and use of computer systems and digital contents to provide the best possible conditions for an independent learning in flipped learning environments in Calculus.We refer to SIACUA ([7]), a system for independent study with parameterized multiple-choice questions with detailed solutions created with MEGUA ( [8]), and videos, providing feedback computed by a Bayesian user model ( [9]), and system PMATE ( [10]) for assessment using the questions from SIACUA.These systems are described in [11].Although students use these two platforms for learning before assessment, it was not clear they used them before the flipped classes if there is no assessment in the class.Our main goal with this work is to make clear the importance of assessment in the beginning of the classes when we provide materials for previous study.We believe that a serious work with the material provided only occurs if students known they are going to be assessed immediately.This seems to be true in our context but indeed, we cannot assume that it is so in general, and flipped learning can be effective, without immediate assessment in other contexts, if students have more intrinsic motivation.In addition, in some simple topics, we have observed that, providing appropriate learning materials, including short videos, is an approach that can improve learning outcomes.

METHODOLOGY
We consider a course with 574 Sciences and Engineering students about Calculus with several variables (Calculus 3) at the University of Aveiro, Portugal.From this universe, we have selected three classes taught by the same teacher avoiding the effect of having different teachers in the sample.Then we split the students in two groups and analyse their behaviour in different topics: with assessment immediately after independent study and without immediate assessment.For this analysis, we observed computer systems accesses before the flipped classes as well as the student's final marks in the relevant topics.Finally, we interview the students with higher grades and ask them about their work before flipped classes.
The first group (G1) with 55 elements was taken from the 99 Industrial Management Engineering (IME) students.The second group (G2) was selected from another class, having only students from Mechanical Engineering (ME) and has 35 elements.In both groups, we have selected all students from the three classes in the continuous assessment type of evaluation, assessed by two main tests with weight 40% each in the final mark together with several mini-tests done in the classes along the semester, weighting the remaining 20% of the final mark.
There were two different topics in Calculus chosen to implement flipped learning: derivative of composite function (T1) and parameterization of curves (T2).
First, we compare the students from G1 in the two topics T1 and T2.For both topics, these students were informed that they should study before the corresponding classes, C1 for topic T1 and C2 for topic T2, and received clear indications about where they could find the material inside SIACUA, including short videos and many exercises with detailed solutions and explanations.
These students had information that, in topic T1, assessment would be performed by a written minitest using contents like those in SIACUA, in the end of class C1.For T2 students knew that a computer test in PMATE would take place in the beginning of class C2.Hence, students had to study T2 by themselves, without opportunity to speak with the teacher before assessment.We point out that topic T2 is a preliminary topic easy to learn with the provided material.
Moreover, we have interviewed the students with higher marks from G1 to obtain a better understating of their behaviour before the flipped classes.
Finally, we have compared the final marks of the two groups G1 and G2 in topic T2.All students of both groups were told to study the same contents before the class.They knew that the lesson would start assuming the work was previously done but, for group G2, no assessment was defined for that topic other than the first main test later.

RESULTS
The first comparation, students from G1 in topics T1 and T2, shows that student's behavior is substantially different if there is a mini-test in the beginning of the class.The second comparation, students from groups G1 and G2 in topic T2, suggest that learning outcomes can be better in a subject if assessment is done immediately after the independent study.

Comparing the first group in the two topics
The following table synthesises the most relevant usage data form SIACUA for group G1 in the three days before lessons C1 and C2, respectively.The first column contains the percentage of active students in the system, before classes C1 and C2.We can see that only 20% of students have used the system before the first class and all students have entered the system before the second.Second column shows the percentage of students watching some of the videos.We see that just 11% of students watched some videos before C1 against 93% before C2.Finally, the third column shows the percentage of students that saw at least one multiple-choice question: 9% before C1 and 98% before C2.
Table 1 shows clearly that assessment is crucial in the flipped learning methodology.Without immediate assessment, only 20% of students used SIACUA at least once, while with assessment in the beginning of the class almost all students saw at least one video and one multiple-choice question.
The following boxplots complement this information by presenting the number of videos and questions seen by students from group G1 in the three days before each of the classes C1 and C2.If this immediate assessment is implemented, flipped learning can be very effective in increasing student's engagement what, according to [12], may be the most important advantage of the methodology.We tested these topics in the main test.The average classification of group G1 in the question about topic T1 is 66% and the average in the question about topic T2 is 64%.A two tailed t-test gives pvalue 0,744 > 0,05 and clearly there is no significant difference between means.Hence this data shows no improvement in the results from T1 to T2.In fact, it is difficult in general to get some conclusive evidence comparing the final marks in two different topics, since we do not know if, for example, the difficulty of the topics or the questions are similar enough for the group of students.
Nevertheless, the student's behavior before classes C1 and C2 is clearly different.To understand why this happens we have interviewed the five students from G1 with higher classification, taking the opportunity of the extra test they had to do to defend their classification above 80%.All stated they have only studied seriously before the flipped class if there was a mini-test in the beginning of the class.The reason is that they have too much work with other courses and they must focus only on what counts for assessment.This raises a different problem we are not dealing here.What really matters is that we cannot assume students do their independent study part in the flipped learning methodology without the immediate assessment.

Comparing the two groups in the second topic
We have also compared the performance of groups G1 and G2 in the same topic T2.SIACUA usage by the two groups is described in the following table.SIACUA usage is clearly better in G1.Only 47% of students from group G2 have seen videos and 32% have seen questions.We believe that the main reason for this results is the fact that students in G1 knew they were going to be assessed in the class and students from G2 knew they would have more time to study before the first main test, although many other variables can influence these results.
The following boxplot resumes system usage by students form group G2 in the three days before class C2.If we compare with the second boxplot in Figure 2, we see there is less system usage, particularly, the number of questions seen by students is much lower.The mean classification of G1 in the question about T2 in the main test is 64% and the mean for G2 is 30%.Although we are now comparing results in the same topic, with the same teacher for both groups, the students, as individuals, are different and so their behavior could be different with or without assessment in the beginning of the class.In fact, the groups may have slightly different levels.
Other variables may influence such different results, but we believe the assessment in the beginning of the class after independent study in group G1 has an important influence.The following boxplot synthesis the students results from G1 and G2 in the question about T2 in the main test.

CONCLUSIONS
Through this experiment, we can conclude that the flipped classroom approach brings a new attitude to teaching and learning environment.In fact, study contents are prepared and provided for independent study and classes are used for problem-solving and discussion at a deeper level.Although some defenders of the flipped classroom approach argue that the success is due to its foundations in active learning, our research shows that student's behavior may remain the same as in traditional education.However, attitudes change when a moment of evaluation is included at the beginning of the lesson.Therefore, it is crucial that this stage of the process is not ignored.
In addition, we found several benefits of flipped classroom approach to learning activities.The habits in preparations of classes change, the levels of efficacy during classes increases, and students are using the material made available for their independent work before classes: the computer systems, the small explanatory videos and questions with proposed detailed explanations.
Although most of the motivation is extrinsic, due to students knowing they will be subject to assessment, students in flipped classroom showed higher preparation efforts during the semester in general.Moreover, the main test results support the assertion that students, in a well implemented flipped classroom, with assessment in the beginning of the classes, prepare themselves better and obtain better marks.

Table 1 .
SIACUA usage by G1 students in the two topics

Table 2 .
SIACUA usage by the two groups in the second topic