The teaching of Portuguese as a Host Language has faced many challenges since its establishment as part of the field of Portuguese as a Foreign Language, mainly for presenting specificities unknown to the teachers and researchers in the field (GROSSO, 2010; DEUSDARÁ; ARANTES; BRENNER, 2018). Among these challenges are the methods employed to analyze the oral performance of the Host Language speakers. With this in mind, this report aims at discussing the evaluation of the oral performance of adult immigrants, beginner students of Brazilian Portuguese as a Host Language, based on two descriptive scales of the measure of Outcome Achievement, a multifaceted measure that looks, mainly, at pragmatic aspects of language use in tasks whose main focus is the communicative outcome of the performance. Even though the proposal of criteria that compose the measure of
O ensino de Português como Língua de Acolhimento (PLAC) tem enfrentado inúmeros desafios desde seu estabelecimento como parte da área do ensino de Português como Língua Estrangeira, por apresentar especificidades desconhecidas até então por professores e pesquisadores da área (GROSSO, 2010; DEUSDARÁ; ARANTES; BRENNER, 2018). Dentre estes desafios estão os métodos de avaliação empregados para medir o desempenho oral dos falantes da língua de acolhimento. Com este cenário em vista, este relato tem o objetivo de discutir a avaliação do desempenho oral de estudantes adultos imigrantes, aprendizes iniciantes de Português como Língua de Acolhimento na variante brasileira, com base em duas escalas da medida
The teaching of Portuguese as a Host Language has increased in importance in the latest years and it became an urgent matter for research to embark on investigating the processes taking place in these classrooms. Although it is of common knowledge that the teaching of Portuguese as a Host Language can be inserted alongside other specific contexts of teaching under the umbrella field of Portuguese as a Foreign Language, the Host context brings its specificities to the discussion of how to teach and evaluate language learning, which has not been addressed by the broader area yet (JENSEN, 2002
Among the specificities brought by the teaching of Portuguese as a Host language is a necessity for specialized teachers, for personalized teaching materials (SILVEIRA; XHAFAJ, 2020
Teaching Portuguese as a Host Language (PHL) involves understanding that the population in focus is composed of immigrants and refugees that have been forcibly displaced
Having in mind this context’s needs for tailored syllabuses that meet the population’s communication needs and, consequently, evaluates their performance accordingly, this study aims at presenting an alternative way of assessing students' oral performance in tasks in the context of PHL for beginners in Brazil. We aim at assessing students’ oral performance not through the well-known CALF measures but adopting the measure of Outcome Achievement (based on FARIAS, 2018
This article is organized as follows. First, we present an overview of the fields of Portuguese as a Host Language and the traditions in evaluating oral language performance in tasks. Then, we describe the method employed to gather data for the study, focusing on the population’s characteristics and the steps followed to first, implement the two tasks in a classroom setting; second, design the two Outcome Achievement descriptive scales; and finally, implement it for analyzing the participants’ oral performances. To conclude, we bring the results of the performances in the two tasks and suggest further steps to investigate the measure.
In order to analyze the proposal of a recently conceptualized measure, as is the Outcome Achievement, it is key to look at the traditions in research that have been using different measures to analyze performance, and specifically, in this case, oral language performance. To do so, it is important to look at performance having in mind the context of teaching and learning to which the measure is being proposed here. Therefore, this section is organized to, first, bring an overview of the context of Portuguese learning and teaching as a Host Language; and second, present the background of studies that inform the analyses of oral language performance.
The concept of Host Language has been frequently addressed by some researchers (GROSSO, 2010
As a consequence, this growing number of immigrants who have seen Brazil as an alternative place in which they can rebuild their lives has caused important changes in the profile of the population in need to speak Portuguese. The urge to understand the specificities of this population has been growing and it has shown the field’s need for language teaching and learning practices that fit this population best. It has been perceived that the tools available, usually used for the teaching of Portuguese as a Foreign Language, do not fit this population’s communication needs in the first moments of welcoming in the new country (DEUSDARÁ; ARANTES; BRENNER, 2018
The need to develop initiatives to assist the immigrants in the societies to which they move is urgent, once the situation of forcibly displaced people highlights necessities related to legal affairs, housing and working issues, and even more urgent matters nowadays such as access to the health systems, taking into consideration the COVID-19 pandemic. And in all these matters, language is a key aspect that may help or hinder the adaptation of forcibly displaced people to their new home.
In this context, speaking Portuguese is not only a matter of being proficient in interacting with native speakers of the target language in diverse situations; it is a matter of being proficient in solving particular and immediate necessities. According to Grosso (2010
Arantes, Brenner and Deusdará (2016) also highlight the idea that the “teaching of a host language to refugee youths and adults presents specificities that are not the same as those of foreign language teaching” (p. 1202
Accordingly, teachers and researchers should find alternatives to prepare the learners of Portuguese as a host language to perform in these situations and to assess their performances in order to develop their oral skills globally, but also giving special attention to those specific communication needs.
In this aspect, we have seen a growing number of studies that have addressed the teaching of Portuguese as a Host Language, presenting options of practices that allow for a focus on the communication needs of this population, as is the case of studies in the field of Task-Based Language Teaching (TBLT) (SILVEIRA; XHAFAJ, 2020
What we need to know about TBLT now, considering the purposes of this article, is that at its core is the idea that people use language to plan, perform and recall the activities done in the “real-world” (LONG, 2015
Moving on to looking at oral tasks in the field of TBLT, there has been a preference for analyzing performance through what is called the CALF measures (which stand for Complexity, Accuracy, Lexical density, and Fluency) (FOSTER; SKEHAN, 1996
In this sense, the pragmatic aspects of language play a more important role in the communication – and consequently in the performance – than the structural elements of language, such as the adequate use of grammar. We understand that these pragmatic elements are related to the “conditions governing the language use, the linguistic practice” (FIORIN, 2003, p.161
In the field of TBLT, Pallotti (2009
The measures of adequacy and outcome achievement are known to assess language performance taking into consideration the context in which the language is being used and the interlocutors the speech addresses, for instance. According to Farias (2018
Adequacy and outcome achievement are constructs that place an important part of successful communication in conveying the appropriate genre of communication that the tasks require, by agreeing with the contexts in which the tasks are inserted, for instance. And, due to the fact that contexts of language use entail different aspects of language production, the alternative to assess performance in these terms is to adopt descriptive scales (PALLOTTI, 2009
Pallotti (2009
Although the descriptions of the measures of adequacy and outcome achievement resemble CELPE-BRAS’s idea of proficiency, the oral part of the exam presents limitations for assessing learners in the host language context of learning. The main limitation being represented, in our point of view, by the fact that the only genre of language use available to assess oral performance in the exam is to have a conversation or interaction
Our experience, as language teachers and researchers has shown that immediate beginners are not yet equipped with the language proficiency necessary to maintain a five-minute conversation about a topic they might not be acquainted with. And that the descriptive scale designed to assess oral performance according to this genre in the exam, might not be the best fit to assess performance according to the immediate communicative needs of learners in the context of the host language. In this context, equipping the learners to perform in specific focused-tasks might be more urgent. In order to illustrate, we can think of the learners being able to communicate the symptoms of a disease to a health professional, as a task that is very urgent for learners in the context of the host language.
We understand that every proficiency test presents limitations and are designed with different objectives in mind, and that the descriptive scales provided by the tests are many times used to guide curriculum development and even to guide teachers’ choices in language classrooms. Therefore, considering the latter, scales such as CELPE-BRAS and CEFR present a comprehensive description on how to assess performance, and might not be a straightforward source for teachers designing and implementing their assessments in the classroom setting of Portuguese as a host language, specially for beginner learners who need to fulfill their immediate communication needs, irrespective of making language mistakes.
Having considered the different approaches to analyze oral performance in the field of Task-Based Language Teaching, the following section describes the steps involved in the data collection and instruments this article presents.
In order to report the process of assessing students' oral performance in the context of PHL for beginners in Brazil, by adopting the measure of Outcome Achievement (based on FARIAS, 2018
First of all, we present the context of teaching on which this study focuses, which comprises a group of students (17 in total) beginner speakers of Brazilian Portuguese, who were immigrants living in
This group of learners was composed of one Iranian, one Jordanian, two Venezuelans, and 13 Haitian immigrants. The group is far from being considered homogenous, once they vary in sex (10 men, 7 women), nationality (Iranian, Jordanian, Venezuelan, and Haitian), and mainly in age range (18 to 76 years old), schooling (a few learners have college degrees, while some others have not completed secondary school) and language backgrounds (while a few students were monolinguals, others spoke two or three languages)
The data collection reported in this article took place in two of those classes, in which students were presented to two cycles of tasks, as part of their regular language classes. Therefore, in the first class, the students participated in the class regularly and performed pedagogical pre-tasks designed to prepare students to perform the final target task (
The cycles of tasks to which the students were presented before the recording tasks were designed taking into consideration the students’ goals for Portuguese language use in their daily-lives outside school
The first task required students to identify three different images of the flu symptoms and record their answers to the task (with their phones or with an audio recorder provided by the researcher
The second task required students to choose an occupation in which they could work, and state previous working experiences that related to the occupation chosen. Once again the students were required to record their answers, this time, to the task: “
In order to assess the students’ performances in these two tasks, two descriptive scales were designed according to the measure of Outcome Achievement. The two scales followed the same rationale, considering the aspects that were important to achieve each task’s communicative outcome. The communicative outcome of the first task was to communicate the symptoms of the flu to a health professional, while the communicative outcome of the second task was to choose an occupation to offer to a colleague/friend and state the previous working experiences associated with the occupation wanted. Thus, because the two tasks presented different characteristics, one of the features that composed the measure of Outcome Achievement was adapted to comprise the specificities of each task.
The first task presented a closed-outcome, meaning that learners’ responses could be characterized as correct or incorrect based on the matching of their responses with the visual support provided. The second task presented an open outcome that required justification, meaning that students’ responses could only be characterized as correct or incorrect depending on the justification provided to their choice for occupation. This characteristic impacted on one of the features of the Outcome Achievement measure, which had to be adapted accordingly. As it is possible to see in
I. | Communicative objective | It aimed at giving a general score for the performance, depending on whether the student achieved the task’s communicative objective or not. |
II. | Correctness (for Task 1) Correctness A Correctness B (for Task 2) | It was specific to each task and concerned whether the students’ responses were correct, in Task 1, according to the images they should describe (Correctness), and in Task 2 if a student’s alleged professional experience (Correctness B) matched with the position they mentioned they were searching (Correctness A). |
III. | Communicative context | It comprised giving indications of the situation of interaction, the place, and the people involved. For example, the response should have contextual clues that the genre of the communication in task 2 – recoding a Whatsapp message to a colleague/friend – characterizes mainly as an informal situation, that implies more colloquial language use; |
IV. | Coherence | It dealt with the level of connection of the ideas conveyed by the students, “whether they followed a well-structured sequence of events in their speech, for instance, by first introducing Daniel (the fictitious character in Task 1) and then stating his health problem and not the other way around” (MARCELINO, 2020, p. 59); |
V. | Clarity | It comprised the transparency of the meanings being conveyed by the students. |
VI. | Prosody | In this analysis, it stands for the suprasegmental features of speech: intonation, rhythm, and speed. |
VII. | Vocabulary | It aimed at analyzing if the words used in the speech were sufficient and adequate to achieve the Communicative outcome of the tasks. |
To create the two scale, the teacher of the class, one volunteer teacher assistant of the same group, along with the first author of this article, analyzed the cycles of tasks to which the students would be presented before performing the task, once the cycle of tasks was designed to prepare the students to perform in these specific tasks. Then they analyzed and raised the key characteristics of each task, and finally, considering the students’ profile – immigrant beginner learners of Brazilian Portuguese – they settled seven features (shown in
The features’ descriptions seen in TABLE 1 were inspired by the studies of Farias (2014, 2018
In summary, in this study the Outcome Achievement measure comprised seven features implemented to analyze students’ oral performance using Likert scales that attributed scores from 0 to 5 (in which 0 was the lowest score and 5 the highest). In the first task 15 learners participated by recording their response, while in the second task, only 12 learners participated. The Outcome Achievement of the learners’ answers was analyzed by nine raters. Five of these raters were experienced Portuguese as Additional Language teachers with an average teaching experience of 6.8 years. These five raters had also had experience teaching other languages such as English, German, and Japanese. The other four raters had an average teaching experience of 9.5 years, teaching English as a Foreign Language (FL). All of the nine raters had a Teaching and/or Bachelor’s degree in
Each of the raters’ analyses took about 2 hours to be completed. The sessions of analyses were accompanied by the first author of this article, who would explain the overall objectives of the study and present the raters with the procedures. The decision to have the researcher present throughout the sessions was made to assure that all raters would have the same or, at least, very similar conditions of evaluation.
First, raters read an instructional text in which concepts such as the Communicative Outcome of a task were explained. Then, the raters could read the descriptive scale of the tasks and read the cycles of tasks that were taught before the performance, followed by a review of the descriptive scale of each task
In summary, raters could listen twice to each recording and then give a score from 0 to 5 to each feature of Outcome Achievement. In addition to giving a score from 0 to 5, raters should also justify their choice for all the scores given to each student in each feature.
As seen in
After evaluations were completed, in order to check for interrater reliability, we ran a Cronbach’s alpha test on SPSS (23.0). According to Taber (2018
Regarding the analysis of each learner’s performance and the group mean values in each feature of Outcome Achievement, the procedures are explained in the following section, alongside the tables presenting all the results of the raters’ assessment.
To conclude, the following section presents the results of the students’ oral performances to tasks assessed through the Outcome Achievement measure scales, designed according to the characteristics of the tasks presented in
This section presents the results from students’ recorded answers to the two tasks, and discusses these results in light of previous studies, bringing possible impacts to understanding speaking performance in the Host Language context.
As mentioned previously, the students’ oral performances were evaluated by nine raters according to the Outcome Achievement measure. In Task 1, 15 students recorded their answers.
In order to achieve the score for each learner in each feature, all the scores attributed by the nine raters to each of the seven features of Outcome Achievement were added and divided by nine, the results of this calculation are presented in TABLE 3. Then the 15 scores of all learners for each feature were added, and finally divided by 15, to achieve the group’s mean value to each feature of Outcome Achievement, presented in TABLE 2 below.
Feature | N | M | Min | Max | SD | SEM |
---|---|---|---|---|---|---|
Communicative Objective | 15 | 3,41 | 1,56 | 5,00 | 1,03 | 0,27 |
Correctness | 15 | 3,62 | 0,67 | 5,00 | 1,46 | 0,38 |
Context | 15 | 3,33 | 1,78 | 5,00 | 0,93 | 0,24 |
Coherence | 15 | 3,47 | 1,67 | 4,89 | 1,00 | 0,26 |
Clarity | 15 | 3,67 | 0,78 | 5,00 | 1,13 | 0,29 |
Prosody | 15 | 3,79 | 1,44 | 4,67 | 0,85 | 0,22 |
Vocabulary | 15 | 3,34 | 1,56 | 4,67 | 0,92 | 0,24 |
Considering that raters evaluated students’ performance in a six-point scale, from 0 to 5, where 0 was the lowest and 5 the highest score that could be attributed by each rater, the group obtained, in Task 1, a mean score higher than 3 in all seven features of Outcome Achievement, as presented in TABLE 2. The Standard Deviation values show the dispersion of the scores compared to the Mean which indicates that students’ scores fluctuated around the mean and are spread around diverse values (LARSON-HALL, 2016
It is interesting to note that when comparing the minimum and maximum scores of the participants for Task 1, five out of the fifteen participants (namely, P1, P5, P6, P7, and P11 in TABLE 3) received maximum scores for some of the features, while only two out of fifteen (P9 and P15) shared the minimum scores. This means that the number of participants who did well in the task outnumbered considerably the participants who did not perform the task satisfactorily.
Participant 6 is one of the students who received the highest scores for Outcome Achievement; therefore, according to the raters, her performance is an example of what would be an adequate speech that aimed at communicating the flu symptoms to a health professional: “
In the opposite end of the scale was Participant 15, who received the lowest mean scores for most features and consequently the lowest score for Outcome Achievement. His answer was as follows: “
Moving on to analyzing performances in Task 2, this time only 12 students recorded their performances. As shown in TABLE 4, one more time, the mean scores of the entire group were higher than 3 for most features of Outcome Achievement, except for the Correctness A feature. Standard Deviation values were low, which indicated a low variance of scores throughout the scale. The lowest and highest means were Correctness A and Prosody (2,97 and 3,44, respectively).
Task 2, as well as Task 1, presented a higher number of participants that received the maximum scores (P4, P5, P8, P9, and P12) in comparison with few participants sharing the minimum scores (P7 and P10, see TABLE 5).
Participant 8 received the best evaluation for Outcome Achievement. In his answer: “
In turn, although Participant 7 received the lowest score for Outcome Achievement, her performance in some ways is not comparable to the performance of the other participants. This is explained by the fact that, in class, she was aided by a teacher assistant and a colleague, to complete the recording. Therefore, although some raters considered that Participant 7 was able to pronounce some of the words spelled out by others, other raters considered that once she did not perform by herself she did not achieve the outcome of the task, resulting in the lowest score given to a student in this task.
The second lowest score was attributed to the performance of Participant 10, whose answer was as follows: “
Furthermore, once we analyzed students’ performances and learned what raters considered and penalized the most, it is interesting to notice the features with the highest and lowest evaluations. We believe that fruitful discussions can be raised from the fact that, in Task 1, Context received the lowest mean score, considering the group’s mean (3,33). We understand that this feature of the Outcome Achievement aimed at evaluating whether the situation of interaction was present in students’ performances, meaning that the students’ choices of language could provide contextual clues of the place where the interaction happened and who were the people involved, which is related to the pragmatic features of language (FIORIN, 2003
In Task 2, the feature that received the lowest mean score was Correctness A (2,79), and interestingly, it was not accompanied by Correctness B (3,26), once the two were expected to be connected. It is possible to suggest that students preferred to describe their work experiences in more detail, leading raters to judge the correspondence with the job wanted based on the experiences described. Most of the times, when the experience did not match the job chosen, or students described solely their past working experiences, it was Correctness A that was penalized, as is the case of Participant 3: “
On the other hand, looking at the feature that received the highest scores for the groups’ mean in both Task 1 and Task 2, we see Prosody (scoring 3,79 and 3,50, respectively). Prosody was the given name for three suprasegmental features considering speech: intonation, rhythm, and speed. On one hand, it could be argued that the highest scores this feature received are connected to the fact that both tasks allowed students to achieve an adequate outcome by using short sentences or even isolated words, like in Task 1, in which considering a “real-world” situation of communication, arriving at a health center and naming isolated symptoms could lead to successful communication, considering its objectives. An interesting further analysis would consider investigating whether there is a correlation between Prosody as an Outcome Achievement feature and the measure of Fluency from the CALF measurements.
To conclude, we should remember that the nine raters, even being all from the same field, native speakers of Portuguese, and being language teachers, diverged in their evaluations. This fact might yield two possible discussions: first of all, the use of raters to validate pedagogical practices might not be adequate as well, since the process of learning is not determined by only one single performance, but is permeated through teaching and learning practices inside classrooms, of which outside evaluators might be unaware. Second, the fact that the components of Outcome Achievement are subjective according to each rater concept of successful communication is an issue that deserves further investigation.
Measures such as the one used in this article to assess oral performance have been just recently included in the field of performance analyses in TBLT and there are a growing number of studies that have chosen to take Outcome Achievement, or Adequacy, in contrast with other well-established measures such as CALF. We agree with Pallotti (2009
In this article, it was seen that the differences between raters analyses of participants’ performances were based on the different proceduralization of the Outcome Achievement measure, indicating that although the measure prioritizes the achievement of the communicative outcome of the tasks, raters’ evaluation is influenced by their perception of fluency and accuracy, reflected in the scores attributed to the performances’ lack of errors, smooth or intricate speech, for instance. This reinforces that this evaluation, as well as others, might be impacted by some degree of subjectivity. Although raters equally understood the features, they presented different evaluations of what was an adequate performance for each of them. This might indicate the importance of having dual-task research designs that comprise a cognitive approach and, as well, outcome fulfillment, as suggested by Skehan (2003). This dual-task design might illuminate the relationship between rater’s subjective perceptions of fluency and accuracy, for instance, and allow for the correlation of their perceptions with measures that provide straightforward scores.
The growth in the field of teaching and learning of Portuguese as a Host Language, due to an increasing number of immigrants who find in Brazil a place to rebuild their lives, once many have been forcibly displaced from their previous homes, has made urgent the need to understand more deeply the specificities of this context (GROSSO, 2010; SILVEIRA; XHAFAJ, 2020, MARCELINO 2020). One of the specificities of this context is the inadequate use of traditional measures concerning oral performance. Mainly in the field of Task-Based Language Teaching the tradition to analyze oral performance has been to adopt the CALF measures (Complexity, Accuracy Lexical density, and Fluency), once these measures are believed to be part of and influence the communicative competence of speakers (FOSTER; SKEHAN, 1996; SKEHAN 1998; 2003). We know that the number of studies showing the efficacy of the CALF measures is a lot higher than the number of studies critiquing them. However, we reiterate that assessing only complexity, accuracy, lexical density and fluency might not be enough to determine if learners are being successful or not in their communication (PALLOTTI, 2009, REVESZ; EKIERT; TORGERSEN, 2016).
Moreover, considering the specific context of the Host Language addressed in this article, in which speakers are mostly focused on solving one specific problem of communication or using language adequately enough to “get one’s message across”, adopting the CALF measures to assess oral performance might be inadequate. In this context, this study aimed at presenting an alternative way of assessing oral performance in tasks that do not look at the CALF measures but that takes into consideration the Outcome Achievement measure, which is a multifaceted construct, that does not rely solely on vocabulary use, or grammatical accuracy, for instance, but instead has a strong concern with meaning and contextual elements, that may dictate the adequacy of the vocabulary or grammar to be used. To do that, 17 adult immigrants, beginner speakers of Brazilian Portuguese performed two oral tasks and had their performances evaluated by nine raters, native speakers of Brazilian Portuguese and experienced language teachers.
These students obtained different scores for Outcome Achievement, reflected in the varied scores attributed to each of the seven features that compose the measure. These seven features were thought to comprise the pragmatic aspects of language (FIORIN, 2003) involved in a situation of communication such as the ones presented in the two tasks investigated here. We believe that this article supports the idea that the process of understanding the communicative needs of the speakers, designing focused tasks according to their needs, and evaluating the speakers’ performances according to the features of these specific tasks is very challenging and can be a complicating factor when a teacher in the classroom has to take decisions on his/her own about how to evaluate the students.
In this sense, the field of TBLT seems to shed light on what to prioritize in the moment of evaluation, and on what to prioritize during the preparation of these students to perform a task, once in the context of the Host Language especially, issues related to the pragmatic elements of language (FIORIN, 2003) are fundamental.
We would like to thank the students who participated in the master’s thesis on which this article was based, the teacher and teacher assistants who were present and aided the challenging process of data collection; as well as the editors and reviewers of the magazine for their valuable contributions to the format and content of this article. Furthermore, we would like to acknowledge that the masters’ thesis on which this article was based was financed by the
For the purposes of this article, the terms immigrants, refugees, and forcibly displaced people are used interchangeably, although we understand that they might hold different legal and social statuses.
Although the situation is characterized as a conversation or interaction, the test taker, many times, might describe or support a point of view, for instance, based on the material dictating the topic of the conversation. Therefore, we agree that different textual genres can be comprised (MARCUSCHI, 2005
We do not intend to criticize the descriptive scales of such tests, but only to point out that in the case of CELPE-BRAS, because the scale presented by the test was designed originally for academic purposes, it might not be adequate to apply to assess the performance of beginner speakers.
It is important to notice that the methods presented here were part of a longer master’s investigation conducted by the first author of this article (MARCELINO, 2020
This project is part of NUPLE (
The research project upon which this article is based was approved by the Ethics Committee of the university (CEP - USFC) under the CAAE number: 07060919.2.0000.0121, and under the Report number: 3166979.
Students had to complete the task in class to assure that they would not have help from outsiders or that they would not read their responses, instead of saying it spontaneously. Furthermore, the completion of the tasks in class assured that students would have access to an internet connection to send their recordings to the researcher or even have access to the recorders, in case the students did not have a mobile phone available to record their responses.
A Needs Analyses was conducted before the design of the tasks, in order to unveil the situations of language use to which this group of learners was exposed the most. For the complete report of this process see MARCELINO, 2020
The first author of this article is the researcher who was present during the tasks’ implementation and who organized and supervised data collection.
All the texts presented to the raters were in a digital format. Raters provided their scores and justifications through an online formulary, which allowed for immediate storage of their evaluations.
Although a few raters tried to discuss their evaluations with the researcher, she replied to them by saying that she was not allowed to share her evaluation of any of the students’ performances in order not to influence their judgment of the performances.