Oral ability components: an analysis of analytic item indexes at Celpe-Bras exam

Laura Marcia Luiza Ferreira
PDF XML

Abstract

Large scale tests are frequently under a challenge as they have to inform what is being tested and also argue about their social accountability. In this paper, I discuss how the oral ability components are being evaluated by the rater avaliador-observador at the Celpe-bras exam. The analytic rubric was combined by the following features or items: comprehension, interactional competence, fluency, lexical adequacy, grammatical adequacy and pronunciation. The aim of this paper is to study the items properties in line with the rubrics descriptions and tasks. The analyses of the item difficulty index and the item discrimination index were made in the light of Bachman’s (1990) model of communicative language ability and was calculated using Rasch’s Partial Credit Model. One-thousand examinees’ scores were analyzed from the first oral exam edition of 2016. Item analysis endorses the idea that tasks influence the way components interact. By analyzing the items indexes, I argue that linguistic competence is more important to place examinees at the higher proficiency bands compared to the strategic competence related items. I also discuss that comprehension may be a requirement for the examinee to develop oral performance at the Celpe-Bras oral exam.

Full-text of the article is available for this locale: Português (Brasil).

References

ARYADOUST, V. S.; GOH, C. A Rasch analysis of an international English language testing system listening sample test. IN: 3 REDESIGNING PEDAGOGY INTERNATIONAL CONFERENCE, Anais…. Singapore, 2009.

BRASIL. Ministério da Educação. Secretaria de Ensino Superior. Certificado de Proficiência em Língua Portuguesa para Estrangeiros: Manual do Examinando. Brasília, 2015.

BRASIL. Ministério da Educação. Secretaria de Ensino Superior. Certificado de Proficiência em Língua Portuguesa para Estrangeiros: grades de avaliação da prova oral. Brasília, 2019.

BRASIL. Ministério da Educação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Edital n. 1, de 28 de janeiro de 2016 - de abertura de inscrições do exame Certificado de Proficiência em Língua Portuguesa para Estrangeiros (Celpe-Bras/2016.1), 2016 Disponível em: http://download.inep.gov.br/outras_acoes/celpe_bras/legislacao/2016/edital_n1_de28012016_celpe_Bras_2016.1.pdf Acesso em: 04 set. 2017.

BACHMAN, L. F. Fundamental considerations in language testing. Oxford: Oxford University Press, 1990.

CANAGARAJAH, S. Translingual practice as spatial repertoires: Expanding the paradigm beyond structuralist orientations. Applied Linguistics, Oxford, v. 39, n. 1, p. 31-54, 2018. Doi: https://doi.org/10.1093/applin/amx041

CANALE, M. From Communicative competence to communicative language pedagogy. In: RICHARDS, J.; SCHMIDT, R. (Org.) Language and Communication. Londres: Longman, 1983, p. 2-27.

DELL’ISOLA. R.; SCARAMUCCI, M. R. V.; SCHLATTER, M.; JUDICE, N. A avaliação de proficiência em português língua estrangeira: o exame Celpe-Bras. Revista Brasileira de Linguística Aplicada, v.3, n. l, p. 153-184, 2003.

DEMARS, C. Item response theory: understanding statistics measurement. Oxford: Oxford University press, 2010.

ECKES, T. Introduction to many-facet rasch measurement: analyzing and evaluating rater-mediated assessments. Frankfurt: PeterLang, 2015.

FERREIRA, L. M. L. Habilidades de Leitura na Proposta de Interação Face a Face do Exame CELPE-BRAS. Dissertação Mestrado. Belo Horizonte: UFMG, 2012.

FULCHER, G. Testing second language speaking. Londres: Routledge, 2003.

FULCHER, G.; DAVIDSON, F. Language testing and assessment: an advanced resource book. Routledge: New York, 2007. p. 91-114.

JASPER. J. The transformative limits of translanguaging. Language & Communication. Elsevier, v. 58, p. 1-10, 2018. DOI: 10.1016/j.langcom.2017.12.001

MAIR, P.; HATZINGER, R.; MAIER M. J. eRm: Extended Rasch Modeling. Versão 0.16-0. 2018. Disponível em: http://r-forge.r-project.org/projects/erm/. Acesso em: 01 mai. 2018.

McNAMARA, Tim. Language Testing. Oxford: Oxford University Press, 2000.

McNAMARA. Language Testing. In: DAVIES, A.; ELDER, C. The handbook of applied linguistics. Oxford: Blackwell, 2004, p. 763-778.

MESSICK, S. Validity. Nova Jersey: Educational Testing Service Princeton, 1987.

SCARAMUCCI, M. V. R. O Projeto Celpe-Bras no Âmbito do Mercosul: contribuições para uma definição de proficiência comunicativa. In: ALMEIDA FILHO, J.C.P (Org.) Português para Estrangeiros: interface com o espanhol. 2.ed. Campinas: Pontes, 2001, p.77-90.

SZABÓ, G. Applying Item Response Theory in Language Test item bank building. Frankfurt: PeterLang, 2007.

SCHOFFEN, J. R. Gêneros do discurso e parâmetros de avaliação de proficiência em português como língua estrangeira no exame Celpe-Bras. Tese de Doutorado. Porto Alegre: UFRGS, 2009.

SCHLATTER, M.; SCARAMUCCI, M. V. R.; PRATI, S. Celpe-Bras and CELU proficiency exams as political acts in Brazil and Argentina. In: ALTE 3rd - International Conference Cambridge 2008, 2008, Cambridge. Programme of the ALTE 3rd - International Conference Cambridge 2008. v. 1. p. 64-64, 2008.

SMITH, R. M., Polytomous mean-square fit statistics, Rasch Measurement Transactions. v.10, n. 3, p. 516-517, 1996. Disponível em: <https://rasch.org/rmt/rmt103a.htm>. Acesso em: 20 mai. 2018.

VERGUTS, T.; BOECK, P. D; A note on the Martin-Lof test for unidimensionality. Methods of Psicological Reseach – ONLINE, v.5, n.1, 2000. Disponível em: <https://www.dgps.de/fachgruppen/methoden/mpr-online/issue9/art4/Verguts.html>. Acesso: 20 mai. 2018.