Evaluation of automatic annotation of syntactic dependencies
- V. 22, N. 2 (2023)
- Submitted: Nov 19, 2022
- Published: Sep 9, 2024
- DOI 10.25189/rabralin.v22i2.2114
Abstract
Considering the importance that syntactic dependencies have been assuming in Natural Language Processing (NLP) tasks and, consequently, in linguistic studies focused on the automatic processing of languages, we present here a qualitative evaluation of a recently released gold-standard treebank for the Portuguese language, with the objective of identifying (a) the linguistic patterns that are more difficult for automatic annotators, (b) the reasons that can lead them to make mistakes in these analyses, and (c) to expand the possibilities of dialogue between the linguistic studies and computational linguistics. The syntactic annotation was performed according to the guidelines of the Universal Dependencies (UD) project, and the evaluation of the annotation was performed using open source tools, in three steps: firstly, we performed an intrinsic evaluation of a syntactic dependencies annotation model, assuming that this type of evaluation reflects the consistency of the corpus annotation with which the model was trained; then, we detail the results of this evaluation, presenting the number of correct answers for each individual linguistic class, which gave us an overview of the linguistic difficulties for automatic learning and, also, information regarding the confidence in the automatic analysis of each linguistic classification. Finally, we selected the classes with the highest number of errors and analyzed all the wrong cases. The results suggest that, on the linguistic side, we can already count on consistent analyzes and in quantity, at least apparently, enough. As far as the quality of automatic parsers is concerned, the room for linguistic improvements is getting smaller and smaller.
References
AFONSO, Susana. Avaliação do grau de concordância entre anotadores: análise e discussão dos resultados do proces-so de re-revisão, 2004.
ARTSTEIN, Ron. Inter-annotator agreement. In: HANDBOOK of linguistic annotation. [S.l.]: Springer, 2017. P. 297–313.
BAIA, Jardel; PRATES, Arley; CLARO, Daniela. CoNLL Dependency Parser: Extrinsic Evaluation through the Open In-formation Extraction task. In: SBC. ANAIS do VIII Symposium on Knowledge Discovery, Mining and Learning. [S.l.: s.n.], 2020. P. 193–200.
BICK, Eckhard et al. Floresta Sintá (c) tica: Ficção ou realidade. In: AVALIAÇÃO Conjunta, Um novo paradigma no processamento computacional da língua portuguesa. [S.l.]: IST Press, 2007. P. 291–300.
CAVALCANTI, Tatiana et al. Os limites da palavra e da sentença no processamento automático de textos. Revista Brasileira de Iniciação Científica, v. 8, e021033–e021033, 2021.
CORDEIRO, Fábio Corrêa. Petrolês-como construir um corpus especializado em óleo e gás em português. PUC-Rio, Rio de Janeiro, RJ-Brasil: PUC-Rio, 2020.
DE MARNEFFE, Marie-Catherine et al. Universal dependencies. Computational linguistics, MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info . . ., v. 47, n. 2, p. 255–308, 2021.
DE SOUZA, Elvis. Construção e avaliação de um treebank padrão ouro. 2023. Mestrado – PUC-Rio.
DE SOUZA, Elvis; CAVALCANTI, Tatiana et al. Diretivas e documentação de anotação UD em português (e para lín-gua portuguesa), 2020.
DE SOUZA, Elvis; FREITAS, Cláudia. ET: A Workstation for Querying, Editing and Evaluating Annotated Corpora. In: PROCEEDINGS of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demon-strations. Online e Punta Cana, Dominican Republic: Association for Computational Linguistics, nov.2021. P. 35–41. DOI: 10.18653/v1/2021.emnlp-demo.5. Disponível em: https://aclanthology.org/2021.emnlp-demo.5.
DE SOUZA, Elvis; FREITAS, Cláudia. Polishing the gold–how much revision do we need in treebanks? In: PROCEED-INGS of the Universal Dependencies Brazilian Festival. [S.l.: s.n.], 2022. P. 1–11.
DE SOUZA, Elvis; SILVEIRA, Aline et al. Documentação da anotação morfossintática do PetroGold, 2021.
DE SOUZA, Elvis; SILVEIRA, Aline et al. PetroGold–Corpus padrão ouro para o domınio do petróleo. In: SBC. ANAIS do XIII Simpósio Brasileiro de Tecnologia da Informaçãoo e da Linguagem Humana. [S.l.: s.n.], 2021. P. 29–38.
FREITAS, Cláudia. Linguística Computacional. [S.l.]: Editora Parábola, 2022.
FREITAS, Cláudia; DE SOUZA, Elvis. A study on methods for revising dependency treebanks: in search of gold. Lan-guage Resources and Evaluation, Springer, p. 1–21, 2023.
FREITAS, Cláudia; TRUGO, Luiza F. et al. Tagsets and datasets: some experiments based on Portuguese language. In: Computational Processing of the Portuguese Language: 13th International Conference, PROPOR 2018, Canela, Brazil, September 24–26, 2018, Proceedings 13. Springer International Publishing, 2018. p. 459-469.
MANNING, Christopher D. Part-of-speech tagging from 97% to 100%: is it time for some linguistics?. In: International conference on intelligent text processing and computational linguistics. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011. p. 171-189.
NIVRE, Joakim; FANG, Chiao-Ting. Universal dependency evaluation. In: Proceedings of the NoDaLiDa 2017 Work-shop on Universal Dependencies (UDW 2017). 2017. p. 86-95.
OLIVEIRA, Claudia et al. A set of np-extraction rules for portuguese: Defining, learning and pruning. In: Computational Processing of the Portuguese Language: 7th International Workshop, PROPOR 2006, Itatiaia, Brazil, May 13-17, 2006. Proceedings 7. Springer Berlin Heidelberg, 2006. p. 150-159.
RADEMAKER, Alexandre et al. Universal dependencies for Portuguese. In: PROCEEDINGS of the Fourth Interna-tional Conference on Dependency Linguistics (Depling 2017). [S.l.: s.n.], 2017. P. 197–206.
SAMPSON, Geoffrey; BABARCZY, Anna. Definitional and human constraints on structural annotation of English. Natural Language Engineering, Cambridge University Press, v. 14, n. 4, p. 471–494, 2008.
STRAKA, Milan; HAJIC, Jan; STRAKOVÁ, Jana. UDPipe: trainable pipeline for processing CoNLL-U files performing to-kenization, morphological analysis, pos tagging and parsing. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16). 2016. p. 4290-4297.
TAYLOR, Ann; MARCUS, Mitchell; SANTORINI, Beatrice. The Penn treebank: an overview. Treebanks: Building and using parsed corpora, p. 5-22, 2003.
ZEMAN, Daniel et al. CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies. In: Proceedings of the CoNLL 2018 Shared Task: Multilingual parsing from raw text to universal dependencies. 2018. p. 1-21.