[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Annotation of semantic roles for the Turkish Proposition Bank

  • Original Paper
  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

In this work, we report large-scale semantic role annotation of arguments in the Turkish dependency treebank, and present the first comprehensive Turkish semantic role labeling (SRL) resource: Turkish Proposition Bank (PropBank). We present our annotation workflow that harnesses crowd intelligence, and discuss the procedures for ensuring annotation consistency and quality control. Our discussion focuses on syntactic variations in realization of predicate-argument structures, and the large lexicon problem caused by complex derivational morphology. We describe our approach that exploits framesets of root verbs to abstract away from syntax and increase self-consistency of the Turkish PropBank. The issues that arise in the annotation of verbs derived via valency changing morphemes, verbal nominals, and nominal verbs are explored, and evaluation results for inter-annotator agreement are provided. Furthermore, semantic layer described here is aligned with universal dependency (UD) compliant treebank and released to enable more researchers to work on the problem. Finally, we use PropBank to establish a baseline score of 79.10 F1 for Turkish SRL using the mate-tool (an open-source SRL tool based on supervised machine learning) enhanced with basic morphological features. Turkish PropBank and the extended SRL system are made publicly available.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. FrameNet is originally built as a lexicon of verbs, however with emergence of SRL task a corpus has been annotated with its semantic frames.

  2. http://www.ii.metu.edu.tr/~corpus/corpus.html.

  3. https://www.mturk.com.

  4. https://crowdflower.com.

  5. Example taken from (Oflazer 2014).

  6. Some exceptional derived verbs like “görüş, tanış” (to meet) that acquired new uses have their own framesets.

  7. Most commonly used causative morphemes in Turkish are –r, -DIr, -D. (D represents the letters d or t, I is used to denote i or ı).

  8. ArgA represents the only causer present or the external most causer in case of multiple causers.

  9. In Figs. 6, 7, 8, 9, 10 and 13 dashed boxes represent IGs containing derivational morphemes. Morphological analysis and rolesets of predicates are given (when available) at the bottom of the box. Top and bottom arcs depict syntactic dependency and semantic relations respectively.

  10. Aggregated answer is calculated as the agreement weighted by contributor trust by our crowdsourcing platform.

  11. It should be noted that, we are able to calculate scores other than precision since annotators could decide if the argument candidate in the question is actually an argument. Therefore the set of arguments labeled by the crowd and the experts may differ.

  12. Copular constructions in Turkish have many different types and deserves another dedicated study. We avoided to discuss them under “nominal verb” section because not all copular constructions are considered derivations. In some cases such as complex time structures (e.g., was going to), copulas can also be observed as inflectional morphemes.

  13. http://universaldependencies.org/.

  14. https://code.google.com/archive/p/mate-tools/.

  15. We have slightly modified eval09 script to account for Universal Dependencies. These changes are only about reading UD files (e.g., different column index for predicate lemma sense) and do not modify the algorithm.

  16. Annotators are not constant from the beginning till the end of the task. They can instantly join/exit a task. Therefore imposing a strict order on the task would introduce a synchronization overhead for the platform

References

  • Akbik, A., Chiticariu, L., Danilevsky, M., Li, Y., Vaithyanathan, S., & Zhu, H. (2015). Generating high quality proposition banks for multilingual semantic role labeling. In Proceedings of the 53rd annual meeting of the association for computational linguistics (pp. 397–407), Beijing, China, July 2015. Association for Computational Linguistics.

  • Akbik, A., & Li, Y. (2016). Towards semi-automatic generation of proposition banks for low-resource languages. In EMNLP.

  • Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998). The berkeley framenet project. In Proceedings of the 36th annual meeting of the association for computational linguistics and 17th international conference on computational linguistics-Volume 1 (pp. 86–90). Association for Computational Linguistics.

  • Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., et al. (2012). Abstract meaning representation (amr) 1.0 specification. Parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing pp. 1533–1544. Seattle: ACL.

  • Basile, V., Bos, J., Evang, K., & Venhuizen, N. (2012). Developing a large semantically annotated corpus. LREC, 12, 3196–3200.

    Google Scholar 

  • Björkelund, A., Hafdell, L., & Nugues, P. (2009). Multilingual semantic role labeling. In Proceedings of the thirteenth conference on computational natural language learning: Shared task (pp. 43–48). Association for Computational Linguistics.

  • Callison-Burch, C., Ungar, L., & Pavlick, E. (2015). Crowdsourcing for NLP. In Proceedings of NAACL 2015. North America Association for Computational Linguistics.

  • Carreras, X., & Màrquez, L. (2004). Introduction to the conll-2004 shared task: Semantic role labeling. In Proceedings of the ninth conference on computational natural language learning. Association for Computational Linguistics.

  • Carreras, X., & Màrquez, L. (2005). Introduction to the conll-2005 shared task: Semantic role labeling. In Proceedings of the ninth conference on computational natural language learning (pp. 152–164). Association for Computational Linguistics.

  • Chang, N., Paritosh, P., Huynh, D., & Baker, C. F. (2015). Scaling semantic frame annotation. In The 9th linguistic annotation workshop held in conjuncion with NAACL 2015 (p. 1).

  • Christensen, J., Soderland, S., Etzioni, O., et al. (2010). Semantic role labeling for open information extraction. In Proceedings of the NAACL HLT 2010 first international workshop on formalisms and methodology for learning by reading, (pp. 52–60). Association for Computational Linguistics.

  • Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12, 2461–2505.

    Google Scholar 

  • Deschacht, K., & Moens, M.-F. (2009). Semi-supervised semantic role labeling using the latent words language model. In Proceedings of the 2009 conference on empirical methods in natural language processing: Volume 1–Volume 1 (pp. 21–29). Association for Computational Linguistics.

  • Duran, M. S., & Aluísio, S. M. (2012). Propbank-br: A Brazilian treebank annotated with semantic role labels. In LREC.

  • Exner, P., & Nugues, P. (2011). Using semantic role labeling to extract events from wikipedia. In Proceedings of the workshop on detection, representation, and exploitation of events in the semantic web (DeRiVE 2011). Workshop in conjunction with the 10th international semantic web conference (pp. 23–24).

  • Feizabadi, P. S., & Padó, S. (2014). Crowd sourcing annotation of non-local semantic roles. In EACL (pp. 226–230).

  • Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378.

    Article  Google Scholar 

  • Fossati, M., Giuliano, C., & Tonelli, S. (2013a). Outsourcing framenet to the crowd. ACL, 2, 742–747.

    Google Scholar 

  • Fossati, M., Tonelli, S., & Giuliano, C. (2013b). Frame semantics annotation made easy with DBpedia. In Crowdsourcing, the, semantic, web.

  • Fürstenau, H., & Lapata, M. (2009). Graph alignment for semi-supervised semantic role labeling. In Proceedings of the 2009 conference on empirical methods in natural language processing Volume 1-EMNLP ’09 (pp. 11–20).

  • Gao, Q., & Vogel, S. (2011). Corpus expansion for statistical machine translation with semantic role label substitution rules. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies: Short papers-Volume 2 (pp. 294–298). Association for Computational Linguistics.

  • Gildea, D. (2002). Automatic labeling of semantic roles. Computational Linguistics, 28(3), 245–288.

    Article  Google Scholar 

  • Hajič, J., Ciaramita, M., Johansson, R., Kawahara, D., Martí, M. A., Màrquez, L., et al. (2009). The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the thirteenth conference on computational natural language learning: Shared task, CoNLL ’09 (pp. 1–18), Stroudsburg, PA, USA, 2009. Association for Computational Linguistics.

  • Haverinen, K., Kanerva, J., Kohonen, S., Missila, A., Ojala, S., Viljanen, T., et al. (2015). The Finnish Proposition Bank. Language Resources and Evaluation, 49(4), 907–926.

    Article  Google Scholar 

  • He, L., Lewis, M., & Zettlemoyer, L. (2015). Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 643–653), Lisbon, Portugal, Association for Computational Linguistics.

  • Hengirmen, M. (2006). Türkçe temel dilbilgisi. Cankaya: Engin Yayınevi.

    Google Scholar 

  • Hong, J., & Baker, C. F. (2011). How good is the crowd at real wsd? In Proceedings of the 5th linguistic annotation workshop (pp. 30–37). Association for Computational Linguistics.

  • Hung, S.-H., Lin, C.-H., & Hong, J.-S. (2010). Web mining for event-based commonsense knowledge using lexico-syntactic pattern matching and semantic role labeling. Expert Systems with Applications, 37(1), 341–347.

    Article  Google Scholar 

  • Irvine, A., & Klementiev, A. (2010). Using mechanical turk to annotate lexicons for less commonly used languages. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with amazon’s mechanical turk (pp. 108–113). Association for Computational Linguistics.

  • Kaisser, M., & Webber, B. (2007). Question answering based on semantic roles. In Proceedings of the workshop on deep linguistic processing (pp. 41–48). Association for Computational Linguistics.

  • Kozhevnikov, M., & Titov, I. (2013). Cross-lingual transfer of semantic role labeling models. ACL, 1, 1190–1200.

    Google Scholar 

  • Lewis, G. L. (1985). Turkish grammar. Oxford: Oxford University Press.

    Google Scholar 

  • Liu, D., & Gildea, D. (2010). Semantic role features for machine translation. In Proceedings of the 23rd international conference on computational linguistics (pp. 716–724). Association for Computational Linguistics.

  • Lo, C.-k., Addanki, K., Saers, M., & Wu, D. (2013). Improving machine translation by training against an automatic semantic frame based evaluation metric. In ACL (pp. 375–381).

  • Madnani, N., Tetreault, J., Chodorow, M., & Rozovskaya, A. (2011). They can help: Using crowdsourcing to improve the evaluation of grammatical error detection systems. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies: Short papers-Volume 2 (pp. 508–513). Association for Computational Linguistics.

  • May, J. (2016). Semeval-2016 task 8: Meaning representation parsing. In Proceedings of SemEval (pp. 1063–1073).

  • Mellebeek, B., Benavent, F., Grivolla, J., Codina, J., Costa-Jussa, M. R., & Banchs, R. (2010). Opinion mining of spanish customer comments with non-expert annotations on mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s mechanical turk (pp. 114–121). Association for Computational Linguistics.

  • Meyers, A., Reeves, R., Macleod, C., Szekely, R., Zielinska, V., Young, B., et al. (2004). The nombank project: An interim report. In HLT-NAACL 2004 workshop: Frontiers in corpus annotation (Vol. 24, p. 31).

  • Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3), 436–465.

    Article  Google Scholar 

  • Oepen, S., Kuhlmann, M., Miyao, Y., Zeman, D., Cinková, S., Flickinger, D., et al. (2015). Semeval 2015 task 18: Broad-coverage semantic dependency parsing. In SemEval@NAACL-HLT.

  • Oepen, S., Kuhlmann, M., Miyao, Y., Zeman, D., Flickinger, D., Hajic, J., et al. (2014). Semeval 2014 task 8: Broad-coverage semantic dependency parsing. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014) (pp. 63–72).

  • Oflazer, K. (2014). Turkish and its challenges for language processing. Language Resources and Evaluation, 48(4), 639–653.

    Article  Google Scholar 

  • Oflazer, K., & El-Kahlout, I. D. (2007). Exploring different representational units in english-to-turkish statistical machine translation. In Proceedings of the second workshop on statistical machine translation (pp. 25–32). Association for Computational Linguistics.

  • Oflazer, K., Say, B., Hakkani-Tür, D. Z., & Tür, G. (2003). Building a Turkish treebank. In A. Abeillé (ed) Treebanks: Building and Using Parsed Corpora (pp. 261–277). Dordrecht: Springer. doi:10.1007/978-94-010-0201-1_15.

    Chapter  Google Scholar 

  • Padó, S., & Lapata, M. (2009). Cross-lingual annotation projection for semantic roles. Journal of Artificial Intelligence Research, 36(1), 307–340.

    Article  Google Scholar 

  • Palmer, M., Bhatt, R., Narasimhan, B., Rambow, O., Sharma, D. M., & Xia, F. (2009). Hindi syntax: Annotating dependency, lexical predicate-argument structure, and phrase structure. In Proceedings of the 7th international conference on natural language processing, ICON’09 (pp. 261–268).

  • Palmer, M., Gildea, D., & Kingsbury, P. (2005). The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1), 71–106.

    Article  Google Scholar 

  • Pavlick, E., Post, M., Irvine, A., Kachaev, D., & Callison-Burch, C. (2014). The language demographics of amazon mechanical turk. Transactions of the Association for Computational Linguistics, 2, 79–92.

    Google Scholar 

  • Reisinger, D., Rudinger, R., Ferraro, F., Harman, C., Rawlins, K., & Van Durme, B. (2015). Semantic proto-roles. Transactions of the Association for Computational Linguistics, 3, 475–488.

    Google Scholar 

  • Rim, K. (2015). Event-participant linking-a crowdsourcing approach to semantic role labeling. Master’s thesis, Brandeis University.

  • Sabou, M., Bontcheva, K., Derczynski, L., & Scharl, A. (2014). Corpus annotation through crowdsourcing: Towards best practice guidelines. In Proceedings of LREC.

  • Saffran, J. R. (2003). Statistical language learning mechanisms and constraints. Current Directions in Psychological Science, 12(4), 110–114.

    Article  Google Scholar 

  • Saffran, J. R., Johnson, E. K., Aslin, R. N., & Newport, E. L. (1999). Statistical learning of tone sequences by human infants and adults. Cognition, 70(1), 27–52.

    Article  Google Scholar 

  • Sahin, G. G. (2016a). Framing of verbs for turkish propbank. In Proceedings of 1st international conference on turkic computational linguistics, TurCLing.

  • Sahin, G. G. (2016b). Verb sense annotation for turkish propbank via crowdsourcing. In Proceedings of 17th international conference on intelligent text processing and computational linguistics. CICLING 2016.

  • Sahin, G. G., & Adalı, E. (2014). Using morphosemantic information in construction of a pilot lexical semantic resource for Turkish. In Workshop on lexical and grammatical resources for language processing (p. 46).

  • Sammons, M., Vydiswaran, V. V., Vieira, T., Johri, N., Chang, M.-W., Goldwasser, D., et al. (2009). Relation alignment for textual entailment recognition. In Text analysis conference (TAC).

  • Schuster, S., & Manning, C. D. (2016). Enhanced english universal dependencies: An improved representation for natural language understanding tasks. In Proceedings of the tenth international conference on language resources and evaluation (LREC 2016).

  • Shen, D., & Lapata, M. (2007). Using semantic roles to improve question answering. In EMNLP-CoNLL (pp. 12–21).

  • Snow, R., O’Connor, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and fast—but is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing (pp. 254–263). Association for Computational Linguistics.

  • Sulubacak, U., Gökırmak, M., & Eryiğit, G. (2016a). Universal dependencies for Turkish. In COLING.

  • Sulubacak, U., Pamay, T., & Eryigit, G. (2016b). Imst: A revisited turkish dependency treebank. In Proceedings of 1st international conference on turkic computational linguistics, TurCLing.

  • Surdeanu, M., Johansson, R., Meyers, A., Màrquez, L., & Nivre, J. (2008). The conll-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of the twelfth conference on computational natural language learning (pp. 159–177). Association for Computational Linguistics.

  • Titov, I., & Klementiev, A. (2011). A bayesian model for unsupervised semantic parsing. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-Volume 1 (pp. 1445–1455). Association for Computational Linguistics.

  • Toutanova, K., Haghighi, A., & Manning, C. D. (2005). Joint learning improves semantic role labeling. In Proceedings of the 43rd annual meeting on association for computational linguistics (pp. 589–596). Association for Computational Linguistics.

  • Wu, D., & Fung, P. (2009). Can semantic role labeling improve SMT. In Proceedings of the 13th annual conference of the EAMT (pp. 218–225).

  • Xue, N., & Palmer, M. (2008). Adding semantic roles to the chinese treebank. Natural Language Engineering, 15(1), 143. (ISSN 1351-3249) .

    Article  Google Scholar 

  • Xue, N., & Palmer, M. (2009). Adding semantic roles to the chinese treebank. Natural Language Engineering, 15(01), 143–172.

    Article  Google Scholar 

  • Zaghouani, W., Diab, M., Mansouri, A., Pradhan, S., & Palmer, M. (2010). The revised Arabic PropBank. In Proceedings of the fourth linguistic annotation workshop (pp. 222–226).

  • Zaidan, O. F., & Callison-Burch, C. (2011). Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-Volume 1 (pp. 1220–1229). Association for Computational Linguistics.

  • Zhai, H., Lingren, T., Deleger, L., Li, Q., Kaiser, M., Stoutenborough, L., et al. (2013). Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. Journal of medical Internet research, 15(4), e73.

    Article  Google Scholar 

  • Zhou, J., & Xu, W. (2015). End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (Volume 1: Long Papers) (pp. 1127–1137).

Download references

Acknowledgements

The first author is funded by Tubitak 2211 domestic Ph.D. Scholarship Program. We would like to sincerely thank Gülşen Eryiğit for fruitful discussions, guidance throughout this work and allowing access to the IMST corpus prior to its public release. Furthermore, we are more than grateful to Celal Şahin, İsmail Hakkı Yadigar, Kurt Bandy and Erdem İşgüder for their invaluable feedback. We are thankful to our anonymous reviewers for their careful reading of our manuscript and their many insightful comments and suggestions. Finally, this work would not be possible without our anonymous crowdworkers from all around the country.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Istanbul Technical University, 34469, Istanbul, Turkey

    Gözde Gül Şahin & Eşref Adalı

Authors
  1. Gözde Gül Şahin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eşref Adalı
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gözde Gül Şahin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Şahin, G.G., Adalı, E. Annotation of semantic roles for the Turkish Proposition Bank. Lang Resources & Evaluation 52, 673–706 (2018). https://doi.org/10.1007/s10579-017-9390-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-017-9390-y

Keywords

Search

Navigation