[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content
Springer Nature Link
Log in

Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models

  • Conference paper
  • First Online:
Knowledge Engineering and Knowledge Management (EKAW 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 15370))

  • 190 Accesses

Abstract

Procedural Knowledge is the know-how expressed in the form of sequences of steps needed to perform some tasks. Procedures are usually described by means of natural language texts, such as recipes or maintenance manuals, possibly spread across different documents and systems, and their interpretation and subsequent execution is often left to the reader. Representing such procedures in a Knowledge Graph (KG) can be the basis to build digital tools to support those users who need to apply or execute them.

In this paper, we leverage Large Language Model (LLM) capabilities and propose a prompt engineering approach to extract steps, actions, objects, equipment and temporal information from a textual procedure, in order to populate a Procedural KG according to a pre-defined ontology. We evaluate the KG extraction results by means of a user study, in order to qualitatively and quantitatively assess the perceived quality and usefulness of the LLM-extracted procedural knowledge. We show that LLMs can produce outputs of acceptable quality and we assess the subjective perception of AI by human evaluators.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 49.99
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 64.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Cf. https://lm-kbc.github.io/.

  2. 2.

    Cf. https://github.com/cefriel/procedural-kg-llm.

  3. 3.

    Cf. http://www.sparontologies.net/.

  4. 4.

    p-plan: http://purl.org/net/p-plan#

    khub-proc: https://knowledge.c-innovationhub.com/k-hub/procedure#

    frapo: http://purl.org/cerif/frapo/

    time: http://www.w3.org/2006/time#.

  5. 5.

    Cf. https://github.com/zharry29/wikihow-goal-step.

  6. 6.

    The actual input texts of the procedures used for the method replication and human assessment can be found on GitHub.

  7. 7.

    Cf. https://www.langchain.com/.

  8. 8.

    Cf. https://www.prolific.com/.

  9. 9.

    We preferred a non-parametric test (instead of the well-known parametric t-test), because we could not assume a normal distribution of the population.

References

  1. Allen, B.P., Stork, L., Groth, P.: Knowledge engineering using large language models. TGDK (2023). https://doi.org/10.4230/TGDK.1.1.3

    Article  Google Scholar 

  2. Babaei Giglou, H., DSouza, J., Auer, S.: LLMs4OL: large language models for ontology learning. In: Payne, T.R., et al. (eds.) ISWC 2023. LNCS, vol. 14265, pp. 408–427. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-47240-4_22

  3. Bellan, P., Dragoni, M., Ghidini, C.: Extracting business process entities and relations from text using pre-trained language models and in-context learning. In: Almeida, J.P.A., Karastoyanova, D., Guizzardi, G., Montali, M., Maggi, F.M., Fonseca, C.M. (eds.) EDOC 2022. Lecture Notes in Computer Science, vol. 13585, pp. 182–199. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-17604-3_11

    Chapter  Google Scholar 

  4. Bellan, P., Dragoni, M., Ghidini, C., van der Aa, H., Ponzetto, S.P.: Process extraction from text: benchmarking the state of the art and paving the way for future challenges. arXiv preprint arXiv:2110.03754 (2021)

  5. Bischof, S., Filtz, E., Parreira, J.X., Steyskal, S.: LLM-based guided generation of ontology term definitions. In: Proceedings of EKAW 2024 - Industry Track. Springer, Cham (2024)

    Google Scholar 

  6. Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)

    Google Scholar 

  7. Chan, C.M., et al.: ChatEval: towards better LLM-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201 (2023)

  8. Ciroku, F., de Berardinis, J., Kim, J., Meroño-Peñuela, A., Presutti, V., Simperl, E.: RevOnt: reverse engineering of competency questions from knowledge graphs via language models. J. Web Semant. 100822 (2024)

    Google Scholar 

  9. Ding, N., et al.: Few-nerd: a few-shot named entity recognition dataset. arXiv preprint arXiv:2105.07464 (2021)

  10. Dubois, Y., et al.: AlpacaFarm: a simulation framework for methods that learn from human feedback. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

  11. Fathallah, N., Das, A., De Giorgis, S., Poltronieri, A., Haase, P., Kovriguina, L.: Neon-GPT: a large language model-powered pipeline for ontology learning. In: Proceedings of ESWC 2024 - Special Track on Large Language Models for Knowledge Engineering (2024)

    Google Scholar 

  12. Frey, J., Meyer, L.P., Arndt, N., Brei, F., Bulert, K.: Benchmarking the abilities of large language models for RDF knowledge graph creation and comprehension: how well do LLMs speak turtle?. arXiv:2309.17122 (2023)

  13. Fu, J., Ng, S.K., Jiang, Z., Liu, P.: GPTscore: evaluate as you desire. arXiv preprint arXiv:2302.04166 (2023)

  14. Garijo, D., Gil, Y.: Augmenting PROV with plans in P-PLAN: scientific processes as linked data. In: Kauppinen, T., Pouchard, L.C., Keßler, C. (eds.) Proceedings of LISC2012 Workshop - Co-located with ISWC 2012. CEUR Workshop Proceedings, vol. 951. CEUR-WS.org (2012)

    Google Scholar 

  15. Groth, P., Simperl, E., van Erp, M., Vrandečič, D.: Knowledge graphs and their role in the knowledge engineering of the 21st century (Dagstuhl Seminar 22372). DROPS-IDN/v2/document/10.4230/DagRep.12.9.60 (2023). https://doi.org/10.4230/DagRep.12.9.60, publisher: Schloss-Dagstuhl - Leibniz Zentrum für Informatik

  16. Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Meas. 1(1), 77–89 (2007)

    Article  Google Scholar 

  17. Hidalgo, C.A., Orghian, D., Canals, J.A., De Almeida, F., Martin, N.: How Humans Judge Machines. MIT Press, Cambridge (2021)

    Google Scholar 

  18. Hofer, M., Frey, J., Rahm, E.: Towards self-configuring knowledge graph construction pipelines using LLMs - a case study with RML. In: Proceedings of KGCW 2024 co-located with ESWC2024. CEUR Workshop Proceedings, vol. 3718. CEUR-WS.org (2024)

    Google Scholar 

  19. Hoseini, S., Burgdorf, A., Paulus, A., Meisen, T., Quix, C., Pomp, A.: Towards LLM-augmented creation of semantic models for dataspaces. In: Proceedings of SDS 2024 Workshop Co-located with ESWC 2024 (2024)

    Google Scholar 

  20. Kamoi, R., Goyal, T., Rodriguez, J.D., Durrett, G.: Wice: real-world entailment for claims in Wikipedia. arXiv preprint arXiv:2303.01432 (2023)

  21. Kumar, A., Pandey, A., Gadia, R., Mishra, M.: Building knowledge graph using pre-trained language model for learning entity-aware relationships. In: Proceedings of 2020 IEEE International Conference on Computing, Power and Communication Technologies (GUCON), pp. 310–315 (2020). https://doi.org/10.1109/GUCON48875.2020.9231227

  22. Kumar, A., Gupta, N., Dana, S.: Constructing micro knowledge graphs from technical support documents. In: Verborgh, R., et al. (eds.) ESWC 2021. LNCS, vol. 12739, pp. 249–253. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-80418-3_37

    Chapter  Google Scholar 

  23. Lee, M., et al.: Evaluating human-language model interaction. arXiv preprint arXiv:2212.09746 (2022)

  24. Li, M., et al.: Coannotating: uncertainty-guided work allocation between human and large language models for data annotation. arXiv preprint arXiv:2310.15638 (2023)

  25. Li, X., et al.: Knowledge-centric prompt composition for knowledge base construction from pre-trained language models. In: KBC-LM/LM-KBC@ ISWC (2023)

    Google Scholar 

  26. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C.: G-Eval: NLG evaluation using GPT-4 with better human alignment. arXiv preprint arXiv:2303.16634 (2023)

  27. Liu, Y., et al.: Revisiting the gold standard: grounding summarization evaluation with robust human evaluation. arXiv preprint arXiv:2212.07981 (2022)

  28. Magni, F., Park, J., Chao, M.M.: Humans as creativity gatekeepers: are we biased against AI creativity? J. Bus. Psychol. 1–14 (2023)

    Google Scholar 

  29. Marreddy, M., Oota, S.R., Gupta, M., Flek, L.: Large language models are human-like annotators (2024). https://sites.google.com/view/lllms-as-human-like-annotators/. Tutorial at KR 2024

  30. McKight, P.E., Najab, J.: Kruskal-Wallis test. The Corsini Encyclopedia of Psychology, p. 1 (2010)

    Google Scholar 

  31. Mihindukulasooriya, N., Tiwari, S., Enguix, C.F., Lata, K.: Text2KGBench: a benchmark for ontology-driven knowledge graph generation from text. In: Payne, T.R., et al. (eds.) ISWC 2023. LNCS, vol. 14266, pp. 247–265. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-47243-5_14

    Chapter  Google Scholar 

  32. Millet, K., Buehler, F., Du, G., Kokkoris, M.D.: Defending humankind: anthropocentric bias in the appreciation of AI art. Comput. Hum. Behav. 143, 107707 (2023)

    Article  Google Scholar 

  33. Nabizadeh, N., Kolossa, D., Heckmann, M.: Myfixit: an annotated dataset, annotation tool, and baseline methods for information extraction from repair manuals. In: Proceedings of LREC 2020, pp. 2120–2128. European Language Resources Association (2020)

    Google Scholar 

  34. Neudert, L.M., Knuutila, A., Howard, P.N.: Global Attitudes Towards AI, Machine Learning & Automated Decision Making. Oxford Commission on AI and Good Governance, Oxford (2020)

    Google Scholar 

  35. Ni, J., Shi, M., Stammbach, D., Sachan, M., Ash, E., Leippold, M.: Afacta: Assisting the annotation of factual claim detection with reliable llm annotators. arXiv preprint arXiv:2402.11073 (2024)

  36. Palan, S., Schitter, C.: Prolific.ac a subject pool for online experiments. J. Behav. Exp. Finance 17, 22–27 (2018)

    Google Scholar 

  37. Pan, J.Z., et al.: Large language models and knowledge graphs: opportunities and challenges. DROPS-IDN/v2/document/10.4230/TGDK.1.1.2 (2023). https://doi.org/10.4230/TGDK.1.1.2

  38. Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., Wu, X.: Unifying large language models and knowledge graphs: a roadmap. IEEE Trans. Knowl. Data Eng. 1–20 (2024)

    Google Scholar 

  39. Ragot, M., Martin, N., Cojean, S.: AI-generated vs. human artworks. a perception bias towards artificial intelligence? In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–10 (2020)

    Google Scholar 

  40. Rula, A., Calegari, G.R., Azzini, A., Bucci, D., Baroni, I., Celino, I.: Eliciting and curating procedural knowledge in industry: challenges and opportunities. In: Proceedings of (Qurator 2022). CEUR Workshop Proceedings, vol. 3234. CEUR-WS.org (2022)

    Google Scholar 

  41. Rula, A., D’Souza, J.: Procedural text mining with large language models. In: Proceedings of the 12th Knowledge Capture Conference 2023. K-CAP ’23, pp. 9–16. Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3587259.3627572

  42. Rula, A., Re Calegari, G., Azzini, A., Baroni, I., Celino, I.: K-hub: a modular ontology to support document retrieval and knowledge extraction in industry 5.0. In: Pesquita, C., et al. (eds.) ESWC 2023. LNCS, vol. 13870, pp. 454–470. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-33455-9_27

  43. Shi, S., Xu, Z., Hu, B., Zhang, M.: Generative multimodal entity linking. In: Proceedings of LREC-COLING 2024, pp. 7654–7665. ELRA and ICCL (2024)

    Google Scholar 

  44. Sola, D., van der Aa, H., Meilicke, C., Stuckenschmidt, H.: Activity recommendation for business process modeling with pre-trained language models. In: Pesquita, C., et al. (eds.) ESWC 2023. LNCS, vol. 13870, pp. 316–334. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-33455-9_19

    Chapter  Google Scholar 

  45. Su, H., et al.: Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975 (2022)

  46. Wei, J., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837 (2022)

    Google Scholar 

  47. Yang, D., Wang, X., Celebi, R.: Expanding the vocabulary of BERT for knowledge base construction (2023). https://doi.org/10.48550/arXiv.2310.08291

  48. Yuan, P., et al.: BatchEval: towards human-like text evaluation. arXiv preprint arXiv:2401.00437 (2023)

  49. Zhang, B., et al.: OntoChat: a framework for conversational ontology engineering using language models. In: Proceedings of ESWC 2024 - Special Track on Large Language Models for Knowledge Engineering (2024)

    Google Scholar 

  50. Zhang, B., Reklos, I., Jain, N., Peñuela, A.M., Simperl, E.: Using large language models for knowledge engineering (LLMKE): a case study on Wikidata (2023). https://doi.org/10.48550/arXiv.2309.08491

  51. Zhang, L.: Reasoning about procedures with natural language processing: a tutorial (2022). https://doi.org/10.48550/arXiv.2205.07455, publication Title: arXiv e-prints ADS Bibcode: 2022arXiv220507455Z

  52. Zhang, L., Lyu, Q., Callison-Burch, C.: Reasoning about goals, steps, and temporal ordering with WikiHow. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (2020). https://doi.org/10.18653/v1/2020.emnlp-main.374

  53. Zhou, Y., Shah, J., Schockaert, S.: Learning household task knowledge from WikiHow descriptions. In: Espinosa-Anke, L., Declerck, T., Gromann, D., Camacho-Collados, J., Pilehvar, M.T. (eds.) Proceedings of SemDeep-5 2019 Workshop, pp. 50–56. Association for Computational Linguistics, Macau, China (2019). https://aclanthology.org/W19-5808

  54. Zlatkov, D., Ens, J., Pasquier, P.: Searching for human bias against AI-composed music. In: Johnson, C., Rodríguez-Fernández, N., Rebelo, S.M. (eds.) EvoMUSART 2023. LNCS, vol. 13988, pp. 308–323. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-29956-8_20

    Chapter  Google Scholar 

Download references

Acknowledgements

This work is partially supported by the PERKS project, co-funded by the European Commission (Grant id 101070186). The authors would like to thank all the participants to the human assessment reported in this paper.

Author information

Authors and Affiliations

  1. Cefriel – Politecnico di Milano, viale Sarca 226, 20126, Milano, Italy

    Valentina Anita Carriero, Antonia Azzini, Ilaria Baroni, Mario Scrocca & Irene Celino

Authors
  1. Valentina Anita Carriero
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonia Azzini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ilaria Baroni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mario Scrocca
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Irene Celino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valentina Anita Carriero .

Editor information

Editors and Affiliations

  1. Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France

    Mehwish Alam

  2. Università degli Studi di Verona, Verona, Italy

    Marco Rospocher

  3. KNAW Humanities Cluster, Amsterdam, The Netherlands

    Marieke van Erp

  4. Centrum Wiskunde and Informatica, Amsterdam, The Netherlands

    Laura Hollink

  5. FIZ Karlsruhe, Eggenstein-Leopoldshafen, Germany

    Genet Asefa Gesese

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Carriero, V.A., Azzini, A., Baroni, I., Scrocca, M., Celino, I. (2025). Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models. In: Alam, M., Rospocher, M., van Erp, M., Hollink, L., Gesese, G.A. (eds) Knowledge Engineering and Knowledge Management. EKAW 2024. Lecture Notes in Computer Science(), vol 15370. Springer, Cham. https://doi.org/10.1007/978-3-031-77792-9_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-77792-9_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-77791-2

  • Online ISBN: 978-3-031-77792-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation