Department of Cultural Technology and Communications, Intelligent Systems Lab, University of the Aegean, University Hillhttps://ror.org/03zsp3p94, 81100 Lesvos, Greece"/> Department of Digital Systems, AI Lab, Gr. Lampraki 126, University of Piraeus, Piraeus, Greece"/>
Search
  • All Journals
Advanced Search
Search articles by Title, Author, Keywords, DOI
Submit Manuscript
7.7CiteScore
2.0Impact Factor
Advanced Search
MAP  >  The Knowledge Engineering Review
2025 Volume 40
Article Contents
Next Previous
RESEARCH ARTICLE   Open Access    

Automating agentic collaborative ontology engineering with role-playing simulation of LLM-powered agents and RAG technology

  • Department of Cultural Technology and Communications, Intelligent Systems Lab, University of the Aegean, University Hillhttps://ror.org/03zsp3p94, 81100 Lesvos, Greece

  • Department of Digital Systems, AI Lab, Gr. Lampraki 126, University of Piraeus, Piraeus, Greece

More Information
  • Abstract: Motivated by the astonishing capabilities of large language models (LLMs) in text-generation, reasoning, and simulation of complex human behaviors, in this paper, we propose a novel multi-component LLM-based framework, namely LLM4ACOE, that fully automates the collaborative ontology engineering (COE) process using role-playing simulation of LLM agents and retrieval augmented generation (RAG) technology. The proposed solution enhances the LLM-powered role-playing simulation with RAG ‘feeding’ the LLM with three different types of external knowledge. This knowledge corresponds to the knowledge required by each of the COE roles (agents), using a component-based framework, as follows: (a) domain-specific data-centric documents, (b) OWL documentation, and (c) ReAct guidelines. The aforementioned components are evaluated in combination, with the aim of investigating their impact on the quality of generated ontologies. The aim of this work is twofold, (a) to identify the capacity of LLM-based agents to generate acceptable (by human-experts) ontologies through agentic collaborative ontology engineering (ACOE) role-playing simulation, at specific levels of acceptance (accuracy, validity, and expressiveness of ontologies) without human intervention and (b) to investigate whether and/or to what extent the selected RAG components affect the quality of the generated ontologies. The evaluation of this novel approach is performed using ChatGPT-o in the domain of search and rescue (SAR) missions. To assess the generated ontologies, quantitative and qualitative measures are employed, focusing on coverage, expressiveness, structure, and human involvement.
  • 加载中
  • Argyle , L. P., Busby , E. C., Fulda , N., Gubler , J., Rytting , C. M. & Wingate , D. 2022. Out of one, many: Using language models to simulate human samples. https://doi.org/10.48550/arXiv.2209.06899

    Google Scholar

    Avila , C. V. S., Vidal , V. M. P., Franco , W. & Casanova , M. A. 2024. Experiments with text-to-sparql based on chatgpt. In 18th IEEE International Conference on Semantic Computing, ICSC 2024, Laguna Hills, CA, USA, February 5–7, 2024, 277–284. IEEE. https://doi.org/10.1109/ICSC59802.2024.00050

    Google Scholar

    Chang , K. K., Cramer , M., Soni , S. & Bamman , D. 2023. Speak, memory: An archaeology of books known to chatgpt/gpt-4. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6–10, 2023, Bouamor , H., Pino , J. & Bali , K. (eds), 7312–7327. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.453

    Google Scholar

    Chen , N., Deng , Y. & Li , J. 2024. The oscars of AI theater: A survey on role-playing with language models. https://doi.org/10.48550/arXiv.2407.11484

    Google Scholar

    DeBellis , M., Duttab , N., Ginoc , J. & Balajid , A. 2024. Integrating ontologies and large language models to implement retrieval augmented generation (rag). Applied Ontology 1, 1–5.

    Google Scholar

    Doumanas , D., Bouchouras , G., Soularidis , A., Kotis , K. & Vouros , G. 2025. From human- to llm-centered collaborative ontology engineering. Applied Ontology. https://doi.org/10.1177/15705838241305067

    Google Scholar

    Doumanas , D., Soularidis , A., Kotis , K. & Vouros , G. A. 2024. Integrating llms in the engineering of a SAR ontology. In Artificial Intelligence Applications and Innovations - 20th IFIP WG 12.5 International Conference, AIAI 2024, Corfu, Greece, June 27–30, 2024, Proceedings, Part IV, Maglogiannis , I., Iliadis , L. S., MacIntyre , J., Avlonitis , M. & Papaleonidas , A. (eds), IFIP Advances in Information and Communication Technology 714, 360–374. Springer. https://doi.org/10.1007/978-3-031-63223-5_27

    Google Scholar

    Fathallah , N., Das , A., De Giorgis , S., Poltronieri , A., Haase , P. & Kovriguina , L. 2024. Neon-gpt: a large language model-powered pipeline for ontology learning. In Extended Semantic Web Conference, ESWC2024. Hersonissos, Greece.

    Google Scholar

    Filippas , A., Horton , J. J. & Manning , B. S. 2024. Large language models as simulated economic agents: What can we learn from homo silicus?. In Proceedings of the 25th ACM Conference on Economics and Computation, EC 2024, New Haven, CT, USA, July 8–11, 2024, Bergemann , D., Kleinberg , R. & Sabán , D. (eds), 614–615. ACM. https://doi.org/10.1145/3670865.3673513

    Google Scholar

    Glimm , B., Horrocks , I., Motik , B., Stoilos , G. & Wang , Z. 2014. Hermit: An owl 2 reasoner. Journal of Automated Reasoning 53, 245–269.

    Google Scholar

    Gui , G. & Toubia , O. 2023. The challenge of using llms to simulate human behavior: A causal inference perspective. https://doi.org/10.48550/arXiv.2312.15524

    Google Scholar

    Hu , Z., Feng , Y., Luu , A. T., Hooi , B. & Lipani , A. 2023. Unlocking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21–25, 2023, Frommholz , I., Hopfgartner , F., Lee , M., Oakes , M., Lalmas , M., Zhang , M. & Santos , R. L. T. (eds), 3953–3957. ACM. https://doi.org/10.1145/3583780.3615220

    Google Scholar

    Huang , K., Meng , X., Zhang , J., Liu , Y., Wang , W., Li , S. & Zhang , Y. 2023. An empirical study on fine-tuning large language models of code for automated program repair. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11–15, 2023, 1162–1174. IEEE. https://doi.org/10.1109/ASE56229.2023.00181

    Google Scholar

    Kommineni , V. K., König-Ries , B. & Samuel , S. 2024. From human experts to machines: An LLM supported approach to ontology and knowledge graph construction. https://doi.org/10.48550/arXiv.2403.08345

    Google Scholar

    Kosinski , M. 2024. Evaluating large language models in theory of mind tasks. Proceedings of the National Academy of Sciences 121(45), e2405460121.

    Google Scholar

    Kotis , K. & Vouros , G. A. 2006. Human-centered ontology engineering: The HCOME methodology. Knowledge and Information Systems 10(1), 109–131. https://doi.org/10.1007/s10115-005-0227-4

    Google Scholar

    Litaina , T., Soularidis , A., Bouchouras , G., Kotis , K. & Kavakli , E. 2024. Towards llm-based semantic analysis of historical legal documents. In SemDH@ESWC. https://ceur-ws.org/Vol-3724/short2.pdf

    Google Scholar

    Liu , Y., Yao , Y., Ton , J., Zhang , X., Guo , R., Cheng , H., Klochkov , Y., Taufiq , M. F. & Li , H. 2023. Trustworthy llms: A survey and guideline for evaluating large language models’ alignment. https://doi.org/10.48550/arXiv.2308.05374

    Google Scholar

    Lo , A., Jiang , A. Q., Li , W. & Jamnik , M. 2024. End-to-end ontology learning with large language models. https://doi.org/10.48550/arXiv.2410.23584

    Google Scholar

    Masa , P., Meditskos , G., Kintzios , S., Vrochidis , S. & Kompatsiaris , I. 2022. Ontology-based modelling and reasoning for forest fire emergencies in resilient societies. In SETN 2022: 12th Hellenic Conference on Artificial Intelligence, Corfu, Greece, September 7–9, 2022, 24:1–24:9. ACM. https://doi.org/10.1145/3549737.3549765

    Google Scholar

    Mateiu , P. & Groza , A. 2023. Ontology engineering with large language models. In 25th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, SYNASC 2023, Nancy, France, September 11–14, 2023, 226–229. IEEE. https://doi.org/10.1109/SYNASC61333.2023.00038

    Google Scholar

    Pan , X., van Ossenbruggen , J., de Boer , V. & Huang , Z. 2024. A RAG approach for generating competency questions in ontology engineering. https://doi.org/10.48550/arXiv.2409.08820

    Google Scholar

    Paparidis , E. & Kotis , K. 2021. Towards engineering fair ontologies: Uunbiasing a surveillance ontology. In 2021 IEEE International Conference on Progress in Informatics and Computing (PIC), 226–231. IEEE.

    Google Scholar

    Park , J. S., O’Brien , J. C., Cai , C. J., Morris , M. R., Liang , P. & Bernstein , M. S. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST 2023, San Francisco, CA, USA, 29 October 2023–1 November 2023, Follmer , S., Han , J., Steimle , J. & Riche , N. H. (eds), 2:1–2:22. ACM. https://doi.org/10.1145/3586183.3606763

    Google Scholar

    Perkovic , G., Drobnjak , A. & Boticki , I. 2024. Hallucinations in llms: Understanding and addressing challenges. In 47th MIPRO ICT and Electronics Convention, MIPRO 2024, Opatija, Croatia, May 20–24, 2024, Babic , S., Car , Z., Cicin-Sain , M., Cisic , D., Ergovic , P., Grbac , T. G., Gradisnik , V., Gros , S., Jokic , A., Jovic , A., Jurekovic , D., Katulic , T., Koricic , M., Mornar , V., Petrovic , J., Skala , K., Skvorc , D., Sruk , V., Svaco , M., Tijan , E., Vrcek , N. & Vrdoljak , B. (eds), 2084–2088. IEEE. https://doi.org/10.1109/MIPRO60963.2024.10569238

    Google Scholar

    Poveda-Villalón , M., Gómez-Pérez , A. & Suárez-Figueroa , M. C. 2014. Oops! (ontology pitfall scanner!): An on-line tool for ontology evaluation. International Journal on Semantic Web & Information Systems 10(2), 7–34. https://doi.org/10.4018/ijswis.2014040102

    Google Scholar

    Reddy , G. P., Pavan Kumar , Y. V. & Prakash , K. P. 2024. Hallucinations in large language models (llms). In 2024 IEEE Open Conference of Electrical, Electronic and Information Sciences (eStream), 1–6.

    Google Scholar

    Salemi , A., Mysore , S., Bendersky , M. & Zamani , H. 2024. Lamp: When large language models meet personalization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11–16, 2024, Ku , L., Martins , A. & Srikumar , V. (eds), 7370–7392. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.399

    Google Scholar

    Shanahan , M., McDonell , K. & Reynolds , L. 2023. Role play with large language models. Nature 623(7987), 493–498. https://doi.org/10.1038/s41586-023-06647-8

    Google Scholar

    Soularidis , A., Kotis , K., Lamolle , M., Mejdoul , Z., Lortal , G. & Vouros , G. 2024. Llm-assisted generation of swrl rules from natural language. In 2024 International Conference on AI x Data and Knowledge Engineering (AIxDKE), 7–12.

    Google Scholar

    Updyke , D., Podnar , T. & Huff , S. 2023. Simulating realistic human activity using large language model directives.

    Google Scholar

    Yao , S., Zhao , J., Yu , D., Du , N., Shafran , I., Narasimhan , K. R. & Cao , Y. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1–5, 2023. OpenReview.net. https://openreview.net/forum?id=WE_vluYUL-X

    Google Scholar

    Zhang , B., Carriero , V. A., Schreiberhuber , K., Tsaneva , S., González , L. S., Kim , J. & de Berardinis , J. 2024. Ontochat: A framework for conversational ontology engineering using language models. https://doi.org/10.48550/arXiv.2403.05921

    Google Scholar

    Zhao , C., Agrawal , G., Kumarage , T., Tan , Z., Deng , Y., Chen , Y. & Liu , H. 2024. Ontology-aware RAG for improved question-answering in cybersecurity education. https://doi.org/10.48550/arXiv.2412.14191

    Google Scholar

  • Cite this article

    Andreas Soularidis, Dimitrios Doumanas, Konstantinos Kotis, George A. Vouros. 2025. Automating agentic collaborative ontology engineering with role-playing simulation of LLM-powered agents and RAG technology. The Knowledge Engineering Review 40(1), doi: 10.1017/S026988892510009X
    Andreas Soularidis, Dimitrios Doumanas, Konstantinos Kotis, George A. Vouros. 2025. Automating agentic collaborative ontology engineering with role-playing simulation of LLM-powered agents and RAG technology. The Knowledge Engineering Review 40(1), doi: 10.1017/S026988892510009X
    shu
Download PDF

Article Metrics

Article views(201) PDF downloads(609)

Other Articles By Authors

RESEARCH ARTICLE   Open Access    

Automating agentic collaborative ontology engineering with role-playing simulation of LLM-powered agents and RAG technology

  • Department of Cultural Technology and Communications, Intelligent Systems Lab, University of the Aegean, University Hillhttps://ror.org/03zsp3p94, 81100 Lesvos, Greece

  • Department of Digital Systems, AI Lab, Gr. Lampraki 126, University of Piraeus, Piraeus, Greece

  • Received: 05 February 2025
  • Revised: 20 November 2025
  • Accepted: 21 November 2025
  • Published online: 19 December 2025
The Knowledge Engineering Review  40,  2025, 40(1)  |  Cite this article

Abstract: Abstract: Motivated by the astonishing capabilities of large language models (LLMs) in text-generation, reasoning, and simulation of complex human behaviors, in this paper, we propose a novel multi-component LLM-based framework, namely LLM4ACOE, that fully automates the collaborative ontology engineering (COE) process using role-playing simulation of LLM agents and retrieval augmented generation (RAG) technology. The proposed solution enhances the LLM-powered role-playing simulation with RAG ‘feeding’ the LLM with three different types of external knowledge. This knowledge corresponds to the knowledge required by each of the COE roles (agents), using a component-based framework, as follows: (a) domain-specific data-centric documents, (b) OWL documentation, and (c) ReAct guidelines. The aforementioned components are evaluated in combination, with the aim of investigating their impact on the quality of generated ontologies. The aim of this work is twofold, (a) to identify the capacity of LLM-based agents to generate acceptable (by human-experts) ontologies through agentic collaborative ontology engineering (ACOE) role-playing simulation, at specific levels of acceptance (accuracy, validity, and expressiveness of ontologies) without human intervention and (b) to investigate whether and/or to what extent the selected RAG components affect the quality of the generated ontologies. The evaluation of this novel approach is performed using ChatGPT-o in the domain of search and rescue (SAR) missions. To assess the generated ontologies, quantitative and qualitative measures are employed, focusing on coverage, expressiveness, structure, and human involvement.

    HTML

    • The ReAct framework is not incorporated into the proposed ACOE approach, instead, the ReAct approached is followed.

    • https://python.langchain.com/docs/introduction/.

    • https://www.langchain.com/langgraph.

    • https://www.langchain.com/langsmith.

    • https://github.com/AndreasSoularidis/LLM-based-OE-Framework-LC3.

    • https://www.w3.org/2007/OWL/draft/owl2-primer/.

    • http://www.opengis.net/ont/swe/2.0.

    Rights and permissions
    • This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike licence (https://creativecommons.org/licenses/by-nc-sa/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the same Creative Commons licence is used to distribute the re-used or adapted article and the original article is properly cited. The written permission of Cambridge University Press or the rights holder(s) must be obtained prior to any commercial use.
References (34)
  • About this article
    Cite this article
    Andreas Soularidis, Dimitrios Doumanas, Konstantinos Kotis, George A. Vouros. 2025. Automating agentic collaborative ontology engineering with role-playing simulation of LLM-powered agents and RAG technology. The Knowledge Engineering Review 40(1), doi: 10.1017/S026988892510009X
    Andreas Soularidis, Dimitrios Doumanas, Konstantinos Kotis, George A. Vouros. 2025. Automating agentic collaborative ontology engineering with role-playing simulation of LLM-powered agents and RAG technology. The Knowledge Engineering Review 40(1), doi: 10.1017/S026988892510009X
    shu

    • Catalog

      /

      DownLoad:  Full-Size Img  PowerPoint
      Return
      Return