Writer 1 / Study Program : Uttsada Jason / Computer Science Writer 2 / Study Program : Daniel Christian Mandolang / Computer Science Writer 3 / Study Program : Nyoo Steven Christopher Handoko / Computer Science Title : AI Chatbot Development for Computer Science Curriculum Based Academic Consultation using Retrieval-Augmented Generation (RAG) Counselor : Denny, S.Kom., M.I.T., Ph.D. Academic consultation is a crucial part of students’ university journey. As the curriculum evolves, there is a growing need for a chatbot application to assist in this process. This development aims to enhance the effectiveness and efficiency of academic consultations. However, due to the nature of large language models (LLMs), which may generate ”hallucinations,” and the application of Retrieval-Augmented Generation (RAG) techniques in chatbot development, our chatbot will adopt this approach. In RAG-based chatbot development, any corpus can be used to obtain context for generation. In this research, the corpora utilized include a vector database and a knowledge graph database. This study focuses on improving performance in terms of answer relevance to questions within the academic consultation context, while also considering response time. Three architectures are tested in our chatbot, such as Hierarchical Tree Retrieval and Graph-Augmented Retrieval which is a vector-based retrieval, as well as Knowledge Graph Retrieval, to answer prerequisite-related questions that are particularly challenging for Collapsed Vector Retrieval. The final chatbot model is integrated into an application deployed on-premises at the Center for Computer Science (Pusilkom) of the Faculty of Computer Science, University of Indonesia. The best retrieval architecture in vector-based retrieval is the Graph-Augmented Retrieval which combines the advantage of vector-based chatbot and the ability to recognize prerequisite information, shown by the relatively high mAP of 0.744 and hit rate of 0.904 that outperforms other vector architectures. Additionally, the Knowledge Graph Retrieval improved the answer relevance significantly in answering prerequisite-related questions, shown by 100% answer accuracy improvement in prerequisite-related questions compared to the base model. Finally, among the models tested, Knowledge Graph Retrieval achieves the highest answer accuracy. However, the Graph-Augmented Retrieval model is considered to be the best for an on-premises chatbot, providing 76% answer accuracy and an average response time of 8.49 seconds. This represents a 6.5 seconds reduction in the response time compared to the Knowledge Graph Retrieval model. All of our chatbot models have been successfully deployed on the specified server and are utilizing all GPU resources optimally. However, the chatbot application is not able to provide an acceptable user experience utilizing the computing resource in the specified server when there are multiple concurrent users (more than one).