UsERQA: An LLM-Driven User-Aware Community Question Answering System

Abstract :

In the present era, question-and-answer communities have become vibrant platforms for sharing knowledge. Every year, millions of questions are posted on these forums with the hope of receiving answers from human experts. Nonetheless, many of these questions fail to receive timely or accurate answers due to experts' limited time or being duplicates. In recent years, a large body of research has focused on identifying entailed questions within community archives and using their accepted answers to fulfill the information needs of newly posed questions. Most of these studies match questions syntactically and semantically while resorting to external knowledge injection or increased model complexity to enhance question understanding. However, the critical role that the topics typically explored by questioners play in disambiguating their queries has been overlooked. This research addresses this gap by introducing UsERQA, a novel retrieval-augmented generation (RAG)-based question-answering system incorporating user knowledge. UsERQA utilizes large language models to represent the questioner's knowledge as a sequence of topical tags. In addition, it employs a question entailment recognition process as a post-retrieval strategy, with a new constraint, mandating the alignment between entailed questions and the questioner's knowledge. Afterward, another large language model generates the final answer using the accepted answers of top entailed questions as context. The goal is to imitate human writing patterns and leverage the knowledge contained in human responses to produce high-quality answers. Experimental results on the CQAD-ReQuEST dataset indicate the efficiency of UsERQA in modeling user knowledge and producing more accurate responses than its user-agnostic counterpart.

References:

[1] B. Patra, ‘A survey of Community Question Answering’, CoRR, abs/1705.04009, May 2017.
[2] D. Hoogeveen, L. Wang, T. Baldwin, and K. M. Verspoor, ‘Real and Misflagged Duplicate Question Detection in Community Question-Answering’, 2018.
[3] A. Figueroa, ‘Automatically generating effective search queries directly from community question-answering questions for finding related questions’, Expert Syst. Appl., vol. 77, pp. 11–19, Jul. 2017.
[4] I. Srba and M. Bielikova, ‘A Comprehensive Survey and Classification of Approaches for Community Question Answering’, ACM Trans. Web, vol. 10, no. 3, pp. 1–63, Aug. 2016.
[5] M. Sulír and M. Regeci, ‘Software Engineers’ Questions and Answers on Stack Exchange’, in 2022 IEEE 16th International Scientific Conference on Informatics (Informatics), Nov. 2022, pp. 304–310.
[6] M. Asaduzzaman, A. S. Mashiyat, C. K. Roy, and K. A. Schneider, ‘Answering questions about unanswered questions of Stack Overflow’, in 2013 10th Working Conference on Mining Software Repositories (MSR), May 2013, pp. 97–100.
[7] S. A. Bhaskar, R. Rungta, J. Route, E. Nyberg, and T. Mitamura, ‘Sieg at MEDIQA 2019: Multi-task Neural Ensemble for Biomedical Inference and Entailment’, in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 462–470.
[8] A. Merchant, N. Shenoy, A. Bharali, and A. K. M, ‘Identifying Similar Questions in the Medical Domain Using a Fine-tuned Siamese-BERT Model’, in 2022 IEEE 19th India Council International Conference (INDICON), Nov. 2022, pp. 1–6.
[9] A. B. Abacha and D. Demner-Fushman, ‘A question-entailment approach to question answering’, BMC Bioinformatics, vol. 20, no. 1, Dec. 2019.
[10] Z. Xu and H. Yuan, ‘Forum Duplicate Question Detection by Domain Adaptive Semantic Matching’, IEEE Access, vol. 8, pp. 56029–56038, 2020.
[11] L. Wang, L. Zhang, and J. Jiang, ‘Duplicate Question Detection With Deep Learning in Stack Overflow’, IEEE Access, vol. 8, pp. 25964–25975, 2020.
[12] D. D. Koswatte and S. Hettiarachchi, ‘Optimized Duplicate Question Detection in Programming Community Q&A Platforms using Semantic Hashing’, in 2021 10th International Conference on Information and Automation for Sustainability (ICIAfS), Aug. 2021, pp. 375–380.
[13] V.-T. Nguyen, A.-C. Le, and H.-N. Nguyen, ‘A Model of Convolutional Neural Network Combined with External Knowledge to Measure the Question Similarity for Community Question Answering Systems’, Int. J. Mach. Learn. Comput., vol. 11, no. 3, pp. 194–201, May 2021.
[14] R. Zhang, Q. Zhou, B. Wu, W. Li, and T. Mo, ‘What do questions exactly ask? mfae: duplicate question identification with multi-fusion asking emphasis’, in Proceedings of the 2020 SIAM International Conference on Data Mining, SDM 2020, 2020, pp. 226–234.
[15] D. Hoogeveen, A. Bennett, Y. Li, K. Verspoor, and T. Baldwin, ‘Detecting Misflagged Duplicate Questions in Community Question-Answering Archives’, Proc. Int. AAAI Conf. Web Soc. Media, vol. 12, no. 1, pp. 112–120, Jun. 2018.
[16] P. K. Roy, S. Saumya, J. P. Singh, S. Banerjee, and A. Gutub, ‘Analysis of community question‐answering issues via machine learning and deep learning: State‐of‐the‐art review’, CAAI Trans. Intell. Technol., vol. 8, no. 1, pp. 95–117, Mar. 2023.
[17] N. Othman, R. Faiz, and K. Smaïli, ‘Enhancing Question Retrieval in Community Question Answering Using Word Embeddings’, Procedia Comput. Sci., vol. 159, pp. 485–494, 2019.
[18] G. Zhou, L. Cai, J. Zhao, and K. Liu, ‘Phrase-based translation model for question retrieval in community question answer archives’, ACL-HLT 2011 - Proc. 49th Annu. Meet. Assoc. Comput. Linguist. Hum. Lang. Technol., vol. 1, pp. 653–662, 2011.
[19] K. Mrini et al., ‘A Gradually Soft Multi-Task and Data-Augmented Approach to Medical Question Understanding’, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 1505–1515.
[20] N. S. Tawfik and M. R. Spruit, ‘Evaluating sentence representations for biomedical text: Methods and experimental results’, J. Biomed. Inform., vol. 104, no. February, p. 103396, Apr. 2020.
[21] M. Faseeh, M. A. Khan, N. Iqbal, F. Qayyum, A. Mehmood, and J. Kim, ‘Enhancing User Experience on Q&A Platforms: Measuring Text Similarity Based on Hybrid CNN-LSTM Model for Efficient Duplicate Question Detection’, IEEE Access, vol. 12, no. January, pp. 34512–34526, 2024.
[22] V. Nguyen, S. Karimi, and Z. Xing, ‘Combining Shallow and Deep Representations for Text-Pair Classification’, ALTA 2021 - Proc. 19th Work. Australas. Lang. Technol. Assoc., pp. 68–78, 2021.
[23] S. Ghasemi and A. Shakery, ‘Harnessing the Power of Metadata for Enhanced Question Retrieval in Community Question Answering’, IEEE Access, vol. 12, no. May, pp. 65768–65779, 2024.
[24] A. M. Monea, ‘Medical Question Entailment based on Textual Inference and Fine-tuned BioMed-RoBERTa’, in 2021 IEEE 17th International Conference on Intelligent Computer Communication and Processing (ICCP), 2021, pp. 319–326.
[25] S. Yadav, V. Pallagani, and A. Sheth, ‘Medical Knowledge-enriched Textual Entailment Framework’, in Proceedings of the 28th International Conference on Computational Linguistics, 2020, pp. 1795–1801.
[26] T. R. Goodwin and D. Demner-Fushman, ‘Bridging the Knowledge Gap: Enhancing Question Answering with World and Domain Knowledge’, CoRR abs/1910.07429, Oct. 2019.
[27] A. B. Abacha, D. Demner-fushman, and U. S. N. Library, ‘Recognizing Question Entailment for Medical Question Answering’, in AMIA Annual Symposium Proceedings, 2016, pp. 310–318.
[28] T. Baldwin, H. Liang, B. Salehi, D. Hoogeveen, Y. Li, and L. Duong, ‘UniMelb at SemEval-2016 Task 3: Identifying Similar Questions by combining a CNN with String Similarity Measures’, in Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016, pp. 851–856.
[29] A. B. Abacha and D. Demner-Fushman, ‘On the Summarization of Consumer Health Questions’, in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 2228–2234.
[30] C. dos Santos, L. Barbosa, D. Bogdanova, and B. Zadrozny, ‘Learning Hybrid Representations to Retrieve Semantically Equivalent Questions’, in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2015, vol. 2, no. 1, pp. 694–699.
[31] D. A. Prabowo and G. Budi Herwanto, ‘Duplicate Question Detection in Question Answer Website using Convolutional Neural Network’, in 2019 5th International Conference on Science and Technology (ICST), Jul. 2019, no. 1, pp. 1–6.
[32] A. Agrawal, R. Anil George, S. S. Ravi, S. Kamath S, and A. Kumar, ‘ARS_NITK at MEDIQA 2019:Analysing Various Methods for Natural Language Inference, Recognising Question Entailment and Medical Question Answering System’, in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 533–540.
[33] N. Tawfik and M. Spruit, ‘UU_TAILS at MEDIQA 2019: Learning Textual Entailment in the Medical Domain’, in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 493–499.
[34] V. Nguyen, S. Karimi, and Z. Xing, ‘ANU-CSIRO at MEDIQA 2019: Question Answering Using Deep Contextual Knowledge’, in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 478–487.
[35] D. Bandyopadhyay, B. Gain, T. Saikh, and A. Ekbal, ‘IITP at MEDIQA 2019: Systems Report for Natural Language Inference, Question Entailment and Question Answering’, in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 517–522.
[36] K. Mrini, F. Dernoncourt, W. Chang, E. Farcas, and N. Nakashole, ‘Joint Summarization-Entailment Optimization for Consumer Health Question Understanding’, in Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations, 2021, pp. 58–65.
[37] W. Zhu et al., ‘PANLP at MEDIQA 2019: Pre-trained Language Models, Transfer Learning and Knowledge Distillation’, in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 380–388.
[38] M. Sarrouti, A. B. Abacha, and D. Demner-Fushman, ‘Multi-Task Transfer Learning with Data Augmentation for Recognizing Question Entailment in the Medical Domain’, in 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI), Aug. 2021, pp. 339–346.
[39] H. Zhou, X. Li, W. Yao, C. Lang, and S. Ning, ‘DUT-NLP at MEDIQA 2019: An Adversarial Multi-Task Network to Jointly Model Recognizing Question Entailment and Question Answering’, in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 437–445.
[40] A. Shang, X. Zhu, M. Danner, and M. Rätsch, ‘Unsupervised question-retrieval approach based on topic keywords filtering and multi-task learning’, Comput. Speech Lang., vol. 87, no. March, p. 101644, Aug. 2024.
[41] K. Jiang, G. Jin, Z. Zhang, R. Cui, and Y. Zhao, ‘Incorporating external knowledge for text matching model’, Comput. Speech Lang., vol. 87, no. March, p. 101638, Aug. 2024.
[42] Y. Sun and J. Song, ‘Research on question retrieval method for community question answering’, Multimed. Tools Appl., vol. 82, pp. 24309–24325, 2023.
[43] D. Hoogeveen, K. M. Verspoor, and T. Baldwin, ‘CQADupStack: A Benchmark Data Set for Community Question-Answering Research’, in Proceedings of the 20th Australasian Document Computing Symposium, Dec. 2015, pp. 1–8.
[44] S. Z. Aftabi, S. M. Seyyedi, M. Maleki, and S. Farzi, ‘ReQuEST: A Small-Scale Multi-Task Model for Community Question-Answering Systems’, IEEE Access, vol. 12, pp. 17137–17151, 2024.
[45] C. Y. Lin, ‘Rouge: A package for automatic evaluation of summaries’, Proc. Work. text Summ. branches out (WAS 2004), no. 1, pp. 25–26, 2004, [Online].
[46] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, ‘BERTScore: Evaluating Text Generation with BERT’, CoRR abs/1904.09675, pp. 1–43, Apr. 2019.
[47] K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu, ‘MPNet: Masked and Permuted Pre-training for Language Understanding’, Adv. Neural Inf. Process. Syst., vol. 2020-Decem, no. NeurIPS, pp. 1–11, Apr. 2020.