全員參與：透過大型語言模型進行大規模逐字回饋問答

AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large Language Models

March 22, 2024

作者: Chaoyun Zhang, Zicheng Ma, Yuhao Wu, Shilin He, Si Qin, Minghua Ma, Xiaoting Qin, Yu Kang, Yuyi Liang, Xiaoyu Gou, Yajie Xue, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

cs.AI

摘要

逐字反饋構成了一個寶貴的用戶經驗、意見和需求存儲庫，對於軟體開發至關重要。從這樣的數據中有效且高效地提取有價值的見解是一項具有挑戰性的任務。本文介紹了Allhands，一個創新的分析框架，旨在通過自然語言界面進行大規模反饋分析，利用大型語言模型（LLMs）。Allhands遵循傳統的反饋分析工作流程，最初對反饋進行分類和主題建模，將其轉換為結構增強格式，並整合LLMs以提高準確性、韌性、泛化性和用戶友好性。隨後，使用LLM代理來解釋用戶在反饋中提出的各種自然語言問題，將其翻譯為Python代碼以執行，並提供全面的多模式響應，包括文本、代碼、表格和圖像。 我們在三個不同的反饋數據集上評估了Allhands。實驗表明，Allhands在所有分析階段，包括分類和主題建模，均實現了卓越的效能，最終為用戶提供了一種“問我任何事”的體驗，提供全面、正確和易讀的響應。據我們所知，Allhands是第一個支持透過自然語言界面提取見解的多樣化和定制化需求的全面反饋分析框架。

English

Verbatim feedback constitutes a valuable repository of user experiences, opinions, and requirements essential for software development. Effectively and efficiently extracting valuable insights from such data poses a challenging task. This paper introduces Allhands , an innovative analytic framework designed for large-scale feedback analysis through a natural language interface, leveraging large language models (LLMs). Allhands adheres to a conventional feedback analytic workflow, initially conducting classification and topic modeling on the feedback to convert them into a structurally augmented format, incorporating LLMs to enhance accuracy, robustness, generalization, and user-friendliness. Subsequently, an LLM agent is employed to interpret users' diverse questions in natural language on feedback, translating them into Python code for execution, and delivering comprehensive multi-modal responses, including text, code, tables, and images. We evaluate Allhands across three diverse feedback datasets. The experiments demonstrate that Allhands achieves superior efficacy at all stages of analysis, including classification and topic modeling, eventually providing users with an ``ask me anything'' experience with comprehensive, correct and human-readable response. To the best of our knowledge, Allhands stands as the first comprehensive feedback analysis framework that supports diverse and customized requirements for insight extraction through a natural language interface.