全员参与：通过大型语言模型进行大规模逐字反馈的问答环节

AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large Language Models

March 22, 2024

作者: Chaoyun Zhang, Zicheng Ma, Yuhao Wu, Shilin He, Si Qin, Minghua Ma, Xiaoting Qin, Yu Kang, Yuyi Liang, Xiaoyu Gou, Yajie Xue, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

cs.AI

摘要

逐字反馈构成了用户体验、意见和需求的宝贵资料库，对软件开发至关重要。从这些数据中有效高效地提取有价值的见解是一项具有挑战性的任务。本文介绍了Allhands，这是一个创新的分析框架，旨在通过自然语言界面进行大规模反馈分析，利用大型语言模型（LLMs）。Allhands遵循传统的反馈分析工作流程，首先对反馈进行分类和主题建模，将其转换为结构化增强格式，整合LLMs以提高准确性、鲁棒性、泛化性和用户友好性。随后，采用LLM代理来解释用户在反馈中提出的多样化问题，将其翻译成Python代码进行执行，并提供全面的多模态响应，包括文本、代码、表格和图片。 我们在三个不同的反馈数据集上评估了Allhands。实验证明，Allhands在所有分析阶段，包括分类和主题建模，都取得了卓越的效果，最终为用户提供了“问我任何事”的体验，提供全面、正确和易读的回应。据我们所知，Allhands是第一个支持通过自然语言界面提取见解的多样化和定制化需求的全面反馈分析框架。

English

Verbatim feedback constitutes a valuable repository of user experiences, opinions, and requirements essential for software development. Effectively and efficiently extracting valuable insights from such data poses a challenging task. This paper introduces Allhands , an innovative analytic framework designed for large-scale feedback analysis through a natural language interface, leveraging large language models (LLMs). Allhands adheres to a conventional feedback analytic workflow, initially conducting classification and topic modeling on the feedback to convert them into a structurally augmented format, incorporating LLMs to enhance accuracy, robustness, generalization, and user-friendliness. Subsequently, an LLM agent is employed to interpret users' diverse questions in natural language on feedback, translating them into Python code for execution, and delivering comprehensive multi-modal responses, including text, code, tables, and images. We evaluate Allhands across three diverse feedback datasets. The experiments demonstrate that Allhands achieves superior efficacy at all stages of analysis, including classification and topic modeling, eventually providing users with an ``ask me anything'' experience with comprehensive, correct and human-readable response. To the best of our knowledge, Allhands stands as the first comprehensive feedback analysis framework that supports diverse and customized requirements for insight extraction through a natural language interface.