LLMs作为人-计算算法中的工作者？使用LLMs复制众包流程
LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs
July 19, 2023
作者: Tongshuang Wu, Haiyi Zhu, Maya Albayrak, Alexis Axon, Amanda Bertsch, Wenxing Deng, Ziqi Ding, Bill Guo, Sireesh Gururaja, Tzu-Sheng Kuo, Jenny T. Liang, Ryan Liu, Ihita Mandal, Jeremiah Milbauer, Xiaolin Ni, Namrata Padmanabhan, Subhashini Ramkumar, Alexis Sudjianto, Jordan Taylor, Ying-Jui Tseng, Patricia Vaidos, Zhijin Wu, Wei Wu, Chenyang Yang
cs.AI
摘要
LLM已显示出在之前被认为是人类专属能力的众包任务中复制人类行为的潜力。然而，目前的努力主要集中在简单的原子任务上。我们探讨LLM是否能够复制更复杂的众包流程。我们发现现代LLM可以模拟一些众包工作者在这些“人类计算算法”中的能力，但成功的程度是不确定的，并受请求者对LLM能力的理解、子任务所需的具体技能以及执行这些子任务的最佳交互模式的影响。我们反思了人类和LLM对指令的不同敏感性，强调了为LLM提供面向人类的保障的重要性，并讨论了训练人类和LLM具有互补技能的潜力。至关重要的是，我们展示了复制众包流程提供了一个宝贵的平台，可以研究LLM在不同任务上的相对优势（通过对它们在子任务上的表现进行交叉比较），以及LLM在复杂任务中的潜力，在这些任务中，它们可以完成部分任务，而将其他任务留给人类。
English
LLMs have shown promise in replicating human-like behavior in crowdsourcing
tasks that were previously thought to be exclusive to human abilities. However,
current efforts focus mainly on simple atomic tasks. We explore whether LLMs
can replicate more complex crowdsourcing pipelines. We find that modern LLMs
can simulate some of crowdworkers' abilities in these "human computation
algorithms," but the level of success is variable and influenced by requesters'
understanding of LLM capabilities, the specific skills required for sub-tasks,
and the optimal interaction modality for performing these sub-tasks. We reflect
on human and LLMs' different sensitivities to instructions, stress the
importance of enabling human-facing safeguards for LLMs, and discuss the
potential of training humans and LLMs with complementary skill sets. Crucially,
we show that replicating crowdsourcing pipelines offers a valuable platform to
investigate (1) the relative strengths of LLMs on different tasks (by
cross-comparing their performances on sub-tasks) and (2) LLMs' potential in
complex tasks, where they can complete part of the tasks while leaving others
to humans.