在生成式人工智慧應用程式中自動測量負責任人工智慧危害的框架
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
October 26, 2023
作者: Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
cs.AI
摘要
我們提出了一個框架，用於自動測量大型語言模型（LLMs）及其相關產品和服務的負責任人工智慧（RAI）指標。我們的自動測量LLMs造成的損害的框架建立在現有的技術和社會技術專業知識之上，並利用最先進的LLMs（如GPT-4）的能力。我們使用這個框架運行幾個案例研究，探討不同LLMs可能如何違反一系列與RAI相關的原則。該框架可以與特定領域的社會技術專業知識一起使用，以便未來為新的損害領域創建測量標準。通過實施這個框架，我們旨在促進更先進的損害測量工作，進一步推動LLMs的負責使用。
English
We present a framework for the automated measurement of responsible AI (RAI)
metrics for large language models (LLMs) and associated products and services.
Our framework for automatically measuring harms from LLMs builds on existing
technical and sociotechnical expertise and leverages the capabilities of
state-of-the-art LLMs, such as GPT-4. We use this framework to run through
several case studies investigating how different LLMs may violate a range of
RAI-related principles. The framework may be employed alongside domain-specific
sociotechnical expertise to create measurements for new harm areas in the
future. By implementing this framework, we aim to enable more advanced harm
measurement efforts and further the responsible use of LLMs.