WebLINX：具有多輪對話的真實世界網站導航

摘要

我們提出了對話式網頁導航問題，其中數位代理控制網頁瀏覽器，並按照使用者指示以多輪對話方式解決真實世界任務。為支持此問題，我們引入了WEBLINX - 一個包含 100K 互動的大規模基準，跨越 2300 個專家示範的對話式網頁導航。我們的基準涵蓋超過 150 個真實世界網站上的廣泛模式，可用於訓練和評估各種情境下的代理。由於信息量巨大，大型語言模型（LLMs）無法即時處理整個網頁。為解決此瓶頸，我們設計了一個受檢索啟發的模型，通過對相關元素進行排名來有效修剪 HTML 頁面。我們使用所選元素，以及屏幕截圖和操作歷史，來評估各種模型在網頁導航時複製人類行為的能力。我們的實驗從僅限小文本到專有的多模式 LLMS。我們發現，較小的微調解碼器勝過最佳的零-shot LLMS（包括 GPT-4V），但也勝過明確預先訓練過屏幕截圖的較大的微調多模式模型。然而，所有微調模型都難以推廣到未見過的網站。我們的研究結果突顯了需要能夠推廣到新型設置的大型多模式模型。我們的程式碼、數據和模型可供研究使用：https://mcgill-nlp.github.io/weblinx

English

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx

WebLINX：具有多輪對話的真實世界網站導航

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

摘要

Support