Plutus：針對低資源希臘語金融領域的大型語言模型基準測試

Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance

February 26, 2025

作者: Xueqing Peng, Triantafillos Papadopoulos, Efstathia Soufleri, Polydoros Giannouris, Ruoyu Xiang, Yan Wang, Lingfei Qian, Jimin Huang, Qianqian Xie, Sophia Ananiadou

cs.AI

摘要

儘管希臘在全球經濟中扮演著關鍵角色，但由於希臘語的語言複雜性及領域特定數據的稀缺，大型語言模型（LLMs）在希臘金融語境中的應用仍未被充分探索。先前在多語言金融自然語言處理（NLP）方面的努力已揭示了顯著的性能差異，然而至今尚未開發出專用的希臘金融基準測試或針對希臘語的金融LLMs。為彌補這一缺口，我們推出了Plutus-ben，首個希臘金融評估基準，以及Plutus-8B，首個基於希臘領域特定數據微調的希臘金融LLM。Plutus-ben涵蓋了希臘語中的五項核心金融NLP任務：數值與文本命名實體識別、問答、摘要生成及主題分類，從而促進了系統化且可重現的LLM評估。為支持這些任務，我們提出了三個全新、高質量的希臘金融數據集，這些數據集由精通希臘語的專家詳細註釋，並輔以兩個現有資源。我們對22個LLMs在Plutus-ben上的全面評估顯示，由於語言複雜性、領域特定術語及金融推理差距，希臘金融NLP仍具挑戰性。這些發現凸顯了跨語言遷移的局限性、希臘語訓練模型中金融專業知識的必要性，以及將金融LLMs適應於希臘文本的挑戰。我們公開釋出Plutus-ben、Plutus-8B及所有相關數據集，以促進可重現的研究並推動希臘金融NLP的發展，從而促進金融領域中更廣泛的多語言包容性。

English

Despite Greece's pivotal role in the global economy, large language models (LLMs) remain underexplored for Greek financial context due to the linguistic complexity of Greek and the scarcity of domain-specific datasets. Previous efforts in multilingual financial natural language processing (NLP) have exposed considerable performance disparities, yet no dedicated Greek financial benchmarks or Greek-specific financial LLMs have been developed until now. To bridge this gap, we introduce Plutus-ben, the first Greek Financial Evaluation Benchmark, and Plutus-8B, the pioneering Greek Financial LLM, fine-tuned with Greek domain-specific data. Plutus-ben addresses five core financial NLP tasks in Greek: numeric and textual named entity recognition, question answering, abstractive summarization, and topic classification, thereby facilitating systematic and reproducible LLM assessments. To underpin these tasks, we present three novel, high-quality Greek financial datasets, thoroughly annotated by expert native Greek speakers, augmented by two existing resources. Our comprehensive evaluation of 22 LLMs on Plutus-ben reveals that Greek financial NLP remains challenging due to linguistic complexity, domain-specific terminology, and financial reasoning gaps. These findings underscore the limitations of cross-lingual transfer, the necessity for financial expertise in Greek-trained models, and the challenges of adapting financial LLMs to Greek text. We release Plutus-ben, Plutus-8B, and all associated datasets publicly to promote reproducible research and advance Greek financial NLP, fostering broader multilingual inclusivity in finance.