系统日志严重性分类任务中轻量化语言模型与轻量化推理语言模型的基准测试

摘要

系统日志对监控和诊断现代计算基础设施至关重要，但其规模与复杂性需要可靠高效的自动化解读。由于严重级别是系统日志消息中预定义的元数据，仅让模型对其进行分类的独立实用价值有限，难以揭示其底层日志解读能力。我们认为，将严重性分类作为探究运行时日志理解能力的基准测试，比作为最终任务更具参考价值。基于Linux生产服务器的真实journalctl数据，我们在零样本、少样本和检索增强生成（RAG）提示下评估了九款小型语言模型（SLM）和小型推理语言模型（SRLM）。结果呈现显著的分层现象：Qwen3-4B在RAG加持下以95.64%准确率居首，而Gemma3-1B从少样本提示的20.25%提升至RAG下的85.28%。值得注意的是，微型模型Qwen3-0.6B在无检索时表现薄弱，却仍达到88.12%的准确率。相反，包括Qwen3-1.7B和DeepSeek-R1-Distill-Qwen-1.5B在内的多款SRLM与RAG结合后性能大幅下降。效率测试进一步区分模型：多数Gemma和Llama变体单条日志推理耗时低于1.2秒，而Phi-4-Mini-Reasoning单条耗时超228秒却仅获不足10%的准确率。这些发现表明：（1）架构设计，（2）训练目标，（3）在严格输出约束下整合检索上下文的能力共同决定模型性能。通过聚焦可部署的小型模型，该基准测试符合数字孪生（DT）系统的实时需求，并证明严重性分类可作为评估模型能力与实时部署潜力的观察窗口，对根本原因分析（RCA）及更广泛的DT集成具有指导意义。

English

System logs are crucial for monitoring and diagnosing modern computing infrastructure, but their scale and complexity require reliable and efficient automated interpretation. Since severity levels are predefined metadata in system log messages, having a model merely classify them offers limited standalone practical value, revealing little about its underlying ability to interpret system logs. We argue that severity classification is more informative when treated as a benchmark for probing runtime log comprehension rather than as an end task. Using real-world journalctl data from Linux production servers, we evaluate nine small language models (SLMs) and small reasoning language models (SRLMs) under zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting. The results reveal strong stratification. Qwen3-4B achieves the highest accuracy at 95.64% with RAG, while Gemma3-1B improves from 20.25% under few-shot prompting to 85.28% with RAG. Notably, the tiny Qwen3-0.6B reaches 88.12% accuracy despite weak performance without retrieval. In contrast, several SRLMs, including Qwen3-1.7B and DeepSeek-R1-Distill-Qwen-1.5B, degrade substantially when paired with RAG. Efficiency measurements further separate models: most Gemma and Llama variants complete inference in under 1.2 seconds per log, whereas Phi-4-Mini-Reasoning exceeds 228 seconds per log while achieving <10% accuracy. These findings suggest that (1) architectural design, (2) training objectives, and (3) the ability to integrate retrieved context under strict output constraints jointly determine performance. By emphasizing small, deployable models, this benchmark aligns with real-time requirements of digital twin (DT) systems and shows that severity classification serves as a lens for evaluating model competence and real-time deployability, with implications for root cause analysis (RCA) and broader DT integration.

系统日志严重性分类任务中轻量化语言模型与轻量化推理语言模型的基准测试

Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification

摘要

Support