透過語言模型函數呼叫的反思性提示微調

摘要

大型語言模型（LLMs）在遵循指令與複雜推理方面已展現出日益增強的能力，使提示（prompting）成為一種無需更新參數即可調整模型的靈活介面。然而，提示設計仍然相當耗費人力，且對格式、措辭及指令順序高度敏感，這促使自動化提示優化方法的發展，以減少人工負擔，同時保留推理時的靈活性。然而，現有方法通常搜尋候選提示，或使用由單一範例或小批次驅動的固定批評-修改流程，因而難以捕捉系統性錯誤模式，也無法根據失敗歷史進行有針對性的編輯。我們提出反思性提示調整（Reflective Prompt Tuning, RPT）框架，該框架利用LLM函數呼叫模擬人類提示工程師的迭代工作流程。LLM優化器呼叫一個診斷函數，該函數在整個優化集上評估目標模型，總結重複出現的失敗模式，並回傳結構化的診斷報告。優化器利用這份報告，加上先前報告的累積記憶，來修改下一輪迭代的提示。RPT還透過在診斷回饋及最終提示選擇中使用校準訊號，支援具信心感知的優化。在三項推理任務中，RPT將初始提示的表現提升了最多12.9個百分點，與現有最佳方法競爭力相當，並改善了信心校準。我們的分析顯示，RPT在多重跳躍與數學推理上尤其有效，能產生與診斷出的失敗模式相符的目標性提示修改，進而帶來任務表現與校準的提升。

English

Large language models (LLMs) have become increasingly capable of following instructions and complex reasoning, making prompting a flexible interface for adapting models without parameter updates. Yet prompt design remains labor-intensive and highly sensitive to formatting, phrasing, and instruction order, motivating automated prompt optimization methods that reduce manual effort while preserving inference-time flexibility. However, existing methods often search over prompt candidates or use fixed critique-refine pipelines driven by individual examples or small batches, limiting their ability to capture systematic error patterns and make targeted edits grounded in failure history. We propose Reflective Prompt Tuning (RPT), a framework that uses LLM function calling to simulate the iterative workflow of human prompt engineers. An LLM optimizer calls a diagnostic function that evaluates the target model over an entire optimization set, summarizes recurring failure modes, and returns a structured diagnostic report. The optimizer uses this report, together with an accumulated memory of prior reports, to revise the prompt for the next iteration. RPT further supports confidence-aware optimization by using calibration signals in diagnostic feedback and final prompt selection. Across three reasoning tasks, RPT improves over initial prompts by up to 12.9 points, remains competitive with state of the art, and improves confidence calibration. Our analyses show that RPT is especially effective on multi-hop and mathematical reasoning, producing targeted prompt revisions that align with diagnosed failure patterns and lead to gains in task performance and calibration.