强化学习中从评分标准到代码的信用分配

摘要

交互式Web应用生成要求模型能够根据自然语言请求生成可用的HTML、CSS和JavaScript应用。与常规代码生成不同，应用质量取决于多个面向用户的功能需求，每个需求通常与局部代码区域相关，如事件处理器、状态更新、DOM片段或CSS选择器。标准GRPO将这些结构化结果压缩为单一的序列级奖励，并将所得优势值均匀应用于所有token，从而削弱了信用分配的效果。我们提出Rubric-to-Code信用分配（RCCA），一种强化学习框架，将评分标准级的功能反馈转化为针对生成代码的局部优化信号。RCCA围绕明确的功能评分标准构建训练任务，使用分层奖励来区分格式、源码、运行时和功能层面的失败，并将评估器生成的文本归因与对应的代码片段及生成token对齐。由此产生的模型Ling-RCCA-Flash在MiniAppBench上取得41.25分，较Ling-3.0-Flash提升32.20分，并略微超越Claude Opus 4.5。该模型在ArtifactsBench上达到76.19分，较SFT模型提升4.48分，并在官方ArtifactsBench排行榜设置下以超过GPT-5达3.64分的成绩刷新最高纪录，这表明其实现层面的改进具有可迁移性。

English

Interactive web application generation requires models to produce usable HTML, CSS, and JavaScript applications from natural language requests. Unlike conventional code generation, application quality depends on multiple user-facing functional requirements, each often tied to localized code regions such as event handlers, state updates, DOM fragments, or CSS selectors. Standard GRPO collapses these structured outcomes into a single sequence-level reward and applies the resulting advantage uniformly to all tokens, weakening credit assignment. We propose Rubric-to-Code Credit Assignment (RCCA), a reinforcement learning framework that converts rubric-level functional feedback into localized optimization signals over generated code. RCCA builds training tasks around explicit functional rubrics, uses a hierarchical reward to separate format, source-code, runtime, and functional failures, and aligns evaluator-generated textual attributions with responsible code spans and generated tokens. The resulting model, Ling-RCCA-Flash, scores 41.25 on MiniAppBench, improving Ling-3.0-Flash by 32.20 points and slightly surpassing Claude Opus 4.5. It also reaches 76.19 on ArtifactsBench, improving the SFT model by 4.48 points and establishing a new top score under the official ArtifactsBench leaderboard setting by surpassing the GPT-5 score by 3.64 points, suggesting transferable implementation-level gains.