arxiv:2610.23234

MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation

Published on Oct 20

· Submitted by

sunyuhan on Oct 26

TaoLive AIGC

Upvote

Authors:

Yuhan Sun ,

Abstract

MultiAgent-OPD jointly trains a hierarchical multi-agent team via on-policy distillation and reinforcement learning, using a frozen teacher for local process supervision to improve end-to-end performance across multiple datasets.

Generated by thinkingmachines/Inkling-Small

In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out. Jointly training the Leader and the subagents is difficult because a final answer gives only indirect feedback to the subagents. An LLM judge can score the performance of the subagents, but these scores depend on the judge's own assessment of the quality of subagents' responses. We present MultiAgent-OPD, which combines on-policy distillation with reinforcement learning to jointly train a shared-parameter Leader-Searcher-Answerer team end to end. We first train a Teacher in a Searcher-Answerer workflow using answer correctness. During team training, the frozen Teacher scores student-generated search and summary tokens to provide local process supervision for the Searchers, while the Leader learns from team success and the Answerer from candidate correctness. Across seven datasets, the trained system reaches 38.68% macro avg@4, exceeding Prompted Critic by 4.06 percentage points and C-MAS + RL, the trained cascaded multiagent system, by 2.61 percentage points. The team also solves more questions within four attempts.

View arXiv page View PDF Project page Add to collection

Community

yuuhan

Paper author Paper submitter 1 day ago

•

edited 1 day ago

✨ Highlights

Role-specific learning signals — MultiAgent-OPD combines local process supervision for Searchers with outcome-based reinforcement learning for the Leader and Answerer, addressing credit assignment in hierarchical multi-agent teams.

Two-stage training recipe — First trains a teacher to search and summarize in a cascaded Search–Answer workflow, then transfers its expertise to Searchers during joint hierarchical training.

Token-level on-policy distillation — A frozen, task-trained teacher scores student-generated queries and summaries on evolving subtask trajectories, providing fine-grained guidance for evidence acquisition and reporting.

Joint training with a shared model — Leader, Searcher, and Answerer roles jointly optimize a shared language model to coordinate task decomposition, multi-wave search, and answer generation. Inference requires only the trained student.

Strong performance across seven QA benchmarks — MultiAgent-OPD achieves 38.68% macro avg@4, outperforming M-GRPO like LLM Judge MAS-RL method by 4.06 percentage points and C-MAS + RL (cascaded MAS-RL method) by 2.61 points at the evaluated checkpoints.