| Lyu Cunchi (吕存驰)* **,Shi Xiao* ****,Lei Zhengyu* **,Li Qingwen* **,Zhao Xiaofang* ** *** *****.[J].高技术通讯(英文),2026,32(2):121~132 |
|
| Triage: optimizing LLM serving goodput in unified scheduling via feedback-driven batch shaping |
| |
| DOI：10. 3772 / j. issn. 1006-6748. 2026. 02. 002 |
| 中文关键词: |
| 英文关键词: large language model serving, iteration-level scheduling, goodput-driven scheduling |
| 基金项目: |
| Author Name | Affiliation | | Lyu Cunchi (吕存驰)* ** | (* University of Chinese Academy of Sciences, Beijing 100049, P. R. China)
(** Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, P. R. China)
(*** University of Chinese Academy of Sciences Nanjing, Nanjing 211135, P. R. China)
(**** Nanjing Institute of InforSuperBahn, Nanjing 211135, P. R. China)
(***** Institute of Intelligent Computing Technology, Suzhou, Chinese Academy of Sciences,Suzhou 215028, P. R. China) | | Shi Xiao* **** | | | Lei Zhengyu* ** | | | Li Qingwen* ** | | | Zhao Xiaofang* ** *** ***** | |
|
| Hits: 116 |
| Download times: 130 |
| 中文摘要: |
| |
| 英文摘要: |
| Existing iteration-level schedulers for large language model (LLM) serving rely on static, single-mode policies (e. g. , prefill-first or chunked-prefill) and fixed batching hyperparameters, which fail to adapt to the dynamic resource demands of production workloads. Consequently, these systems often struggle to balance the competing pressures of prefill and decode phases, leading to significant degradation in goodput, the rate of requests satisfied within strict service level objectives ( SLOs).To address these limitations, we propose Triage, a unified scheduling framework that optimizes goodput via feedback-driven batch shaping. Unlike prior state-aware schedulers that operate in abstract design spaces, Triage leverages a hierarchical feedback control loop to directly manipulate concrete engine-level knobs, enabling dynamic switching between decode-only, chunked-prefill, and prefillonly regimes. By continuously monitoring workload composition and key value (KV) cache pressure, Triage dynamically steers batch composition and execution regimes without requiring manual hyperparameter tuning. We implement Triage on top of the vLLM, open-source LLM serving engine,and experimental results demonstrate that it achieves up to 10. 50 × higher goodput compared to state-of-the-art baselines while maintaining robust performance across diverse workload distributions and distributed serving configurations. |
|
View Full Text
View/Add Comment Download reader |
| Close |
|
|
|
