Siyuan Zhuang

Hoplite: Efficient Collective Communication for Task-Based Distributed Systems.

2020 · arXiv (Cornell University)

Collective communication systems such as MPI offer high performance group communication primitives at the cost of application flexibility. Today, an increasing number of distributed applications (e.g, reinforcement learning) require flexibility in expressing dynamic and asynchronous …

A Statistical Framework for Ranking LLM-Based Chatbots

2024 · arXiv (Cornell University)

Large language models (LLMs) have transformed natural language processing, with frameworks like Chatbot Arena providing pioneering platforms for evaluating these models. By facilitating millions of pairwise comparisons based on human judgments, Chatbot Arena has become …

Efficient Memory Management for Large Language Model Serving with PagedAttention

2023

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and …