Dimitris Papailiopoulos
4 papers in the PaperMetrix corpus
Papers by this author
-
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation
2019 · arXiv (Cornell University)
To improve the resilience of distributed training to worst-case, or Byzantine node failures, several recent approaches have replaced gradient averaging with robust aggregation methods. Such techniques can have high computational costs, often quadratic in the …
-
Transformers as Algorithms: Generalization and Stability in In-context Learning
2023 · arXiv (Cornell University)
In-context learning (ICL) is a type of prompting where a transformer model operates on a sequence of (input, output) examples and performs inference on-the-fly. In this work, we formalize in-context learning as an algorithm learning …
-
Mini-Batch Optimization of Contrastive Loss
2023 · arXiv (Cornell University)
Contrastive learning has gained significant attention as a method for self-supervised learning. The contrastive loss function ensures that embeddings of positive sample pairs (e.g., different samples from the same class or different views of the …
-
Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries
2024 · arXiv (Cornell University)
We introduce Lexico, a novel KV cache compression method that leverages sparse coding with a universal dictionary. Our key finding is that key-value cache in modern LLMs can be accurately approximated using sparse linear combination …