Xingyou Song

Gradientless Descent: High-Dimensional Zeroth-Order Optimization

2019 · arXiv (Cornell University)

Zeroth-order optimization is the process of minimizing an objective $f(x)$, given oracle access to evaluations at adaptively chosen inputs $x$. In this paper, we present two simple yet powerful GradientLess Descent (GLD) algorithms that do …

Rethinking Attention with Performers

2020 · arXiv (Cornell University)

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity …

OmniPred: Language Models as Universal Regressors

2024 · arXiv (Cornell University)

Regression is a powerful tool to accurately predict the outcome metric of a system given a set of parameters, but has traditionally been restricted to methods which are only applicable to a specific task. In …