Krzysztof Choromański

CWY Parametrization for Scalable Learning of Orthogonal and Stiefel Matrices

2020 · arXiv (Cornell University)

We introduce an efficient approach for optimization over orthogonal groups on highly parallel computation units such as GPUs or TPUs. As in earlier work, we parametrize an orthogonal matrix as a product of Householder reflections. …

Rethinking Attention with Performers

2020 · arXiv (Cornell University)

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity …