Ali Ghodsi

Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher

2021 · arXiv (Cornell University)

With ever growing scale of neural models, knowledge distillation (KD) attracts more attention as a prominent tool for neural model compression. However, there are counter intuitive observations in the literature showing some challenging limitations of …

Johnson-Lindenstrauss Lemma, Linear and Nonlinear Random Projections,\n Random Fourier Features, and Random Kitchen Sinks: Tutorial and Survey

2021 · arXiv (Cornell University)

This is a tutorial and survey paper on the Johnson-Lindenstrauss (JL) lemma\nand linear and nonlinear random projections. We start with linear random\nprojection and then justify its correctness by JL lemma and its proof. Then,\nsparse random …

Attention Mechanism, Transformers, BERT, and GPT: Tutorial and Survey

2024

This is a tutorial and survey paper on the attention mechanism, transformers, BERT, and GPT. We first explain attention mechanism, sequence-to-sequence model without and with attention, self-attention, and attention in different areas such as natural …