Pratyush Maini

Characterizing Datapoints via Second-Split Forgetting

2022 · arXiv (Cornell University)

Researchers investigating example hardness have increasingly focused on the dynamics by which neural networks learn and forget examples throughout training. Popular metrics derived from these dynamics include (i) the epoch at which examples are first …

Scaling Laws for Data Filtering—Data Curation Cannot be Compute Agnostic

2024

Vision-language models (VLMs) are trained for thousands of GPU hours on carefully selected subsets of massive web scrapes. For instance, the LAION public dataset retained only about 10% of the total crawled data. In recent …