Patrick Huber

CoDi: Conversational Distillation for Grounded Question Answering

2024 · arXiv (Cornell University)

Distilling conversational skills into Small Language Models (SLMs) with approximately 1 billion parameters presents significant challenges. Firstly, SLMs have limited capacity in their model parameters to learn extensive knowledge compared to larger models. Secondly, high-quality …

Scaling Parameter-Constrained Language Models with Quality Data

2024 · arXiv (Cornell University)

Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization. In this paper, …