Capacity Scaling of Encoding-Through-Loading: Application vs. Cloze Asymmetry Across Three Orders of Magnitude
At a glance
- Citations
- 0
- References
- 0
- Comments
- 0
Öz
Abstract. Large language models solve cloze retrieval and derivation-application on the same underlying knowledge base, but at different scales. We document this asymmetry on a single invented knowledge domain ("Zorbetik") designed to eliminate pretraining-prior confounds, across nine models from 0.5B to 671B parameters, using frontloaded in-context learning to expose encoding-to-retrieval dynamics without weight updates. Three findings. (1) Application scales monotonically with capacity (Spearman ρ = +1.000 on the Qwen2.5 sub-ladder, n=5, p = 0.0083 one-tailed; cross-family panel ρ = +0.92, n=9, p = 0.0005 two-tailed; +40.8 percentage points per decade); cloze does not, saturating at ~90% by 8B parameters. (2) The bottleneck migrates with capacity: at 0.5B retrieval fails; at 14B retrieval is saturated and 36% of questions show "retrieval succeeds, derivation fails". (3) Mixture-of-Experts models scale on active, not total, parameters: a 235B/22B-active MoE behaves on application tasks like a 22B dense model. Methodological contribution. We propose frontloaded in-context learning as an operational substitute for fine-tuning in encoding-to-retrieval studies. Fast (~5 seconds per inference vs hours for FT), cheap (cents vs dollars), unified across model families, and produces dense per-token friction data. Implications. Capacity is single-axis but loads task types differentially; cloze is indexing-bound, application is composition-bound. The three-dimensional friction framework from companion Paper 1 decomposes the C-axis into operational handles. Data and code. Per-token logprob datasets, analysis scripts, and figures available at https://github.com/tplund/friction-theory-p2-capacity-scaling (CC BY 4.0).
Publication details
- DOI
- 10.5281/zenodo.20013492
- OpenAlex
- W7159974110
- Document type
- preprint
- Language
- EN
- Source
- Zenodo (CERN European Organization for Nuclear Research)
- Last metadata update
Comments
Oturum Açın to join the discussion.