preprint Open access

Capacity Scaling of Encoding-Through-Loading: Application vs. Cloze Asymmetry Across Three Orders of Magnitude

  • Zenodo (CERN European Organization for Nuclear Research)
  • European Organization for Nuclear Research
Research footprint

At a glance

Citations
0
References
0
Comments
0
Paper overview

Abstract

Abstract. Large language models solve cloze retrieval and derivation-application on the same underlying knowledge base, but at different scales. We document this asymmetry on a single invented knowledge domain ("Zorbetik") designed to eliminate pretraining-prior confounds, across nine models from 0.5B to 671B parameters, using frontloaded in-context learning to expose encoding-to-retrieval dynamics without weight updates. Three findings. (1) Application scales monotonically with capacity (Spearman ρ = +1.000 on the Qwen2.5 sub-ladder, n=5, p = 0.0083 one-tailed; cross-family panel ρ = +0.92, n=9, p = 0.0005 two-tailed; +40.8 percentage points per decade); cloze does not, saturating at ~90% by 8B parameters. (2) The bottleneck migrates with capacity: at 0.5B retrieval fails; at 14B retrieval is saturated and 36% of questions show "retrieval succeeds, derivation fails". (3) Mixture-of-Experts models scale on active, not total, parameters: a 235B/22B-active MoE behaves on application tasks like a 22B dense model. Methodological contribution. We propose frontloaded in-context learning as an operational substitute for fine-tuning in encoding-to-retrieval studies. Fast (~5 seconds per inference vs hours for FT), cheap (cents vs dollars), unified across model families, and produces dense per-token friction data. Implications. Capacity is single-axis but loads task types differentially; cloze is indexing-bound, application is composition-bound. The three-dimensional friction framework from companion Paper 1 decomposes the C-axis into operational handles. Data and code. Per-token logprob datasets, analysis scripts, and figures available at https://github.com/tplund/friction-theory-p2-capacity-scaling (CC BY 4.0).

Record transparency

Publication details

DOI
10.5281/zenodo.20013492
OpenAlex
W7159974110
Document type
preprint
Language
EN
Source
Zenodo (CERN European Organization for Nuclear Research)
Last metadata update
Community

Comments

Log in to join the discussion.

  1. No comments yet. Start the discussion.