A Quantitative Framework for Measuring Knowledge Structure and Fine-Tuning Efficacy in Large Language Models

The process of fine-tuning large language models (LLMs) to instill specific domain expertise is often treated as an art, lacking a rigorous, quantitative framework for diagnosing the quality of the resulting knowledge structure. This paper introduces a novel, dual-metric framework for evaluating fine-tuned models, derived from our previously established ”Knowledge Provenance Maps” methodology. We define two complementary sets of metrics: (1) Knowledge Structure Metrics (Sparsity, Concentration, Density) to assess the anatomy of the learned knowledge, and (2) Fine-Tuning Efficacy Metrics (Specialization, Efficiency, Utilization) to measure the performance of the model’s parameters. We present a detailed comparative case study, demonstrating how these metrics provide a precise, quantitative diagnosis that differentiates a well-learned domain from a poorly-learned one, directly predicting and explaining qualitative model behaviors. This framework transforms model analysis from a subjective art into a reproducible science, providing a powerful diagnostic toolkit for building more robust, efficient, and reliable AI systems.