The Grounding Tax: How Domain Fine-Tuning Suppresses Context Utilisation in Retrieval-Augmented Generation

Abstract

Large language models (LLMs) are routinely adapted to specialised domains through fine-tuning and grounded in external evidence through retrieval-augmented generation (RAG). The prevailing assumption is that these methods are complementary even though minimal empirical work examines whether domain-specific fine-tuning influences a model’s responsiveness to retrieved context. Thus, this study examines the interaction between fine-tuning and RAG through a controlled evaluation in a high-stakes public-finance domain. The evaluation is conducted through 12 model configurations built on two open-weight model families (LLaMA-3.1-8B and Qwen2.5-7B), using a frozen 100-query benchmark covering factual question-answering (factual-QA), risk assessment, and summarisation under both closed-book and context-matched conditions. The evidence confirms successful knowledge injection as domain fine-tuning raised closed-book numeric-fact accuracy (NFA) from 0.148 to 0.248. However, when retrieved context is introduced, the benefit collapses as the base model gained +0.183 from context while the fine-tuned model gained only +0.021. Across the backbone pairs, the change in context gain ranged from -0.161 to -0.348 and was statistically significant for the LLaMA Instruct and Qwen Base pairs. Further ablation analysis shows that prompt-template and instruction alignment can amplify this effect. We term this phenomenon the grounding tax: domain fine-tuning imposes a measurable cost on the model’s ability to use retrieved evidence because the adapted model relies on internally encoded knowledge rather than faithfully using the evidence supplied to it. Thus, these findings show that conventional closed-book evaluation can mask degradation in context utilisation and that fine-tuned RAG systems must be evaluated jointly under both closed-book and context-conditioned settings. Therefore, we argue for treating domain adaptation, instruction alignment, and retrieval grounding as distinct and potentially competing design dimensions.