LLM-MRI Python module: a brain scanner for LLMs - paper

Abstract. LLMs (Large Language Models) have demonstrated human-level lan- guage and knowledge acquisition skills in several tasks. However, despite the recent success and broad use, understanding how these skills learned are and encoded inside the underlying neural network is still challenging. The goal of the LLM-MRI package is to simplify the study of activation patterns in any transformer-based LLM, similarly to how MRI (magnetic resonance imaging) simplifies with biological brains. The package, written for the Python lan- guage, allows the mapping of neural regions using a parameterized reduction of the model’s dimensionality. Neural regions can be viewed according to the forward-pass activations stimulated by a set of documents. , the pack- age enables the creation of graph models representing the interlayer network of stimulated by a set of documents. These features allow for which- itative and quantitative assessments of the underlying structure of activations, depending on the type of documents that the LLM model is exposed to.