Guilhem Fouilhé

Summary

We investigate the influential factors governing the trade-off between parametric memory and in-context information during factual recall in Large Language Models.

Abstract

We reproduce and stress-test the work of Yu et al. (2023), who characterize how language models (LMs) arbitrate between memorized knowledge and contradictory in-context statements. We replicate their world-capitals experiments on 31 models spanning Pythia, GPT-2, Qwen3, and Ministral families, including base and post-trained variants, and extend evaluations to five additional knowledge relation types from the ParaConflict dataset. We empirically confirm most of their original findings: larger models and higher-frequency entities tend to favor memorized answers, with substantial family-level variance. However, several conclusions do not generalize cleanly: entity-frequency effects disappear on Qwen3-14B and 32B; post-training shifts the memory-context trade-off inconsistently across families; question phrasing alone can change a model’s reliance on memorized knowledge by up to 80 percentage points; and semantically unrelated prose can mimic coherent supporting context. Our results clarify where Yu et al.’s claims hold and to what extent they generalize to other prompts.

Summary

Oral presentation at JCAD 2026 in Nantes, sharing practical feedback and engineering insights from scaling Scaled Signed Averaging (SSA) to pretrain SSA-Luciole (a 1B parameter model trained on 3T tokens) on the Kairos supercomputer at CALMIP.

Abstract

Understanding the mathematical foundations of modern Large Language Models (LLMs) is essential for designing architectures that are more robust, interpretable, and reliable in real-world settings. However, theoretical analyses often rely on unrealistic assumptions, making large-scale empirical validation necessary.

In previous work, we introduced Scaled Signed Averaging (SSA) [1] as a drop-in replacement for the softmax scoring function in Attention to prevent attention from collapsing to a single token in large/out-of-distribution settings. Simple A/B testing on a 114M parameters model showed promising results but this scale is still far away from the models deployed today.

Through a Mesochallenge/collaboration with CALMIP during the deployment phase of the Kairos supercomputer (64 NVIDIA H200 GPUs, 4.77 Pflop/s, ranked #1 in the GREEN500 November 2025 list), and with Linagora via the OpenLLM France project, we pretrained SSA-Luciole, a 1B-parameter model trained on 3T tokens. This model and training dataset size are closer to current trends. We are currently on the process of transitioning this project to national compute resources for the final stages of the training (continued pretraining with cosine learning rate decay on 1.5T tokens then linear annealing on 0.5T).

This work provides practical feedback on scaling a novel attention scoring function in an academic environment. We discuss the implementation of SSA through custom Triton kernels, optimization challenges encountered during distributed training across 16 nodes, and strategies for monitoring large-scale runs and diagnosing GPU failures. We also present preliminary observations regarding training stability and efficiency compared to standard softmax-based architectures. Results suggests that this architecture replacement could improve any transformer model without any additional computational cost.

Our objective is to provide actionable lessons for academic teams seeking to experiment with novel LLM architectures at scale on modern HPC infrastructures.

[1] Naim, O., Bhar, S., Bolte, J., & Asher, N. (2026). SSA: Improving Performance With a Better Scoring Function. ACL 2026.

Summary

🏆 Best Paper Nominee at EUMAS 2026

We formalize and implement an agentic, multi-agent LLM architecture for conversational explanations in planning, and evaluate it with a human-in-the-loop user study.

Abstract

When automating plan generation for a real-world sequential decision problem, the goal is often not to replace the human planner, but to facilitate an iterative reasoning and elicitation process, where the human’s role is to guide the AI planner according to their preferences and expertise. In this context, explanations that respond to users’ questions are crucial to improve their understanding of potential solutions and increase their trust in the system. To enable natural interaction with such a system, we present a multi-agent Large Language Model (LLM) architecture that is agnostic to the explanation framework and enables user- and context-dependent interactive explanations. We also describe an instantiation of this framework for goal-conflict explanations, which we use to conduct a user study comparing the LLM-powered interaction with a baseline template-based explanation interface.

Summary

This is a demo paper about a platform for iterative planning with interactive goal-conflict explanations. My most notable contribution is the design of an agentic multi-agent LLM architecture to present explanations to users through a chat interface, including human-in-the-loop evaluation via online user studies.

Abstract

When automating plan generation for a real-world sequential decision problem, the objective is often not to replace the human planner, but rather to facilitate an iterative reasoning and elicitation process, where the human’s role is to guide the planner according to their preferences and expertise. In this context, explanations that address users’ questions are crucial to improve their understanding of potential solutions and increase their trust in the system. We present a platform that implements this iterative planning approach and provides explanations to user questions based on conflicting goals and preferences. The platform supports both a classical template-based interface and a multi-agent Large Language Model (LLM) architecture that enables interactive explanations tailored to the user and context. The integration of online user studies allows for the evaluation of the effectiveness of the explanations and the impact of the communication interface.

Summary

We formalize and implement an agentic, multi-agent LLM architecture for conversational explanations in planning.

Contribution

This is my first first-author paper!

Abstract

When automating plan generation for a real-world sequential decision problem, the goal is often not to replace the human planner, but to facilitate the tedious work. In an iterative pro- cess, the human’s role is to guide the planner according to their preferences and expert experience. Explanations that re- spond to users’ questions are crucial to increase trust in the system and improve understanding of the sample solutions. To enable natural interaction with such a system, we present an explanation framework agnostic architecture for interac- tive natural language explanations that enables user and con- text dependent interactions. We propose conversational inter- faces based on Large Language Models (LLMs) and instan- tiate the explanation framework with goal-conflict explana- tions. As a basis for future evaluation, we provide a tool for domain experts that implements our interactive natural lan- guage explanation architecture.

Summary

We investigate how transformer models learn linear functions through in-context learning, challenging common assumptions about their algorithmic capabilities.

Contribution

This work provides new insights into the limitations of in-context learning by studying a controlled setup with synthetic data.

Abstract

We explore in-context learning (ICL), a popular paradigm for inference with Large Language Models (LLMs), in a controlled experimental setup using synthetic training data. Using a range of small transformer models trained from scratch, we focus on a mathematical task with simple yet precise prompts: learning a linear function f from a sequence of inputs and their corresponding function values . Our findings challenge the prevailing narrative that transformers adopt algorithmic approaches like linear regression to in-context learn (ICL) a linear function. We observe that all models have “boundary values” that limit generalizability. While we can extend boundary values with training distributions over a wider range, we lose the precision of models trained on distributions with more restricted ranges. Thus, we see a dilemma for ICL at least in some tasks: either models will lack generalizability or precision.

Summary

We present a mathematical framework to systematically investigate neural dynamics in a hierarchical perceptual system, showing how stability and wave propagation properties emerge from predictive coding principles.

Contribution

This paper provides a rigorous mathematical analysis of hierarchical neural networks with predictive coding dynamics, revealing conditions for stability and characterizing wave propagation properties.

Abstract

Sensory perception (e.g., vision) relies on a hierarchy of cortical areas, in which neural activity propagates in both directions, to convey information not only about sensory inputs but also about cognitive states, expectations and predictions. At the macroscopic scale, neurophysiological experiments have described the corresponding neural signals as both forward and backward-travelling waves, sometimes with characteristic oscillatory signatures. It remains unclear, however, how such activity patterns relate to specific functional properties of the perceptual apparatus. Here, we present a mathematical framework, inspired by neural network models of predictive coding, to systematically investigate neural dynamics in a hierarchical perceptual system. We show that stability of the system can be systematically derived from the values of hyper-parameters controlling the different signals (related to bottom-up inputs, top-down prediction and error correction). Similarly, it is possible to determine in which direction, and at what speed neural activity propagates in the system. Different neural assemblies (reflecting distinct eigenvectors of the connectivity matrices) can simultaneously and independently display different properties in terms of stability, propagation speed or direction. We also derive continuous-limit versions of the system, both in time and in neural space. Finally, we analyze the possible influence of transmission delays between layers, and reveal the emergence of oscillations.

Summary

We study the asymptotic behavior of nonlocal bistable reaction-diffusion equations.

Abstract

In this paper, we study the asymptotic behavior of the solutions of nonlocal bistable reaction-diffusion equations starting from compactly supported initial conditions. Depending on the relationship between the nonlinearity, the interaction kernel and the diffusion coefficient, we show that the solutions can either: propagate, go extinct or remain pinned. We especially focus on the latter regime where solutions are pinned by thoroughly studying discontinuous ground state solutions of the problem for a specific interaction kernel serving as a case study. We also present a detailed numerical analysis of the problem.

Summary

We review the application of predictive coding - a popular neuroscience framework - to deep neural networks.

Contribution

This work was produced for TIR (Travaux d’Initiation `a la Recherche) course in M1 Computer Science of UPS during year 2021/2022.

Abstract

Predictive Coding is a popular framework in neurosciences for explaining cortical function. In this model, higher-level cortical areas try to predict lower-level neural activity and prediction errors are passed back to higher layers. Deep Neural Networks (DNN) , which use brain-inspired architecture, could be augmented with such a model, providing robustness and a better understanding of spatio-temporal dependencies. We investigate research in this direction and give a quick review on tasks in which Predictive Coding (PC) for DNN has demonstrated its interest, with a strong emphasis on vision-related tasks.