Ax Francesco D'Angelo, Oguz Kaan Yuksel, Swathi Shree Narashiman, Nicolas Flammarion 7/7/2026

Induction Heads Interpolate N-Grams

Study of induction heads in transformers, identifying attention circuits that implement soft context-matching estimators for in-context learning via smoothing mechanisms.

Ax Stefan Horoi, Benjamin Th\'erien, Guy Wolf, Eugene Belilovsky 7/7/2026

Can Model Merging Improve Aggregation in DiLoCo?

Investigation of model merging techniques for improving aggregation in DiLoCo distributed learning, combining independent finetuned models.

Ax Zhilong Zhang, Hongli Yu, Huan-ang Gao, Hanlin Wu, Yuxuan Song, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou 7/7/2026

Spectral Rewiring for Exploration, Purification, and Model Merging

Spectral rewiring technique for RL post-training of LLMs, addressing reasoning saturation and model merging interference through targeted parameter updates.

Ax Muhammad Sabih, Frank Hannig, J\"urgen Teich 7/7/2026

Rethinking Neural Nonlinearity as Gating

Shows neural nonlinearity can be achieved through input-conditioned threshold gating as alternative to activation functions, unifying standard activations.