Effective Distillation to Hybrid xLSTM Architectures
ICML 2026
Near-lossless linearization of 7–8B models into hybrid xLSTM architectures, with up to 40k context and 2D parallelism (FSDP + TP).
Research
Papers on efficient language models, parameter-efficient fine-tuning, and bias mitigation.
KVpop — Key-Value Cache Compression with Predictive Online Pruning
ICLR TTU Workshop 2026
Efficient long-context reasoning that retains most of the accuracy of dense attention on math benchmarks.
Parameter-Efficient Fine-Tuning via Explained Variance Adaptation
NeurIPS 2025 · Shared first authorship
Data-driven LoRA initialization, with a fast GPU implementation of incremental PCA.
Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks
Findings of ACL 2023
Unlearning biased representations in pre-trained language models with attribute-removal subnetworks.
Capability Retention under Post-Training Linearization of Small LMs
NeurIPS SLM-Agents Workshop 2026
Distilling 0.6–4B Qwen3 and SmolLM3 models into hybrid architectures, with 3.2–7.1× throughput on long generations.
On Subquadratic Architectures: From Applications to Principles
arXiv preprint 2026
A comparison of xLSTM, Mamba-2, and Gated DeltaNet for pre-training and distillation.
Citations are on Google Scholar.
Google Scholar