Sliding window beats linear attention, but only on short answers
At equal memory, a sliding window matches a hybrid layer on short answers, but a linear mixer's state pays off on math problems that need long answers.
16 min readLukas Hauzenberger
Machine learning research and notes on what I’m learning.
Recent writing
All writingEfficient LLM Pretraining: Packed Sequences and Masked Attention
Making every token count with sequence packing, attention masks, and correctly aligned position IDs.
8 min readIntroducing pytora — a minimal PyTorch implementation of LoRA
A small, flexible implementation of LoRA using native PyTorch parametrizations.
4 min readMultilabel Classification using Mistral-7B on a single GPU with quantization and LoRA
Fine-tuning Mistral for multilabel classification with quantization and LoRA on a single GPU.
7 min readResearch
All papersEffective Distillation to Hybrid xLSTM Architectures
ICML 2026
Near-lossless linearization of 7–8B models into hybrid xLSTM architectures, with up to 40k context and 2D parallelism (FSDP + TP).
KVpop — Key-Value Cache Compression with Predictive Online Pruning
ICLR TTU Workshop 2026
Efficient long-context reasoning that retains most of the accuracy of dense attention on math benchmarks.
Parameter-Efficient Fine-Tuning via Explained Variance Adaptation
NeurIPS 2025 · Shared first authorship
Data-driven LoRA initialization, with a fast GPU implementation of incremental PCA.
Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks
Findings of ACL 2023
Unlearning biased representations in pre-trained language models with attribute-removal subnetworks.