iondy — Lukas Hauzenbergerhttps://iondy.comMachine learning research and notes on what I’m learning.enSliding window beats linear attention, but only on short answershttps://iondy.com/writing/sliding-window-beats-linear-attention-but-only-on-short-answers/https://iondy.com/writing/sliding-window-beats-linear-attention-but-only-on-short-answers/Fri, 30 Oct 2026 00:00:00 GMTAt equal memory, a sliding window matches a hybrid layer on short answers, but a linear mixer's state pays off on math problems that need long answers.Efficient LLM Pretraining: Packed Sequences and Masked Attentionhttps://iondy.com/writing/efficient-llm-pretraining-packed-sequences-and-masked-attention/https://iondy.com/writing/efficient-llm-pretraining-packed-sequences-and-masked-attention/Mon, 07 Oct 2024 00:00:00 GMTMaking every token count with sequence packing, attention masks, and correctly aligned position IDs.Introducing pytora — a minimal PyTorch implementation of LoRAhttps://iondy.com/writing/introducing-pytora-a-minimal-pytorch-implementation-of-lora/https://iondy.com/writing/introducing-pytora-a-minimal-pytorch-implementation-of-lora/Sun, 26 May 2024 00:00:00 GMTA small, flexible implementation of LoRA using native PyTorch parametrizations.Multilabel Classification using Mistral-7B on a single GPU with quantization and LoRAhttps://iondy.com/writing/multilabel-classification-using-mistral-7b-on-a-single-gpu-with-quantization-and-lora/https://iondy.com/writing/multilabel-classification-using-mistral-7b-on-a-single-gpu-with-quantization-and-lora/Tue, 16 Jan 2024 00:00:00 GMTFine-tuning Mistral for multilabel classification with quantization and LoRA on a single GPU.What I learned doing Data Science Interviews in the 2020shttps://iondy.com/writing/what-i-learned-doing-data-science-interviews-in-the-2020s/https://iondy.com/writing/what-i-learned-doing-data-science-interviews-in-the-2020s/Mon, 18 Dec 2023 00:00:00 GMTNotes on preparation, technical questions, and the interview process from both sides of the table.Training and serving PyTorch models in the Google Cloud with Vertex AI pipelineshttps://iondy.com/writing/training-and-serving-pytorch-models-in-the-google-cloud-with-vertex-ai-pipelines/https://iondy.com/writing/training-and-serving-pytorch-models-in-the-google-cloud-with-vertex-ai-pipelines/Fri, 07 Apr 2023 00:00:00 GMTA practical walkthrough from PyTorch training code to an automated pipeline on Google Cloud.Adversarial Traininghttps://iondy.com/writing/adversarial-training/https://iondy.com/writing/adversarial-training/Mon, 22 Aug 2022 00:00:00 GMTImplementing gradient reversal and adversarial heads to study and reduce bias in language models.A practical introduction to Diff-Pruning for BERThttps://iondy.com/writing/a-practical-introduction-to-diff-pruning-for-bert/https://iondy.com/writing/a-practical-introduction-to-diff-pruning-for-bert/Wed, 23 Feb 2022 00:00:00 GMTThe ideas behind diff-pruning, and how to implement parameter-efficient fine-tuning in PyTorch.