Sliding window beats linear attention, but only on short answers
At equal memory, a sliding window matches a hybrid layer on short answers, but a linear mixer's state pays off on math problems that need long answers.
16 min readWriting
Research, implementations, and what I learned along the way.
Efficient LLM Pretraining: Packed Sequences and Masked Attention
Making every token count with sequence packing, attention masks, and correctly aligned position IDs.
8 min readIntroducing pytora — a minimal PyTorch implementation of LoRA
A small, flexible implementation of LoRA using native PyTorch parametrizations.
4 min readMultilabel Classification using Mistral-7B on a single GPU with quantization and LoRA
Fine-tuning Mistral for multilabel classification with quantization and LoRA on a single GPU.
7 min readWhat I learned doing Data Science Interviews in the 2020s
Notes on preparation, technical questions, and the interview process from both sides of the table.
9 min readTraining and serving PyTorch models in the Google Cloud with Vertex AI pipelines
A practical walkthrough from PyTorch training code to an automated pipeline on Google Cloud.
16 min readAdversarial Training
Implementing gradient reversal and adversarial heads to study and reduce bias in language models.
8 min readA practical introduction to Diff-Pruning for BERT
The ideas behind diff-pruning, and how to implement parameter-efficient fine-tuning in PyTorch.
11 min readNo notes found.
Try a different word or explore another topic.
8 entries in the notebook.