I think Yann Lecun was right about LLMs (but perhaps only by accident)

Please Commit More Blatant Academic Fraud (2021)

Demystifying Noise Contrastive Estimation

It's time to become an ML engineer (2022)

Approximating KL Divergence (2020)

The Ultra-Scale Playbook: Training LLMs on GPU Clusters

Implementing LLaMA3 in 100 Lines of Pure Jax

Outperforming cuBLAS on H100: A Worklog

Gravel Map

Stochastic Integration for Poets

NASA writes space-proof code [video]

Large Lambda Model

Softmax forever, or why I like softmax

Diffusion Without Tears

AGI Safety Course Workbook

AI Nationalism – Ian Hogarth

A Beginners' Guide to Misprints in Magic

Flow with What You Know

Going with the Flow: An Introduction to Normalizing Flows

Diffusion Meets Flow Matching: Two Sides of the Same Coin

Muon: An optimizer for hidden layers in neural networks

Honeycrisp: An Apple-First Deep Learning Framework

My Gear

DOGE for AI

How to Backpack

GRPO with Verifiable Rewards Is Contrastive Loss

Bit Prediction

How Could Telepathy Work?

How to Scale Your Model

RLHF Book

Reading notes: unsupervised word translation

The Art of Debugging

What a $500k grant proposal looks like

Deep Reinforcement Learning Doesn't Work Yet

How far can you get in 40 minutes from each subway station in NYC?

Attention Sinks in LLMs for endless fluency

Flow with What You Know: An Introduction to Flow-Based Models

A History of Nvidia Stream Multiprocessor (2020)

Gaming TruthfulQA: Simple Heuristics Exposed Dataset Weaknesses

Ε, a Nuisance No More

Learning CUDA by Optimizing Softmax

History of Residuals and a Word of Caution

MixBox: Practical Pigment Mixing for Digital Painting [pdf]

Detecting Tanks (2017)

The Bittersweet Lesson

Why transformers are obviously good models of language

What would happen if you made a planet out of fish?

Diffusion Meets Flow Matching: Two Sides of the Same Coin

Educating Silicon

Quick software tips for new ML researchers

The Baked Data architectural pattern

What Is Entropix Doing?

DeltaNet Explained (Part I)

How To Change Your Behavior

Infini-Gram: Scaling Unbounded N-Gram Language Models to a Trillion Tokens

Making Transformers Do Math

Sunsethue – Today's Sunset Forecast

An Unserious Take on Axiomatic Knowledge in the Era of Foundation Models

A Meticulous Guide to Advances in Deep Learning Efficiency over the Years

Clowning in Pennsylvania

Contextual Document Embeddings

Experiments in Self-Assembly

Matrixmultiplication.xyz

Not Quite Past – Real Ceramic Tiles Designed by AI

Data Compression with Arithmetic Coding

Int4 Decoding GQA CUDA Optimizations for LLM Inference

Einsum Is Easy and Useful

Optimizing Matrix Multiplication

Kullback-Leibler (KL) Is All You Need

How Do Language Models Put Attention Weights over Long Context?

Chess Engines: A Zero to One

A Personal History of Legion, by Way of Its Papers

Bananagrams Is NP-Complete

A Better Lesson

Seeking the Productive Life: Some Details of My Personal Infrastructure

A Neighborhood with Friends

How to graduate your PhD when you have no hope

PyTorch Word Embeddings Tutorial

Integer Tokenization Is Insane (2023)

Diffusion models from scratch, from a new theoretical perspective

A Refined Similarity-Based Bigram Model

Speculative Sampling

An Introduction to Optimization: Combinatorial Optimization

Definite Optimism as Human Capital

Singular Value Decomposition Part 1: Perspectives on Linear Algebra

Singular Value Decomposition as Simply as Possible

Too much efficiency makes everything worse: overfitting and Goodhart's law

Detecting Mismatches in Machine-Learning Systems