Research Notebooks

Key Ideas From
Important Papers

Sharing articles and tutorials on the finer details of training and tuning deep neural networks for maximum performance.

2024-04-10 optimization

Contrastive Language-Image Pretraining

Connecting text and images.

Read More
2024-04-06 optimization

Mode Connectivity

Local minima in loss landscapes are connected by high accuracy pathways.

Read More
2024-03-24 regularization optimization


Learning optimal transformation pipelines for data augmentation.

Read More
2024-03-19 optimization

Gradient Boosting

Ensembles where new members are trained to correct previous mistakes.

Read More
2024-03-08 compression

Knowledge Distillation

Training a small model on the outputs of a larger and more accurate model.

Read More
2024-02-26 optimization

Double Descent

A phenomena where generalization gets worse then better with larger models and bigger datasets.

Read More
2024-02-15 optimization generation

Denoising Diffusion

A class of generative latent variable models inspired by nonequilibrium thermodynamics.

Read More
2024-01-29 compression

Optimal Brain Damage

An early method for pruning networks according to parameter saliency.

Read More
2024-01-22 optimization transfer

Low-Rank Adaptation

Reducing the storage requirements for fine tuned task specific networks.

Read More
2024-01-21 optimization ensemble

Snapshot Ensembles

A low-cost method that leverages checkpoints throughout the training trajectory.

Read More
2024-01-20 optimization reinforcement

Proximal Policy Optimization

A computationally efficient on-policy reinforcement learning algorithm.

Read More
2024-01-19 optimization neuroevolution

Natural Evolution Strategies

A family of algorithms for evolving the parameters of search distributions.

Read More
2024-01-18 regularization


Masking random neurons on each forward pass during training.

Read More
2024-01-17 optimization reinforcement

World Models

Dreaming with generative models of reinforcement learning environments.

Read More
2024-01-16 optimization reinforcement

Deep Q-Learning

A foundational off-policy algorithm that kickstarted deep reinforcement learning.

Read More
2024-01-14 optimization

Stochastic Weight Averaging

An optimization trick for the final phases of training with SGD.

Read More
2024-01-13 optimization reinforcement

Trust Region Policy Optimization

The monotonic on-policy reinforcement learning algorithm.

Read More
2024-01-13 compression

Lottery Ticket Hypothesis

Finding sparse subnetworks that train as well as dense networks from scratch.

Read More


Top Recent Impactful Featured
All Time Past Year Past Month Past Week


all attention augmentation bayesian classification clustering diffusion distillation efficient embedding ensemble evolution federated generative interpretability latent optimization pruning quantization regression regularization reinforcement segmentation sparsity supervised theory transfer tuning unsupervised