Here are some helpful resources I used while making this course. I’m sharing them with you in case they help. Enjoy!
- Deep Dive into LLMs like ChatGPT (Theory)
- Building GPT from Scratch (Practice 1)
- Building a GPT Tokenizer (Practice 2)
- Reproducing GPT-2 (Practice 3)
- Train a language model using WhatsApp group chats
- Create an AI version of yourself using WhatsApp chats
- Extract WhatsApp key/database without root access
- Train a language model on your WhatsApp chats
- Fine-tune an LLM to create your digital twin
- Fine-tuning LLMs using QLoRA
- Guide to fine-tuning large language models
- Understanding LoRA and DoRA for model tuning
- Step-by-step guide to fine-tune LLMs with LoRA
- Fine-tuning LLMs for multi-turn conversations
- Fine-tuning mixtral 7bx8 with LoRA
- Positional Encoding Explained: A Deep Dive into Transformer PE
- You could have designed state of the art positional encoding
- Relative Positional Encoding
- What is grouped query attention (GQA)?
- Linear Attention Is All You Need
- Attention Variations — MQA vs GQA vs MHA vs MLA
- Understanding Multi-Head Latent Attention
- DeepSeek's Multi-Head Latent Attention
- Sliding Window Attention
- Can LLMs learn from a single example?
- Normalization Layer Placement (Pre-LN vs Post-LN)
- Batch Normalization, Layer Normalization and Root Mean Square Layer Normalization: A Comprehensive Guide with Python Implementations
- Deep Dive into Deep Learning: Layers, RMSNorm, and Batch Normalization
- Exploring SwiGLU : The Activation Function Powering Modern LLMs
- All the Activation Functions (and a history of deep learning)
- Understanding and Using Supervised Fine-Tuning (SFT) for Language Models)
- Finetuned Llama 2-7B using WhatsApp chats
- How to train your model
- Exporting full WhatsApp chat history
- Normalization in transformers
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- LINEAR ATTENTION IS (MAYBE) ALL YOU NEED (TO UNDERSTAND TRANSFORMER OPTIMIZATION)
- Attention Is All You Need
- Longformer: The Long-Document Transformer
- Effective Approaches to Attention-based Neural Machine Translation
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Big Bird: Transformers for Longer Sequences
- Root Mean Square Layer Normalization
- Dropout: A Simple Way to Prevent Neural Networks from Overfitting
- On Layer Normalization in the Transformer Architecture
- Root Mean Square Layer Normalization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Layer Normalization
- Dropout Reduces Underfitting
- GLU Variants Improve Transformer
- The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities (Version 1.0)