rasbt/LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
LLMs from Scratch is rasbt's (Sebastian Raschka's) PyTorch code companion to the bestselling book 'Build a Large Language Model (From Scratch).' If you want to truly understand transformers, attention, and LLM training instead of just calling an API, this is the canonical learning resource.
Our Review
LLMs from Scratch is Sebastian Raschka's deep-dive companion code to the bestselling book "Build a Large Language Model (From Scratch)" -- 95,000 GitHub stars and 14,500 forks as of May 2026. Published by rasbt (Sebastian Raschka, ML researcher and author) and released under a custom permissive license, the repository walks through implementing a ChatGPT-style transformer LLM in PyTorch from the bare math up, with every chapter shipped as a runnable Jupyter notebook.
What LLMs from Scratch does:
- •Chapter-by-chapter PyTorch implementation attention, multi-head attention, layer norm, residual connections, full transformer block, training loop, evaluation, fine-tuning, and instruction tuning, each in its own notebook.
- •Companion to the bestselling book the repo is the official code for Sebastian Raschka's book "Build a Large Language Model (From Scratch)" published by Manning.
- •Bonus material additional notebooks beyond the book on classification fine-tuning, parameter-efficient training (LoRA), and pretraining at scale.
- •Pure PyTorch no LangChain, no abstractions; every line is explicit so you can see exactly what happens at each step.
- •Mathematical explanations LaTeX equations and diagrams alongside every implementation block, making the math transparent.
- •Active maintenance bonus content added regularly as new techniques emerge.
- •Translated companion content community translations of book exercises into multiple languages.
LLMs from Scratch ecosystem:
- •The book itself "Build a Large Language Model (From Scratch)" by Sebastian Raschka, published by Manning Publications.
- •Author's blog and Substack Sebastian publishes related articles at sebastianraschka.com and the Ahead of AI newsletter.
- •YouTube companion accompanying video lessons that walk through key concepts visually.
- •Bonus directories chapter folders include extra exercises and alternative implementations.
Getting started:
Clone github.com/rasbt/LLMs-from-scratch and install PyTorch with pip install torch tiktoken matplotlib jupyter. Start with Chapter 2 (Tokenization) and progress chapter by chapter. The book is the recommended companion, but the code stands alone with extensive markdown commentary inside each notebook. You can train a small model on a CPU; full pretraining benefits from a GPU.
Limitations:
This is a learning resource, not a production training framework -- the code is optimized for clarity, not throughput. License is NOASSERTION on GitHub (custom permissive terms in the LICENSE file), so commercial use should be reviewed by your legal team. The book is paid (Manning Publications); the code is free but reads best alongside the chapters. Training a competitive model requires GPU resources well beyond what most laptops provide.
Our Verdict
LLMs from Scratch in 2026 is the most popular code-along resource for learning how large language models actually work under the hood. 95,000 GitHub stars, 14,500 forks, weekly bonus updates, and authored by rasbt (Sebastian Raschka), one of the most respected ML educators in the field.
For Vibe Builders, this repo will be over your head unless you are comfortable with Python, PyTorch, and basic linear algebra. If you want to build AI APPS without understanding internals, skip this and choose Awesome LLM Apps or Microsoft's Generative AI for Beginners. If you want to truly understand what is happening when you call an LLM API, this is the canonical resource.
For Developers wanting to go from "I use LLMs" to "I understand LLMs," this is the bar. Each notebook is small enough to read end-to-end and rich enough to teach the underlying mechanics. The bonus directories cover production-relevant topics like LoRA and instruction fine-tuning.
Skip LLMs from Scratch if you only need to USE LLMs through APIs -- a high-level tutorial fits better. Skip if you have no PyTorch experience and do not want to acquire it. For an even more theoretical, paper-first treatment, Andrej Karpathy's neural networks course (nanoGPT, makemore) covers similar ground with a more research-oriented lens.
Frequently Asked Questions
What is LLMs from Scratch?
LLMs from Scratch is the official companion code repository for Sebastian Raschka's book 'Build a Large Language Model (From Scratch)' published by Manning. Maintained by rasbt on GitHub, it walks through implementing a ChatGPT-style transformer LLM in PyTorch step by step. As of May 2026 it has 95,000 GitHub stars and 14,500 forks.
Do I need the book to use the code?
You do not strictly need the book. The notebooks contain extensive markdown commentary and stand on their own as a tutorial. The book provides deeper explanations, exercises, and historical context. Most learners get the most value from reading the book chapters alongside running and modifying the notebooks. The book is published by Manning and is the canonical reference.
What background do I need?
Comfortable Python, basic PyTorch, and basic linear algebra are the minimum. You should know what a tensor is, how a feedforward network works, and how gradient descent updates parameters. The book and code build everything else (attention, transformers, training loops) from these foundations. Absolute ML beginners may want to study a basic deep learning course first.
Can I train a competitive LLM with this code?
No. The code is optimized for clarity and pedagogical value, not for training-cluster throughput. You can train a small model that demonstrates the concepts on a single GPU or even a CPU, but training a model competitive with GPT-4 or Claude requires datacenter-scale compute, optimized training frameworks like Megatron-LM, and proprietary techniques beyond the scope of this repo.
Who is Sebastian Raschka?
Sebastian Raschka (rasbt on GitHub) is a machine learning researcher and educator, author of multiple ML books including the bestselling 'Build a Large Language Model (From Scratch).' He writes the Ahead of AI Substack newsletter and previously worked as a Staff Research Engineer at Lightning AI. He is widely regarded as one of the most accessible ML educators in the field.
What is LLMs-from-scratch?
LLMs from Scratch is rasbt's (Sebastian Raschka's) PyTorch code companion to the bestselling book 'Build a Large Language Model (From Scratch).' If you want to truly understand transformers, attention, and LLM training instead of just calling an API, this is the canonical learning resource.
How do I install LLMs-from-scratch?
Visit the GitHub repository at https://github.com/rasbt/LLMs-from-scratch for installation instructions.
What license does LLMs-from-scratch use?
LLMs-from-scratch uses the NOASSERTION license.
What are alternatives to LLMs-from-scratch?
Explore related tools and alternatives on My AI Guide.
Open source & community-verified
NOASSERTION licensed: free to use in any project, no strings attached. 95,266 developers have starred this, meaning the community has reviewed and trusted it.
Reviewed by My AI Guide for relevance, quality, and active maintenance before listing.
Topics