The solutions are summarized in Appendix C, and the corresponding code notebooks are available in the main chapter folders of this repository (for example, ./ch02/01_main-chapter-code/exercise-solutions.ipynb. In addition, this book includes code for loading the weights of larger pretrained models for finetuning. In Build a Large Language Model (From Scratch), you'll learn and understand how large language models (LLMs) work from the inside out by coding them from the ground up, step by step. This repository contains the code for developing, pretraining, and finetuning a GPT-like LLM and is tescoslots the official code repository for the book Build a Large Language Model (From Scratch). If you find this book or code useful for your research, please consider citing it.

In addition to the code exercises, you can download a free 170-page PDF titled Test Yourself On Build a Large Language Model (From Scratch) from the Manning website. It starts with a pretrained model and implements different reasoning approaches, including inference-time scaling, reinforcement learning, and distillation, to improve the model's reasoning capabilities. The course is organized into chapters and sections that mirror the book's structure so that it can be used as a standalone alternative to the book or complementary code-along resource. A 17-hour and 15-minute companion video course where I code through each chapter of the book. The code in the main chapters of this book is designed to run on conventional laptops within a reasonable timeframe and does not require specialized hardware. This book uses PyTorch to implement the code from scratch without using any external LLM libraries. With this knowledge, you will be well prepared to explore the fascinating world of LLMs and understand the concepts and code examples presented in this book.

In this book, I'll guide you through creating your own LLM, explaining each stage with clear text, diagrams, and examples.

Repository Files Navigation

If you have downloaded this code bundle from the Manning website and are viewing it on your local computer, I recommend using a Markdown editor or previewer for proper viewing. Implement a ChatGPT-like LLM in PyTorch from scratch, step by step Similar to Build A Large Language Model (From Scratch), Build A Reasoning Model (From Scratch) takes a hands-on approach implementing these methods from scratch. Build A Reasoning Model (From Scratch), while a standalone book, can be considered as a sequel to Build A Large Language Model (From Scratch). Additionally, the code automatically utilizes GPUs if they are available. This approach ensures that a wide audience can engage with the material. The mental model below summarizes the contents covered in this book.