Build, Train, and Fine-Tune a GPT-Style LLM in PyTorch

Read before you buy: the PDF here is a free sample — the first 20 pages of the 449-page book (payments, sales and reviews on this demo site are sample data). Get the full e-book at thenexbooks on Gumroad.

The best way to understand a large language model is to build one — line by line, on the computer you already have.

Most developers now use language models through a chat window or an API. But when a model repeats itself, invents a fact, runs out of context, or costs more than expected, the developers who know what happens inside — tokens, embeddings, attention, logits, sampling — are the ones who can explain why and fix it.

This book starts from an empty folder and builds a complete GPT-style model in PyTorch. You write the tokenizer, the attention mechanism, the transformer, the training loop, and the text generator yourself, train the model in minutes on a CPU, measure it honestly, and then load the real released GPT-2 small weights into the same code. From there you fine-tune, instruction-tune, add LoRA, speed up generation with a KV cache, and shrink the model with quantization. The models are small on purpose; the ideas are the same ones used in the largest models, and every chapter explains what changes when you scale up.

What's inside

• What a language model really does: next-token prediction, probabilities, logits, and a first bigram model built from counts and then from a neural network
• PyTorch essentials for language models: tensors, shapes, broadcasting, autograd, modules, and a CPU-friendly setup
• Text into numbers: building a corpus, character-level tokenization, byte pair encoding written from scratch, and a comparison with a production tokenizer (tiktoken)
• Training windows: sliding-window datasets, DataLoaders, batches, and why the targets are shifted by one
• Embeddings and position, self-attention computed by hand, causal masking, multi-head attention, layer normalization, feed-forward networks, and residual connections
• The complete model: assembling QuillGPT, counting parameters, weight tying, and running it end to end
• Training: cross-entropy and perplexity, AdamW, learning-rate warmup and cosine decay, gradient clipping, checkpoints, overfitting, and regularization
• Generation: greedy decoding, temperature, top-k, and top-p sampling, with honest evaluation of the results
• Pretrained weights: the GPT-2 architecture and checkpoint format, loading GPT-2 small into your own code, and verifying it against a reference implementation
• Fine-tuning: a classification head, instruction tuning with prompt templates and loss masking, and LoRA written from scratch
• Efficiency and scale: the KV cache, int8 quantization, efficient inference, and what really changes with billions of parameters — data pipelines, compute budgets, distributed training, and modern architecture choices
• Shipping: packaging the model, a small API, safety, licensing, and a production checklist

One running example, start to finish. You help Quillbrook Press, a fictional small publisher of public-domain fables, build QuillGPT — a language model of its own. To keep every result reproducible and free of licensing questions, the training text is generated by a seeded script printed in the book and dedicated to the public domain. You train QuillGPT-mini (under one million parameters), load GPT-2 small into the same code, and finish with QuillGPT-Instruct, which answers simple questions about fables. The book also shows how to swap in real public-domain books when you are ready.

Current and honest. The code targets Python 3.12 and recent versions of PyTorch, and every training and generation listing was run on a CPU with two threads. Techniques that need a GPU — mixed precision, 4-bit loading, multi-GPU training — are explained and clearly labeled rather than shown with invented results. The book contains no invented benchmarks or statistics. Every chapter includes runnable code, Tips, Notes, Warnings, Try It Yourself exercises, and Key Takeaways, and the appendices give you a cheat sheet, templates and reference tables, a production checklist, and a glossary.

Format: PDF, 449 pages (A4), linked table of contents and bookmarks. English. First Edition, 2026. Published by thenexbooks. Requires basic Python; no GPU needed. This book is independent and is not affiliated with, sponsored by, or endorsed by OpenAI, the PyTorch project, Hugging Face, or any other company mentioned.

Table of Contents

Part I — Foundations

• Ch 1 · What a Language Model Is: Next-Token Prediction and the Road to QuillGPT
• Ch 2 · Setting Up the Workshop: Python, PyTorch on the CPU, and the Project Layout
• Ch 3 · PyTorch Essentials for Language Models: Tensors, Autograd, and Modules
• Ch 4 · A First Language Model: Bigram Counts and a Neural Bigram

Part II — Text into Numbers

• Ch 5 · The Quillbrook Fables Corpus: Collecting, Generating, and Cleaning Text
• Ch 6 · Character-Level Tokenization: Vocabularies, Encoding, and Decoding
• Ch 7 · Byte Pair Encoding from Scratch
• Ch 8 · Production Tokenizers: tiktoken, Special Tokens, and Trade-Offs
• Ch 9 · Training Windows: Datasets, DataLoaders, and Batches

Part III — Attention and the Transformer

• Ch 10 · Embeddings: Turning Token IDs into Vectors with Position
• Ch 11 · Self-Attention from Scratch: Queries, Keys, and Values
• Ch 12 · Causal Masking and Multi-Head Attention
• Ch 13 · The Transformer Block: Layer Normalization, Feed-Forward Networks, and Residuals
• Ch 14 · The Complete QuillGPT Model: Assembling, Counting, and Running It

Part IV — Training QuillGPT

• Ch 15 · Loss, Cross-Entropy, and Perplexity
• Ch 16 · The Training Loop: Optimizers, Steps, and Checkpoints
• Ch 17 · Training Well: Learning-Rate Schedules, Clipping, Overfitting, and Regularization
• Ch 18 · Generating Text: Greedy Decoding, Temperature, Top-k, and Top-p
• Ch 19 · Evaluating a Language Model: Validation Loss, Perplexity, and Reading Samples

Part V — Pretrained Weights and Fine-Tuning

• Ch 20 · Inside GPT-2: Architecture, Configuration, Checkpoints, and Licenses
• Ch 21 · Loading GPT-2 Weights into QuillGPT
• Ch 22 · Fine-Tuning for Classification: Sorting Fables by Lesson
• Ch 23 · Instruction Tuning: Building QuillGPT-Instruct
• Ch 24 · LoRA from Scratch: Parameter-Efficient Fine-Tuning

Part VI — Efficiency and Real Scale

• Ch 25 · Faster Generation with a KV Cache
• Ch 26 · Quantization and Efficient Inference
• Ch 27 · What Changes at Real Scale: Data, Compute, and Modern Architectures
• Ch 28 · Shipping QuillGPT: Packaging, a Small API, Safety, and Next Steps

Appendices
A · Language Model Cheat Sheet
B · Templates and Reference
C · Production Checklist
D · Glossary

Book details