Adapt Small Open Models with PyTorch, PEFT, and TRL — from Dataset to Deployment

Read before you buy: the PDF here is a free sample — the first 20 pages of the 429-page book (payments, sales and reviews on this demo site are sample data). Get the full e-book at thenexbooks on Gumroad.

Turn a small open model into a reliable specialist for your own task — and prove that it works.

Open language models are now small enough to run on ordinary hardware, but out of the box they ignore your output format, miss your business rules, and answer in a tone nobody asked for. Fine-tuning closes that gap: a few hundred good examples can teach a model your format, your vocabulary, and your decisions, so it runs privately and cheaply on machines you control. Done badly, it wastes time and produces a model that only looks good in a demo.

This book teaches the whole process with code you can run on an ordinary computer. You write the training loop yourself before letting a library hide it, measure every change against a fixed test set, and learn why each technique works, so you can apply it to your own data and models.

What's inside

• When fine-tuning beats prompting or retrieval-augmented generation (RAG), and when it does not
• How language models learn: tokens, logits, cross-entropy loss, gradients, and why the prompt is masked out of the loss
• Chat templates, special tokens, and generation settings — and the silent bugs they cause
• Measuring an honest baseline before you train anything
• Designing the task and output format, building a dataset from synthetic data and real logs, annotation guidelines
• Cleaning data: validation, de-duplication, PII redaction, leakage checks, and reproducible splits
• Tokenization, loss masking, padding, packing, and the Hugging Face datasets library
• A full fine-tune in plain PyTorch, then the same job with the Trainer API, and the hyperparameters that really matter
• LoRA adapters with PEFT, explained from the matrix math up, and QLoRA memory math for training larger models on one GPU
• Supervised fine-tuning and Direct Preference Optimization (DPO) with TRL, preference data collection, and an overview of KTO, ORPO, reward models, and RL-based methods
• Evaluation before and after: valid-JSON rate, field accuracy, exact match, confusion tables, rubric and pairwise judging, human review
• Debugging overfitting, catastrophic forgetting, and data bugs; red-teaming for prompt injection and unsafe behavior
• Merging adapters, safetensors, model cards, int8 quantization, ONNX and GGUF export, and serving with FastAPI
• Cost estimation, open-model licenses, and governance

One running example, start to finish. You build a claims triage assistant for Marrowfield Mutual, a fictional insurer. Customers report a cracked windshield, a burst pipe, a lost suitcase, or a sick dog, and the model must return clean JSON with the claim type, severity, route, and missing information. A seeded script generates 300 synthetic messages with known answers, so every result in the book can be reproduced. You measure the untuned model, fine-tune it fully and with LoRA, add preference tuning, evaluate and red-team it, and finish with a merged, quantized model behind an API.

Practical and honest. The examples use a deliberately small open model and short runs that finish on a CPU, so you can follow along without a GPU; GPU-only code such as 4-bit QLoRA is clearly labeled, and the book explains what changes at full scale. Every number printed in the book comes from running the code. It contains no invented benchmarks or statistics. Every chapter includes runnable code, Tips, Notes, Warnings, exercises, and Key Takeaways, and the appendices give you a cheat sheet, templates and reference tables, a production checklist, and a glossary.

Format: PDF, 429 pages (A4), linked table of contents and bookmarks. English. First Edition, 2026. Published by thenexbooks. Requires basic Python. This book is independent and is not affiliated with, sponsored by, or endorsed by any model publisher or library maintainer.

Table of Contents

Part I — Foundations

• Ch 1 · Why Fine-Tune? Prompting, RAG, and Fine-Tuning Compared
• Ch 2 · Setting Up Your Workbench: Python, PyTorch, and Small Open Models
• Ch 3 · How Language Models Learn: Tokens, Logits, Loss, and Gradients
• Ch 4 · Chat Templates, Special Tokens, and Generation Settings
• Ch 5 · Baselines First: Measuring the Untuned Model

Part II — Data

• Ch 6 · Designing the Task: Output Formats, Labels, and Acceptance Criteria
• Ch 7 · Building a Training Set: Synthetic Data, Real Logs, and Annotation
• Ch 8 · Cleaning and Validating Data: Duplicates, PII, Leakage, and Splits
• Ch 9 · From Records to Tensors: Tokenization, Loss Masking, Padding, and Packing
• Ch 10 · Working with Hugging Face Datasets: Loading, Mapping, and Versioning

Part III — Training

• Ch 11 · A Full Fine-Tune in Plain PyTorch
• Ch 12 · The Trainer API: Arguments, Callbacks, Checkpoints, and Logging
• Ch 13 · Hyperparameters That Matter: Learning Rate, Batch Size, Epochs, and Schedules
• Ch 14 · LoRA with PEFT: Low-Rank Adapters Explained and Applied
• Ch 15 · QLoRA and Memory Math: Training Bigger Models on One GPU

Part IV — Alignment with TRL

• Ch 16 · Supervised Fine-Tuning with TRL's SFTTrainer
• Ch 17 · Preference Data: Chosen, Rejected, and How to Collect Them
• Ch 18 · Direct Preference Optimization (DPO) in Practice
• Ch 19 · Beyond DPO: KTO, ORPO, Reward Models, and RL-Based Tuning

Part V — Evaluation and Safety

• Ch 20 · Evaluating a Fine-Tuned Model: Held-Out Sets, Field Accuracy, and Error Analysis
• Ch 21 · Beyond Exact Match: Rubrics, Model-Based Judges, and Human Review
• Ch 22 · Debugging Fine-Tunes: Overfitting, Forgetting, and Data Bugs
• Ch 23 · Safety, Bias, and Red-Teaming a Fine-Tuned Model

Part VI — Shipping

• Ch 24 · Merging Adapters and Saving Models: safetensors and Model Cards
• Ch 25 · Export and Quantization: int8, ONNX, and GGUF
• Ch 26 · Serving Your Model: FastAPI, Batching, and Inference Servers
• Ch 27 · Cost, Licensing, and Governance of Open Models
• Ch 28 · Case Study: Shipping the Marrowfield Claims Assistant

Appendices
A · Fine-Tuning Cheat Sheet
B · Templates and Reference
C · Production Checklist
D · Glossary

Book details