HowLLMsWork: from tokens to text generation from scratch

A Python/NumPy implementation for understanding Transformer and LLM internals, from tokenization and attention to training, sampling, autoregressive generation, and KV cache.

Start a conversation View related solution

Context

Production language models are complex, and many internals disappear behind high-level frameworks. This project is not an attempt to reproduce production scale; it is an attempt to make the core data flow and mathematics from tokens to next-token prediction explicit and inspectable.

Problem

The project follows the full path: text to tokens, embeddings, Q/K/V projections and attention, Transformer blocks, loss and gradients, then generation and sampling, while keeping the core implementation independent of a high-level model API.

Constraints & requirements

Python and NumPy as the implementation foundation
Small, inspectable components for attention, training, and inference
Explicit backward and gradient-flow implementations for important components
Greedy, temperature, top-k, and top-p sampling
Cached inference with prefill/decode and KV cache
Clear separation between backbone, generation, and sliding-context policy

Approach

text

Text -> Tokenization -> Embeddings -> Q/K/V
     -> Attention -> Transformer Block -> Logits
     -> Loss -> Backpropagation -> Update
     -> Autoregressive Generation -> Sampling -> KV Cache

The repository is organized as small components so each stage can be studied and tested independently. Inference has both direct and cached paths, with explicit prefill/decode interfaces.

Key decisions & trade-offs

Decision	Reason
Build from scratch with NumPy	The objective is understanding mechanics, not production-scale performance.
Separate training and inference	Training flow stays distinct from generation and cache semantics.
Sampling as a strategy	Temperature, top-k, and top-p effects remain explicit and comparable.
KV cache with prefill/decode	Repeated autoregressive computation becomes a visible system concern.
Keep sliding-window policy above the backbone	The backbone retains a clear context contract while generation owns window policy.

Outcome

The result is an end-to-end learning path from tokenization to generation, with Transformer components, training objectives, sampling, and KV-cache inference turned into small, inspectable implementations. It is a learning-oriented technical system rather than an attempt to compete with production LLMs.

Technical depth

Token embeddings and positional information
Scaled dot-product attention and causal masking
Multi-head attention, residual connections, LayerNorm, and FFN
Cross-entropy, backpropagation, and parameter updates
Greedy, temperature, top-k, and top-p sampling
KV cache, cached attention, and prefill/decode
Sliding context window and separated generation policy

Scaled dot-product attention

\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_h}}\right)V

Evidence

The implementation and learning path are publicly inspectable on GitHub.

GitHub — HowLLMsWork

The repository contains tokenization, attention, Transformer, training, inference, and KV-cache components in a study-oriented structure.

Open source

Need an AI capability beyond an API wrapper?

The problem can start from data, model choice, and evaluation, then move into a capability that fits the product.

Start a conversation