What a Neural Network Actually Computes: From Token IDs to Matrix Multiplication

NLP models cannot process words directly, which is why tokenization[1] exists. Tokenization ends with a list of integers — token IDs like [30642, 1634, 318, ...]. That list is what actually gets fed into a model. This article answers why NLP models cannot process raw text directly. The answer is simple: NLP models consist of neural networks that perform matrix multiplication, which needs a list of numbers, not characters. One Layer of a Neural Network: Multiply, Add, Then a Small Nonlinear Step A neural network is built from layers, and each layer does one core operation: take a vector of numbers in, multiply it by a matrix of learned weights, add a bias, and produce a new vector of numbers out. Example A layer takes in a 3-number input vector. Input vector: [1.0, 0.5, 2.0] Weight matrix (2 rows, 3 columns — these numbers are what the network learns during training, starting from random values) import numpy as np weight_matrix = np.array([ [0.1, 0.2, 0.3], [0.4, 0.5, 0.6] ]) input_vector = np.array([1.0, 0.5, 2.0]) output = weight_matrix @ input_vector # @ is matrix multiplication in NumPy print(output) Output: [0.8 1.85] Output value 1 = (0.1×1.0) + (0.2×0.5) + (0.3×2.0) = 0.1 + 0.1 + 0.6 = 0.8 Output value 2 = (0.4×1.0) + (0.5×0.5) + (0.6×2.0) = 0.4 + 0.25 + 1.2 = 1.85 Activation Function This is the step applied after multiplication. A small non-linear function like ReLU, which is the rule "turn any negative number into zero," applied to the output. Without a non-linear step, stacking layers is mathematically pointless: a layer that doubles its input, followed by a layer that triples its input, is exactly equivalent to one layer that multiplies by 6 — no matter how many purely linear layers you chain, they always collapse into one. A non-linear function like ReLU breaks that collapse, which is why it's included after every layer. Stacking Layers: The "Deep" in Deep Learning A real network isn't one layer — it's many, stacked one after another. Layer 1's output vector becomes layer 2's input vector, and so on, each with its own learned weight matrix and activation function. Deep learning refers to networks with many such layers stacked in sequence. Each layer transforms the vector a little further, building progressively more useful representations of the original input as it passes through. Tying This Back to the RNN[2] The RNN's hidden state vector — "combine the current word with a running summary of everything seen so far" — is exactly this same operation. At every timestep, the RNN takes the current word's vector and the previous hidden state vector, multiplies each by its own learned weight matrix, adds the results together, and passes the combined output through an activation function to produce the new hidden state. At every timestep, the same weight matrix is applied to the previous hidden state. Because the hidden state is a fixed-size vector, and every timestep applies the same fixed-size weight matrices to it, there is only so much a fixed-size matrix multiplication can preserve as more and more information gets folded in step after step. The compression problem is a direct consequence of forcing arbitrarily long input through the same fixed-size matrix operation over and over, resulting in information dilution. Why Grouping Multiplications Into a Matrix Matters? You could describe a layer's computation as many individual multiply-and-add operations done sequentially. Writing it as one matrix multiplied by one vector instead, lets an entire layer's output be computed as a single operation — and this is exactly why GPUs matter for deep learning. GPUs are hardware built specifically to perform huge numbers of these multiplications and additions simultaneously, in parallel, rather than sequentially. This is also part of why transformers[2] could train at a scale that the RNN's sequential computation never could. What "Training" Actually Means at This Level? The numbers inside each weight matrix (for e.g. 0.1, 0.2, 0.3) aren't set by a person — they start as random values and get adjusted repeatedly during training, based on how far the network's output was from what it should have been (gradient descent[3]). "Training a neural network" means finding the specific numbers in every weight matrix, across every layer, that make the chain of matrix multiplications produce useful output. Where This Leaves Us? A neural network, essentially, is: token IDs get converted into meaningful vectors, those vectors get passed through a series of layers, each of which is a matrix multiplication followed by a small non-linear step, and the specific numbers inside those matrices are learned from data rather than hand-coded. The RNN's hidden state update is one particular arrangement of this operation, while the transformer's self-attention is another. A transformer reuses the same weight matrices across every token position within a layer — same underlying idea as the RNN — but it applies them all at once instead of sequentially. Across layers (depth), transformers use different weights per layer, unlike the RNN's single reused set. Token IDs are arbitrary integers assigned to tokens — they carry no meaning on their own. The next article covers how meaning gets assigned to a token ID. [1]: For details on tokenization, refer to How Text Becomes Numbers: Tokenization Explained from First Principles [2]: For details on RNN and transformers, refer to Rules to Learning: Why NLP Needed Transformers [3]: Gradient descent is the process a network uses to gradually improve its weight matrix numbers in the direction that reduces error.