The previous lesson established the mental model: vectors represent objects, matrices transform representations, and AI systems compose these transformations. This lesson turns that intuition into operational fluency.

You will learn to calculate and debug the operations that appear in neural layers, embeddings, attention, losses, normalization, and batched model code.

By the end, you will be able to:

  • distinguish vector addition, elementwise multiplication, dot products, and outer products;
  • calculate matrix-vector and matrix-matrix products and predict their shapes;
  • explain linear combinations, span, independence, basis, and rank;
  • use transpose, reshape, axes, and broadcasting deliberately;
  • solve a linear system without explicitly computing an inverse;
  • translate the same operation between mathematics, NumPy, and PyTorch.

Four vector operations that must not blur together#

Let a = [1, 2, 3] and b = [4, 5, 6].

Addition combines matching coordinates

a + b = [5, 7, 9]

Addition requires vectors in the same coordinate system. In a Transformer, token and positional representations can be added because they have the same hidden dimension and the model is trained around that shared representation.

Scalar multiplication changes magnitude and possibly direction

3a = [3, 6, 9]

A positive scalar preserves direction, zero collapses the vector, and a negative scalar reverses direction. Learning rates scale gradient updates; normalization layers rescale activations; attention weights scale value vectors.

Elementwise multiplication applies a coordinate-wise gate

a elementwise-times b = [4, 10, 18]

This is called the Hadamard product. It appears in masks, feature gates, dropout, and gated neural architectures.

The dot product contracts two vectors into one scalar

a dot b = 4 + 10 + 18 = 32

The word “contracts” is useful: an axis of length three disappears because its products are summed.

import numpy as np

a = np.array([1.0, 2.0, 3.0])
b = np.array([4.0, 5.0, 6.0])

print(a + b)       # addition
print(3 * a)       # scalar multiplication
print(a * b)       # elementwise / Hadamard product
print(a @ b)       # dot product

The same * and @ distinction applies to PyTorch tensors. Confusing them is dangerous because both operations may run and return plausible-looking output.

The outer product creates pairwise interactions#

The outer product preserves both vector axes instead of summing them:

a outer b = [[a_1b_1, a_1b_2], [a_2b_1, a_2b_2], ...]

For a = [1, 2] and b = [3, 4, 5], the result has shape [2, 3]:

a = np.array([1, 2])
b = np.array([3, 4, 5])

outer = np.outer(a, b)
print(outer)
print(outer.shape)
[[ 3  4  5]
 [ 6  8 10]]
(2, 3)

Outer products appear in covariance calculations, low-rank updates, some gradient derivations, and pairwise feature interactions. LoRA constructs a low-rank matrix update from products of smaller matrices—the higher-dimensional relative of this idea.

Linear combinations, span, and basis#

A linear combination scales vectors and adds them:

c_1v_1 + c_2v_2 + ... + c_kv_k

The span of a set of vectors is every vector reachable through their linear combinations. If e_1 = [1,0] and e_2 = [0,1], any 2D vector can be written as xe_1 + ye_2. These two vectors form the standard basis for 2D space.

Vectors are linearly independent when none can be reconstructed from the others. Replace e_2 with [2,0], and both vectors lie on the same horizontal line. They span only one direction.

This language becomes practical in AI:

  • an embedding matrix defines learned directions available to represent tokens;
  • rank measures the number of independent input-output directions a matrix can express;
  • low-rank adaptation assumes a useful update can live in a much smaller subspace;
  • PCA searches for a compact set of directions explaining variation in data.

Matrix-vector multiplication is a set of dot products#

Let:

A = [[2, 1], [-1, 3]], x = [4, 2]

Each row of A takes a dot product with x:

Ax = [2(4)+1(2), -1(4)+3(2)] = [10, 2]

Depending on notation and library layout, examples may place input vectors in rows and multiply x @ W, or use column vectors and write Wx. Neither convention is inherently more correct. Never transpose mechanically; state the convention and follow the axes.

Matrix multiplication composes transformations#

If A maps from n input features to m intermediate features and B maps from m to p output features, their composition maps directly from n to p.

[batch, n] @ [n, m] @ [m, p] → [batch, p]

Matrix multiplication is associative:

(XA)B = X(AB)

but not generally commutative:

AB != BA

Order matters because the second transformation operates on the space created by the first. Shapes may also make only one order legal.

Linear maps preserve structure

A matrix represents a linear map because multiplying by it preserves vector addition and scalar multiplication:

A(x + z) = Ax + Az, \qquad A(cx) = c(Ax)

These rules imply A0=0. They also explain why the span of vectors is a subspace: it contains zero and remains closed under addition and scalar multiplication. A basis is a minimal set of independent directions spanning that subspace, and rank is the dimension of the output subspace reachable through a matrix.

Adding a bias gives Ax+b, an affine rather than linear map. Affine maps can translate the origin while the matrix still controls how directions are mixed. This is why a neural-network “linear” layer can move an activation cloud as well as reshape it.

Interactive lab

Build a legal matrix product

Change the outer and shared dimensions. Watch the shared axis contract while the two outer axes become the output shape.

Calculate a product manually

A = [[1, 2, 3],
     [4, 5, 6]]       shape [2, 3]

B = [[1, 2],
     [3, 4],
     [5, 6]]          shape [3, 2]

The result has shape [2, 2]. Entry (0, 0) is row 0 of A dotted with column 0 of B:

1(1) + 2(3) + 3(5) = 22

The complete result is:

[[22, 28],
 [49, 64]]

Try it yourself

Compute and explain a batch projection

An input tensor X has shape [64, 512]. A weight matrix W has shape [512, 2048], and bias b has shape [2048].

  1. Predict the shape of X @ W.
  2. Explain how b is added.
  3. Estimate the number of scalar multiply-add terms used to produce the batch output.
  4. State what each axis means.
Reveal solution

X @ W has shape [64, 2048]. Broadcasting treats b as one output-feature bias applied to each of the 64 rows. The computation produces 64 × 2048 outputs, each summing 512 products: about 64 × 2048 × 512 = 67,108,864 multiply-accumulate positions. Axis 0 selects a batch example, the contracted axis represents 512 input features, and the final axis represents 2,048 output features.

Transpose reorients axes#

For a 2D matrix, transpose swaps rows and columns:

(A^T)_{ij} = A_{ji}
A = np.arange(6).reshape(2, 3)
print(A.shape)    # (2, 3)
print(A.T.shape)  # (3, 2)

For tensors with more than two axes, “transpose” is incomplete without naming the axes. In attention, keys might move from [B, H, T, D] to [B, H, D, T] so query-key multiplication contracts D and produces [B, H, T, T].

import torch

K = torch.randn(2, 4, 16, 32)       # [B, H, T, D]
K_t = K.transpose(-2, -1)            # [B, H, D, T]
print(K_t.shape)

reshape, transpose, and permute do different jobs. Reshape changes grouping while preserving element order when layout permits; transpose or permute changes axis order. A tensor can have the desired shape and still attach the wrong meaning to an axis.

Broadcasting aligns from the right#

NumPy and PyTorch compare dimensions from right to left. Dimensions are compatible when equal or when one is 1.

X: [32, 128, 768]
b:           [768]
result: [32, 128, 768]

This applies one 768-feature bias to every token and batch item.

mask: [32, 128,   1]
X:    [32, 128, 768]

This applies one mask value per token across all features. Both operations are legal but mean different things.

Axes tell reductions what to remove#

Given X with shape [batch, features]:

  • X.sum() removes every axis and returns one scalar;
  • X.sum(axis=0) removes the batch axis and returns one value per feature;
  • X.sum(axis=1) removes the feature axis and returns one value per example;
  • keepdims=True retains the reduced axis with length one for later broadcasting.

Stable softmax over the final dimension uses this logic:

def softmax(x, axis=-1):
    shifted = x - np.max(x, axis=axis, keepdims=True)
    exp = np.exp(shifted)
    return exp / exp.sum(axis=axis, keepdims=True)

The axis is part of the operation’s meaning. Softmax over vocabulary answers “which token?” Softmax over batch items answers a different and usually unintended question.

Rank measures independent transformation directions#

Consider:

A = [[1, 2],
     [2, 4]]

The second row is twice the first. The matrix has rank one even though it has two rows and two columns. It preserves only one independent direction and collapses information along another.

A = np.array([[1.0, 2.0], [2.0, 4.0]])
print(np.linalg.matrix_rank(A))  # 1

Rank depends on numerical tolerance in floating-point computation. A theoretically nonzero singular value may be too small to be useful, which leads to the practical idea of numerical rank and conditioning.

Inverse is a concept; solve is the operation#

For a square invertible matrix, A^{-1}A = I. The inverse reverses the transformation. But explicitly forming an inverse is usually the wrong way to solve Ax=b.

A = np.array([[3.0, 1.0], [1.0, 2.0]])
b = np.array([9.0, 8.0])

x = np.linalg.solve(A, b)
print(x)                 # [2. 3.]
print(np.allclose(A @ x, b))

solve can use an appropriate factorization without materializing the inverse, improving efficiency and often numerical behavior. If a matrix is singular or nearly singular, the problem requires least squares, regularization, or a different model—not a forced inverse.

Batch operations: write the loop, then remove it#

Start with the meaning:

outputs = []
for row in X:
    outputs.append(row @ W + b)
Y_loop = np.stack(outputs)

Then express the same calculation as one batched operation:

Y_vectorized = X @ W + b
assert np.allclose(Y_loop, Y_vectorized)

Vectorization moves iteration into optimized kernels, reduces interpreter overhead, and exposes regular parallel work to CPUs and accelerators. It does not remove computation; it organizes computation for hardware.

Try it yourself

Debug three operations

For each expression, decide whether it is valid and whether it matches the stated intent.

  1. scores = Q * K.T for pairwise query-key dot products, where both are [T, D].
  2. centered = X - X.mean(axis=0) for centering every feature across a dataset [N, D].
  3. prediction = np.linalg.inv(A) @ b for solving a well-conditioned square system.
Reveal solution
  1. It is generally shape-incompatible when T != D, and even if square it performs elementwise multiplication rather than all pairwise dot products. Use Q @ K.T.
  2. It is valid and intentional: the mean has shape [D] and broadcasts across N rows. keepdims=True can make the aligned [1, D] shape explicit.
  3. It can produce an answer but should be replaced with np.linalg.solve(A, b) for efficiency and numerical reasons.

Common engineering mistakes#

  • Transposing until code runs: shapes become legal while axis semantics become wrong.
  • Dropping a batch dimension accidentally: [1, D] and [D] often broadcast differently in later code.
  • Using view after a non-contiguous permutation: in PyTorch, use reshape or make layout contiguous when required.
  • Reducing the wrong axis: loss, normalization, and softmax may silently optimize a different objective.
  • Assuming exact rank: floating-point thresholds and conditioning determine usable information.
  • Materializing huge intermediates: an outer product or pairwise matrix may require quadratic memory.
What does [m, n] @ [n, p] produce?

Why should np.linalg.solve(A, b) usually replace np.linalg.inv(A) @ b?

Study optional flashcards

Hadamard product

Elementwise multiplication of matching coordinates.

Outer product

Pairwise products of two vectors, producing a matrix.

Linear combination

A sum of vectors multiplied by scalar coefficients.

Span

Every vector reachable through linear combinations of a set.

Rank

The number of independent directions represented by a matrix.

Transpose

An operation that swaps or reorders axes.

Broadcasting

Rules for aligning compatible dimensions, beginning from the right.

Memory chart#

a + b       combine matching coordinates
a * b       elementwise gate
a @ b       dot product → scalar
outer(a,b)  pairwise interactions → matrix

[m,n] @ [n,p] → [m,p]
inner axis contracts; outer axes survive

transpose: reorder axes
reshape: regroup elements
reduce(axis=k): remove axis k
broadcast: align dimensions from the right

rank: independent directions
solve(A,b): preferred numerical solution to Ax=b

The next lesson uses these operations to solve linear systems, understand when solutions exist, derive least squares, and connect residual geometry to regression.

References#