Norms, Distances, and Similarity for AI Engineers

An embedding model turns an item into a vector, but the vector alone does not say which other item is “closest.” A recommendation system, clustering pipeline, anomaly detector, and training loss all need a rule for comparing vectors. That rule is a modeling decision: different rules can return different neighbors from the same data.

This lesson develops that decision from first principles. By the end, you will be able to:

• compute and interpret L_1, L_2, general L_p, and max norms;
• distinguish a norm, a distance metric, a similarity score, and a loss;
• calculate Manhattan distance, Euclidean distance, dot product, and cosine similarity by hand;
• track batch and feature axes in NumPy and PyTorch;
• explain when feature scale or vector magnitude changes a ranking;
• implement pairwise distance and cosine retrieval with explicit zero-vector handling;
• choose and validate a comparison rule for retrieval and clustering.

Prerequisites: vectors, dot products, array axes, and the residual norm from Linear Systems and Least Squares.

Four related objects with different contracts#

Let x,y \in R^D, where D is the number of features.

A norm must satisfy three properties for vectors x,y and scalar a:

1. Positive definiteness: ||x|| \ge 0, and ||x||=0 only when x=0.
2. Absolute homogeneity: ||ax||=|a|\,||x||.
3. Triangle inequality: ||x+y|| \le ||x||+||y||.

A norm creates a distance by measuring the difference:

That distance is a metric: it is non-negative, equals zero only for identical inputs, is symmetric, and satisfies the triangle inequality. Not every useful score has this contract. A dot product can be negative, and cosine similarity is a similarity rather than a distance. Names in APIs can be pragmatic: always check the mathematical behavior an algorithm requires.

The L_p family measures vector size#

For p \ge 1, the L_p norm is

Here, x_i is feature i, D is the feature count, and p controls how strongly large coordinate differences dominate.

L_1: total absolute movement

For x=[3,-4], ||x||_1=3+4=7. The induced Manhattan distance adds the absolute movement along each coordinate. It does not square large deviations, so one extreme coordinate has less leverage than under squared L_2 loss.

L_2: straight-line length

For the same vector, ||x||_2=\sqrt{9+16}=5. This is Euclidean length. It connects directly to the dot product and to the least-squares residual from the previous lesson.

L_\infty: largest coordinate magnitude

For [3,-4], the max norm is 4. Use it when the worst coordinate deviation is the binding constraint—for example, when no component may exceed a tolerance.

The expression sometimes called “L_0” counts nonzero components. It is useful as a sparsity measure, but it is not a norm: multiplying a nonzero vector by two does not multiply its nonzero count by two, so absolute homogeneity fails. Likewise, the L_p formula with 0<p<1 is not a norm because the triangle inequality can fail.

Work three comparisons completely by hand#

Take

Their difference is x-y=[-3,-4].

Manhattan distance:

Euclidean distance:

Cosine similarity uses the angle rather than the difference:

Calculate each term:

The vectors are five Euclidean units apart yet point in almost the same direction. Distance and angular similarity answer different questions; neither result contradicts the other.

Cosine removes magnitude—and that is a choice#

For nonzero vectors, cosine similarity lies between -1 and 1 by the Cauchy–Schwarz inequality:

• 1: same direction;
• 0: orthogonal under the standard dot product;
• -1: opposite directions.

If a>0, then

so [1, 0] and [100, 0] are identical by cosine despite very different magnitudes. This is useful when direction carries semantics and magnitude is a nuisance. It is harmful when magnitude carries confidence, frequency, intensity, or another real signal.

Cosine similarity is undefined if either vector is zero because its denominator is zero. Libraries choose different practical policies. Your application contract should be explicit: reject zero embeddings, filter them, replace them upstream, or define a documented fallback. Silently adding an epsilon changes the function near zero.

For row-normalized vectors x'=x/||x||_2 and y'=y/||y||_2:

and

Therefore dot-product, cosine, and Euclidean rankings are closely related after exact L_2 normalization. Without normalization, dot product also rewards magnitude.

Drag canvas to move−

100%

+FitFull screen

Diagram key and text version

• Entry
• Process
• Outcome

1. vectors [N, D] → validate units and scale: inspect contract
2. validate units and scale → Lp distance: coordinate difference: magnitude meaningful
3. validate units and scale → L2 normalize rows: remove magnitude
4. L2 normalize rows → cosine: direction: compare
5. validate units and scale → dot product: direction + magnitude: retain magnitude

Text fallback: validate vector shape, units, and feature scale first. Use an L_p distance to compare coordinate differences, normalize before cosine when direction should dominate, or retain raw magnitudes for dot-product scoring when magnitude is meaningful.

Shapes: reduce the feature axis, preserve the batch#

Suppose a query matrix has Q rows, a candidate matrix has N rows, and every embedding has D features:

queries:    [Q, D]
candidates: [N, D]
pairwise differences: [Q, N, D]
reduce feature axis D → distances [Q, N]

For one vector per row, axis=-1 in NumPy or dim=-1 in PyTorch reduces features and preserves batch dimensions.

import numpy as np

queries = np.array([[1.0, 0.0], [0.0, 1.0]])       # [Q=2, D=2]
candidates = np.array([[1.0, 1.0], [2.0, 0.0]])    # [N=2, D=2]

delta = queries[:, None, :] - candidates[None, :, :]  # [2, 2, 2]
euclidean = np.linalg.norm(delta, ord=2, axis=-1)      # [2, 2]

query_unit = queries / np.linalg.norm(queries, axis=-1, keepdims=True)
candidate_unit = candidates / np.linalg.norm(candidates, axis=-1, keepdims=True)
cosine = query_unit @ candidate_unit.T                 # [2, 2]

keepdims=True keeps the reduced feature axis as size one, so division broadcasts back across [Q, D]. For large Q and N, explicitly materializing [Q,N,D] may exhaust memory. Use a library pairwise routine, matrix identities, or blocks sized for the target hardware.

Current APIs make the axis contract visible:

# NumPy: vector norm for every row
row_norms = np.linalg.vector_norm(candidates, axis=-1)

# PyTorch: same shape rule, optional higher-precision accumulation via dtype
import torch
tensor = torch.tensor(candidates, dtype=torch.float32)
row_norms_torch = torch.linalg.vector_norm(tensor, ord=2, dim=-1)

Prefer torch.linalg.vector_norm for vector norms; the older torch.norm interface is deprecated. Scikit-learn’s cosine_similarity(X,Y) accepts [samples, features] matrices and returns [samples_X, samples_Y] scores, including sparse inputs under its documented options.

Feature scale can decide the neighbor#

Consider a customer represented as [age_in_years, income_in_dollars]:

query A:     [18, 55,000]
centroid B:  [20, 40,000]
centroid C:  [50, 56,000]

Raw Euclidean distance is dominated by dollar differences, so A is assigned to C despite a 32-year age gap. The formula is behaving correctly; the representation mixes incomparable units.

Common responses include:

• standardize each feature with training-set mean and standard deviation;
• use robust scaling when outliers make mean and standard deviation unstable;
• apply domain weights with documented units and validation;
• learn a representation or metric from task feedback;
• avoid including a feature whose scale or meaning cannot be made consistent.

Fit preprocessing on training data only, persist it with the model or index, and apply the identical transformation to queries and candidates. Re-fitting on each request or on evaluation data creates incompatible geometry or leakage.

Try it yourself

Predict the ranking before calculating

A query is q=[1,0]. Candidates are a=[10,1] and b=[1,0].

1. Which candidate has the larger dot product with q?
2. Which has the larger cosine similarity?
3. What property creates the disagreement?

Reveal solution

q·a=10 and q·b=1, so dot product chooses a. Cosine gives approximately 10/sqrt(101)=0.995 for a and exactly 1 for b, so cosine chooses b. Dot product retains magnitude while cosine divides it out and compares direction.

Retrieval and clustering inherit the metric#

In retrieval, the comparison rule defines relevance operationally. If embeddings were trained with a normalized cosine objective, serving with raw dot product may change rankings. If vector norms encode useful confidence, normalizing at serving time may delete signal. Treat the training objective, index configuration, stored-vector preprocessing, and query preprocessing as one versioned contract.

In nearest-centroid clustering, assigning point x means selecting the centroid with the smallest configured distance. Changing from L_2 to L_1, standardizing features, or normalizing every row can change assignments and the meaning of a cluster. Classic k-means specifically minimizes squared Euclidean distances to arithmetic means; swapping in an arbitrary distance does not preserve that objective. Other clustering algorithms are appropriate for other geometries.

Metric choice should be evaluated with the downstream task:

1. Define what a relevant neighbor or useful cluster means.
2. Create representative labeled pairs, queries, or stability checks.
3. Compare candidate metrics with the exact preprocessing and serving precision.
4. Slice results by vector norm, sparsity, language, tenant, or other meaningful cohort.
5. Measure latency, memory, recall, and task quality together.

Production failure modes and debugging#

A repeatable debugging pass is:

1. Assert shapes and identify the feature axis.
2. Reject non-finite values and inspect zero-vector frequency.
3. Print feature units, ranges, and preprocessing version.
4. Recompute a few examples by hand and with an independent implementation.
5. Check symmetry and zero diagonal for a claimed metric’s distance matrix.
6. Inspect score distributions and rankings, not only average scores.
7. Compare the exact offline and serving paths on fixed fixtures.
8. Block pairwise computation and benchmark realistic sizes.

Do not clamp all suspicious outputs into an accepted range before finding their cause. Clipping a cosine value from 1.0000001 to 1 is a reasonable final roundoff guard; converting NaN from a zero vector to zero is an undocumented semantic decision.

Try it yourself

Choose a comparison contract

You are clustering products represented by [price_in_dollars, weight_in_grams, category_embedding...]. List two problems with raw Euclidean distance and propose a validation plan.

Reveal solution

Price and weight use different units and may dominate the embedding coordinates; their raw ranges can also dwarf semantic dimensions. Begin by deciding whether price and weight should affect semantic grouping. Fit a documented scaler or domain weighting on training data, compare raw/scaled/semantic-only representations on labeled product pairs, inspect results by category and price band, and reuse the chosen transformation unchanged in serving.

Which statement is required of every mathematical norm?

Scaling a vector by a scalar scales its norm by the scalar's absolute valueEvery norm is computed from a dot productEvery norm squares its coordinatesA norm can be negative for an opposite direction

Check answer

Why can cosine and dot-product retrieval rank the same candidates differently?

Dot product includes vector magnitude, while cosine divides by both magnitudesCosine can only compare integer vectorsDot product always lies between negative one and oneCosine uses Manhattan distance internally

Check answer

What is wrong with calling squared Euclidean distance a metric?

It can violate the triangle inequalityIt can be negativeIt is asymmetricIt is undefined for equal vectors

Check answer

Study optional flashcards

Norm

A non-negative vector-size function satisfying positive definiteness, absolute homogeneity, and the triangle inequality.

Flip card

Metric

A distance satisfying identity, symmetry, and the triangle inequality.

Flip card

L1 norm

Sum of absolute coordinate magnitudes.

Flip card

L2 norm

Square root of the sum of squared coordinates; Euclidean length.

Flip card

Cosine similarity

Normalized dot product measuring orientation of two nonzero vectors.

Flip card

Feature scaling

A fitted transformation that makes coordinate magnitudes comparable under the chosen geometry.

Flip card

Memory chart#

vector x [D]                  batch X [N,D]
||x||p = (sum |xi|^p)^(1/p)  reduce feature axis → [N]

L1: total absolute change
L2: straight-line length
L∞: largest coordinate

distance:   d(x,y) = ||x-y||
dot:        xᵀy                  direction + magnitude
cosine:     xᵀy/(||x|| ||y||)   direction; undefined at zero

same vectors + different metric or scaling → different neighbors
validate units → fit preprocessing → match training and serving → test task quality

The next lesson studies matrix transformations and eigenvalues. Norms will let us measure how a transformation stretches vectors, while eigenvectors reveal directions whose orientation the transformation preserves.

References#

• Deisenroth, Faisal, and Ong: Mathematics for Machine Learning, Sections 3.1–3.4 — norm axioms, L_1 and L_2, metrics, inner products, angles, and orthogonality.
• Goodfellow, Bengio, and Courville: Deep Learning, Section 2.5 — L_p, squared L_2, L_1, max, and Frobenius norms in machine learning.
• NumPy: linalg.vector_norm — vector norms, axes, batches, orders, and keepdims behavior.
• PyTorch: torch.linalg.vector_norm — tensor dimensions, dtypes, and vector-norm behavior.
• PyTorch: cosine similarity — dimension reduction and numerical denominator behavior.
• scikit-learn: cosine similarity — normalized dot product, sparse inputs, and pairwise output shapes.
• scikit-learn: pairwise distances — supported metrics, input forms, and pairwise matrix behavior.