Recently, I've been self-studying vector knowledge from high school mathematics. The dot product is the most commonly used vector operation in machine learning. This article will introduce the dot product operation and its underlying mathematical significance.
Background Knowledge
For two vectors $a$ and $b$ with length $n$,
$$
\begin{aligned}
a &= [a_1, a_2, a_3,\dots,a_n]\\
b &= [b_1, b_2, b_3,\dots,b_n]
\end{aligned}
$$
the dot product can be used to represent their position relationship. For example, do they point in the same direction or in opposite directions? Are they perpendicular to each other?
The result of the dot product of two vectors is a scalar, hence the dot product is also sometimes referred to as scalar product.
There are two forms of dot product expression - geometric expression and algebraic expression - let's take a look at these two forms separately
Geometric Perspective
The geometric form of the dot product is as follows:
$$
a \cdot b = \Vert a \Vert \Vert b \Vert \cos\theta
$$
The formula above consists of three parts:
- $\Vert a \Vert$: The magnitude of vector $a$
- $\Vert b \Vert$: The magnitude of vector $b$
- $\enspace\theta\enspace$: The angle between vectors $a$ and $b$
Magnitude of a Vector
The length of a vector visually appears as the length of the hypotenuse calculated using the Pythagorean theorem. For a 2-dimensional vector, it is $\sqrt{x^2 + y^2}$; for a 3-dimensional vector, it is $\sqrt{x^2+y^2+z^2}$. By extension, the length of an $n$-dimensional vector is
$$
\Vert v \Vert = \sqrt{v_1^2 + v_2^2 + \dots + v_n^2}
$$
Consine
$\cos\theta$ projects vector $a$ onto vector $b$. In the illustration above, vector $a$ and vector $b$ point in different directions, so $\Vert a \Vert \cos\theta$ projects part of vector $a$ onto the direction of vector $b$. Conversely, projecting vector $b$ onto vector $a$ is also possible.
Example
When the angle and magnitude between vectors are known, the geometric perspective is very useful, as shown in the example above. In this case, calculating the dot product is straightforward.
$$
\begin{aligned}
\Vert a \Vert &= \sqrt{(-6)^2+8^2} = 10\\
\Vert b \Vert &= \sqrt{5^2+12^2} = 13\\
\theta &= {59.5}^\circ\\
\therefore a \cdot b &= \Vert a \Vert \Vert b \Vert \cos\theta\\
&=10 \times 13 \times \cos(59.5) \\
&=65.9799871849
\end{aligned}
$$
Geometric Meaning
What is the significance of the result above?
We all know that when two vectors point in the same direction, the angle between them is $\theta = 0^\circ$ or $0$ radians. This means that the result of $\cos(\theta)$ is $1$, and the dot product is maximized. If two vectors point in opposite directions, the angle between them is $\theta = 180^\circ$ or $\pi$ radians. This means that the result of $\cos(\theta)$ is $-1$, and the dot product is minimized. When $\theta = 90^\circ$ or $\pi/2$ radians, the result of $\cos(\theta)$ is $0$, and the dot product is $0$. When the dot product is $0$, it means the two vectors are perpendicular or orthogonal to each other.
| Direction Relationship | Calculation Result | |
|---|---|---|
 |
Completely Same Direction | $+AB$ |
 |
Essentially Same Direction | $\lt +AB$ |
 |
Perpendicular | $0$ |
 |
Essentially Opposite Direction | $\gt -AB$ |
|
Completely Opposite Direction | $-AB$ |
Algebraic Perspective
From the coordinate perspective, we only know the coordinates of the vectors and not the angle between them. The algebraic form of the dot product in this perspective doesn't require angles to calculate the dot product. From the coordinate perspective, simply multiply the corresponding components of each vector and then sum them up to get the final result, which is equivalent to the result from the geometric perspective. The algebraic form of the dot product is:
$$
a \cdot b = [a_0b_0+a_1b_1+\dots+a_nb_n] = \sum_{i=1}^na_ib_i
$$
Using the previous example, calculate the dot product of vectors $a$ and $b$ from the coordinate perspective:
$$
a \cdot b = (-6) \times 5+8 \times 12 = -30+96 = 66
$$
The result is very close to the geometric perspective calculation, and the error can be ignored.
Equivalence of the Two Perspectives
To explain the equivalence of the two perspectives, classical textbooks will introduce the concept of standard basis (also known as natural basis or canonical basis). This explanation often involves linear algebra, which can be quite abstract and not intuitive. To make it easier and more straightforward, I'll use a method that even middle school students can understand to demonstrate the equivalence of these two perspectives.
As illustrated in the figure below, for any given vectors $v$ and $u$, pointing from the origin to coordinates $(x_1, y_1)$ and $(x_2, y_2)$, respectively, vector $v$ forms an angle $\alpha$ with the $y$-axis. Similarly, vector $u$ forms an angle $\beta$ with the $x$-axis. The angle between vectors $u$ and $v$ is $\theta$.
Apparently, we have
$$
\begin{aligned}
x_1 &= \Vert v \Vert \sin\alpha \qquad y_1 = \Vert v \Vert \cos\alpha \\
x_2 &= \Vert u \Vert \cos\beta \qquad y_2 = \Vert u \Vert \sin\beta
\end{aligned}\tag{1}
$$
Therefore, the algebraic expression for the dot product of vectors $u$ and $v$ is
$$
u \cdot v = x_1x_2+y_1y_2
$$
By substituting into equation (1), we get
$$
\begin{aligned}
u \cdot v &= x_1x_2+y_1y_2 \\
&= \Vert v \Vert \sin\alpha \times \Vert u \Vert \cos\beta + \Vert v \Vert \cos\alpha \times \Vert u \Vert \sin\beta \\
&= \Vert u \Vert \Vert v \Vert(\sin\alpha \cos\beta+\cos\alpha \sin\beta) \\
&= \Vert u \Vert \Vert v \Vert \sin(\alpha+\beta) \\
&= \Vert u \Vert \Vert v \Vert \sin(\pi/2 - \theta)\\
&= \Vert u \Vert \Vert v \Vert \cos(\theta)
\end{aligned}
$$
Thus, we derived the geometric form from the algebraic form. This demonstrates that the two forms of vector dot product are equivalent.
