Fight4354

The learning process over the past few days has gradually clarified my understanding of linear algebra. In this post, I will summarize some key new content.

When dealing with matrices and singular values, the following understanding should be established:
✅ A matrix is a spatial operator.
✅ Singular value decomposition helps you break down the essence of a matrix: rotation → stretching → rotation.
✅ The size ordering of singular values tells you: in which directions the matrix truly has strength, and which directions are ineffective.

1. Orthogonal Matrix#

The core definition of an Orthogonal Matrix

An $n \times n$ real matrix $Q$ is called an orthogonal matrix if it satisfies

Copy$$Q^{\mathsf T}\,Q \;=\; Q\,Q^{\mathsf T}\;=\;I_n$$

Here, $Q^{\mathsf T}$ is the transpose, and $I_n$ is the $n$-dimensional identity matrix.

The inverse is the transpose

This simplifies calculations and ensures numerical stability.
An orthogonal matrix is a real matrix that "preserves inner products"—it rotates or flips the coordinate system but never stretches or distorts it.

2. Angle Brackets#

Here

is the symbol for the "inner product." In the most common case—real vector space $\mathbb R^n$—it is equivalent to the dot product we are familiar with:

3. Matrix Position Exchange#

1. Eliminate the left $Q$

• The $Q$ attached to the left of $S$

• Multiply both sides by its inverse $Q^{-1}$ from the left:

Note:

• You must multiply from the left consistently on both sides;

• Do not attempt to multiply $Q^{-1}$ to the right (that would disrupt the order of multiplication).

2. Eliminate the right $Q^{\mathsf T}$

• The $Q^{\mathsf T}$ attached to the right of $S$

• Multiply both sides by its inverse $(Q^{\mathsf T})^{-1}$ from the right:

If $Q$ is an orthogonal matrix, $Q^{-1}=Q^{\mathsf T}$, then

Why can't the order be reversed?

• Once you multiply incorrectly, the symbols will "insert" themselves in different positions:
  $AQ^{-1} \neq Q^{-1}A$.

• The same operation must be performed symmetrically on both sides of the equation for the equality to hold.

• This is essentially the same as the order of function composition or coordinate transformation: the order in which transformations are performed must be written in the corresponding positions of the product, and cannot be arbitrarily swapped.

4. Similar Diagonalizable Matrix#

A similar diagonalizable matrix (commonly referred to as a "diagonalizable matrix") means:

There exists an invertible matrix $P$ such that

Copy$$P^{-1}AP = D,$$

where $D$ is a diagonal matrix.
At this point, we say that $A$ can be diagonalized through a similarity transformation, or simply that $A$ is diagonalizable.

The "mechanical process" of diagonalization

1. Find the eigenvalues: Solve $\det(A-\lambda I)=0$.

2. Find the eigenvectors: For each $\lambda$, solve $(A-\lambda I)x=0$.

3. Assemble $P$: Arrange the $n$ linearly independent eigenvectors as columns in matrix $P$.

4. Obtain $D$: Fill in the corresponding eigenvalues along the diagonal: $D=\operatorname{diag}(\lambda_1,\dots,\lambda_n)$.
   Thus, $A = P D P^{-1}$.

5. Singular Value Decomposition#

The expression $A=Q S Q^{\mathsf T}$ is called orthogonal similarity diagonalization; geometrically, it means "rotate (or mirror) the coordinate system → A only retains independent stretching."

1. Why can "real symmetric matrices always be orthogonally diagonalized"?

Spectral Theorem:

For any real symmetric matrix $A$, there exists an orthogonal matrix $Q$ such that $Q^{\mathsf T} A Q$ is diagonal, and the diagonal elements are the eigenvalues of $A$.

• Real eigenvalues: Symmetry ensures that all eigenvalues are real numbers.
• Orthogonal eigenvectors: If $\lambda_i \ne \lambda_j$, the corresponding eigenvectors must be orthogonal.
• Repeated roots can also take orthogonal bases: The same eigenvalue may correspond to multiple vectors; in this case, perform Gram–Schmidt in the subspace they span.

2. Step-by-step textual analysis

• If $\lambda$ is repeated (geometric multiplicity >1), first take a random set of linearly independent vectors, then perform Gram-Schmidt in that subspace to make them orthogonal and normalize (adjust length to 1). |
  | 4. Arrange the modified eigenvectors horizontally according to the order of eigenvalues on the diagonal to obtain the orthogonal matrix $Q$ | Arrange the modified eigenvectors as columns in $Q=[q_1\ \cdots\ q_n]$ according to the order of the diagonal eigenvalues. At this point, $Q^{\mathsf T}Q=I$, and $A = Q S Q^{\mathsf T}$. |

3. A specific small example of $2 \times 2$

Let

① Find the eigenvalues

② Find the eigenvectors

• $\lambda_1=5$: $(A-5I)v=0 \ \Rightarrow\ v_1=\begin{bmatrix}1\\1\end{bmatrix}$
• $\lambda_2=3$: $(A-3I)v=0 \ \Rightarrow\ v_2=\begin{bmatrix}1\\-1\end{bmatrix}$

③ Normalize

④ Assemble and verify

6. Determinant (det · )#

The determinant is an operation that maps a square matrix $A$ to a scalar $\det A$.
This scalar encapsulates the most essential geometric and algebraic information of the matrix: volume scaling factor, invertibility, product of eigenvalues, etc.

Formula

3×3 Hand Calculation Example

Let

Expand along the first row:

In summary

"Taking the determinant" means: collapsing an $n\times n$ square matrix into a single number through a set of alternating, linear rules, and this number encodes key information about the matrix's volume scaling, direction, invertibility, and product of eigenvalues.

7. Rank of a Matrix#

What exactly is the "rank" of a matrix?

Linear Independence
Below are three comparative cases using a 3 × 3 small matrix to make the statement "rank = how many linearly independent column (or row) vectors can be selected" clear.

| Matrix $A$ | Column Vectors Written as $\bigl(v_1\,|\,v_2\,|\,v_3\bigr)$ | Linear Relationship | Rank |
|------------|-----------------------------------------------|-----------|--------|
| $\displaystyle\begin{bmatrix}1&2&3\\2&4&6\\3&6&9\end{bmatrix}$ | $v_1=\begin{bmatrix}1\\2\\3\end{bmatrix}$
$v_2=\begin{bmatrix}2\\4\\6\end{bmatrix}=2v_1$
$v_3=\begin{bmatrix}3\\6\\9\end{bmatrix}=3v_1$ | All three columns lie on the same line—only 1 independent vector | 1 |
| $\displaystyle\begin{bmatrix}1&0&1\\0&1&1\\1&1&2\end{bmatrix}$ | $v_1=\begin{bmatrix}1\\0\\1\end{bmatrix}$
$v_2=\begin{bmatrix}0\\1\\1\end{bmatrix}$
$v_3=v_1+v_2$ | $v_1, v_2$ are not collinear ⇒ 2-dimensional plane; $v_3$ lies in this plane | 2 |
| $\displaystyle\begin{bmatrix}1&0&1\\0&1&1\\1&1&0\end{bmatrix}$ | $v_1=\begin{bmatrix}1\\0\\1\end{bmatrix}$
$v_2=\begin{bmatrix}0\\1\\1\end{bmatrix}$
$v_3=\begin{bmatrix}1\\1\\0\end{bmatrix}$ | Any two columns cannot linearly express the third column ⇒ All three columns span the entire $\mathbb R^{3}$ | 3 |

How to determine "independence"?

• Manual Calculation Combine the columns into a matrix and perform elimination → The number of non-zero rows is the rank.

• Concept If there exist constants $c_1,c_2,c_3$ such that $c_1v_1+c_2v_2+c_3v_3=0$ and not all are 0, the vectors are dependent; otherwise, they are independent.

  • Case 1: $2v_1-v_2=0$ → dependent

  • Case 2: Only $v_3=v_1+v_2$ is dependent, while $v_1,v_2$ are independent

  • Case 3: Any non-trivial combination ≠ 0 → All three vectors are independent

In summary: Rank = how much independent information (dimension) this matrix can truly "hold."

8. Low-Rank Approximation#

Why does truncating SVD (low-rank approximation) only require storing k (m+n)+k numbers?

When the original matrix

is truncated to rank $k$, it can be written as

Adding the three blocks together gives

• $U_k$ and $V_k$: each has $k$ columns, with each column storing a vector of length equal to the number of rows
  $\Rightarrow mk + nk$ numbers.

• $\Sigma_k$: is a diagonal matrix, only requiring those $k$ diagonal elements—not $k^2$.

Therefore, using rank-$k$ SVD approximation instead of the original $m\times n$ storage, the parameter count reduces from $mn$ to $k(m+n)+k$.
If $k \ll \min(m,n)$, the saved space becomes quite considerable.

Lowering rank = reducing information dimension, low-rank storage = reducing parameter count/memory simultaneously

9. Norms#

The "double vertical bars" $|,\cdot,|$ in linear algebra represent norms.

• For a vector $v\in\mathbb R^m$, the most commonly used is the Euclidean norm: $\|v\|=\sqrt{v^{\mathsf T}v}=\sqrt{\sum_{i=1}^{m}v_i^{2}},\qquad \|v\|^{2}=v^{\mathsf T}v.$
  In the figure, $\|Xw-y\|^{2}$ represents the sum of the squares of each component of the vector $Xw-y$.

• For a matrix $A$, if also written as $\|A\|$, it commonly refers to the Frobenius norm: $\|A\|_F=\sqrt{\sum_{i,j}A_{ij}^{2}}$. However, the figures here involve vectors.

In contrast, the single vertical bar $|\,\cdot\,|$ typically represents absolute value (scalar) or determinant $|A|$. Thus, double vertical bars denote the "length" of vectors/matrices, while single vertical bars denote the magnitude of scalars or determinants—different objects and meanings.

Common Euclidean Distance for Vectors -- 2-Norm (L2 Norm)#

import torch

b = torch.tensor([3.0, 4.0])
print(b.norm())  # Outputs 5.0

.norm() is a method of PyTorch tensors (torch.Tensor).

Common Frobenius Norm for Matrices#

Matrices also have "length"—the commonly used is the Frobenius norm

1. Why can it also be written as "matrix dot product"

The commonly used inner product in matrix space is

where $\operatorname{tr}(\cdot)$ is the trace operation (sum of diagonal elements).
Taking the inner product of $A$ with itself gives:

Thus:

  

Copy$$\boxed{\;\|A\|_F^{2}= \operatorname{tr}(A^{\mathsf T}A)\;}$$

This is the matrix version of $\|v\|^{2}=v^{\mathsf T}v$—just replacing vector dot product with "trace dot product."
The Frobenius norm is indeed equal to the square root of the sum of the squares of all singular values, that is:

Here:

• $| A |_F$ is the Frobenius norm of matrix $A$

• $\sigma_i$ are the singular values of $A$

The Frobenius norm is indeed equal to the square root of the sum of the squares of all singular values
Expanded explanation:

The Frobenius norm is defined as:

But singular value decomposition (SVD) tells us:

where $\Sigma$ is a diagonal matrix, with singular values $\sigma_1, \sigma_2, \dots$ on the main diagonal.

Since the Frobenius norm is invariant (unitary orthogonal transformations do not change the norm), we can directly compute:

Thus, ultimately:

Beware of misconceptions

Note:
✅ Not the square root of a single singular value, nor the maximum singular value
✅ Is the sum of the squares of all singular values taken square root

The spectral norm looks at "the direction that stretches the most," while the Frobenius norm sums all energies.

Spectral Norm of a Matrix#

✅ Definition of Spectral Norm

The spectral norm of a matrix $A$ is defined as:

In simple terms, it is the maximum value to which matrix $A$ stretches a unit vector.
Singular values inherently represent the stretching transformations of the matrix.

It equals the maximum singular value of $A$:

From another perspective: the spectral norm ≈ the maximum length to which a unit vector is stretched after being input into the matrix.
✅ Its relationship with the Frobenius norm

• Frobenius Norm → Looks at overall energy (sum of squares of matrix elements)

• Spectral Norm → Looks at the maximum stretching amount in a single direction

In other words:

• Frobenius is like the total "volume" of the matrix

• Spectral norm is like the "most extreme" stretching rate in a single direction

✅ Example: Why is it important?

Imagine a linear layer $W$ in a neural network:

• If $\| W \|_2$ is very large, even small perturbations in the input will be amplified,
  making the network prone to overfitting and sensitive to noise.

• If $\| W \|_2$ is moderate, the output changes will be stable against input perturbations,
  leading to better generalization.

Thus, modern methods (like spectral normalization)
will directly constrain the spectral norm of $W$ within a certain range during training.

⚠ Directly stating the drawbacks

The spectral norm is powerful, but:

• It only focuses on a single maximum direction, ignoring other directions of stretching;

• It is more complex to compute than the Frobenius norm (requires singular value decomposition, rather than simple element-wise square sums).

Summary Comparison

1. Same idea, different dimensions

• The Euclidean norm is the square root of a vector's dot product with itself.

• The Frobenius norm treats all matrix elements as a long vector and does the same; in matrix language, it can be expressed as

  Copy$$\|A\|_F=\sqrt{\operatorname{tr}(A^{\mathsf T}A)}.$$

  This is "transpose → multiply → take trace."

2. When to use which?

3. Intuitive differences

• Euclidean: Measures the length in a single direction;

• Frobenius: Measures the total sum of each element's energy, thus for matrices, no particular column or row is special; all elements are treated equally.

One-sentence memory:

Euclidean Norm: The "ruler" for vectors.
Frobenius Norm: Measures the overall size of a matrix as if "flattened" with the same ruler.

10. Transpose of Matrix Multiplication#

In matrix algebra, there is a fixed "reversal order" rule for the transpose of the product of two (or more) matrices:

This means transpose each matrix first, then reverse the order of multiplication.
This property holds for any dimension-matching real (or complex) matrices and can be recursively extended:

xLog Editing Markdown Document Notes

• Ensure all mathematical expressions are enclosed in $$ … $$

• If there are single $n \times n$ types, change to n × n or $$n\\times n$$

Reference video:

Click here to watch the Bilibili video