第16章 主成分分析

第16章 主成分分析

习题16.1

  对以下样本数据进行主成分分析：

解答：

解答思路：

1. 给出主成分分析算法
2. 自编程基于numpy库实现主成分分析算法

解答步骤：

第1步：主成分分析算法

  主成分分析方法主要有两种，可以通过相关矩阵的特征值分解或样本矩阵的奇异值分解进行。

1. 相关矩阵的特征值分解算法

  根据书中第16.2.2节的相关矩阵的特征值分解算法具体步骤：

（1）对观测数据按照如下公式进行规范化处理，得到规范化数据矩阵，仍以X表示

x_{ij}^* = \frac{x_{ij} - \bar{x}i}{\sqrt{s{ii}}}, \ i=1,2,\cdots,m; \ j = 1,2,\cdots,n

\begin{array}{ll}
\displaystyle \bar{x}i = \frac{1}{n} \sum{j=1}^n x_{ij}, \quad i = 1,2,\cdots,m \
\displaystyle s_{ii} = \frac{1}{n-1} \sum_{j=1}^n (x_{ij} - \bar{x}_i)^2, \quad i = 1,2,\cdots,m
\end{array}

R = [r_{ij}]_{m \times m} = \frac{1}{n-1} X X^T

r_{ij} = \frac{1}{n-1} \sum_{l=1}^n x_{il}x_{lj}, \quad i,j=1,2,\cdots,m

|R - \lambda I| = 0

\lambda_1 \geqslant \lambda_2 \geqslant \cdots \geqslant \lambda_m

a_i = (a_{1i}, a_{2i}, \cdots, a_{mi})^T, \quad i=1,2,\cdots,k

y_i = a_i^T x, \quad i=1,2,\cdots,k

y_{ij} = (a_{1i}, a_{2i}, \cdots, a_{mi})(x_{1j}, x_{2j}, \cdots, x_{mj})^T = \sum_{l=1}^m a_{li} x_{lj} \
i = 1,2,\cdots,m, \quad j=1,2,\cdots, n

\eta_k = \frac{\lambda_k}{\displaystyle \sum_{i=1}^m \lambda_i} \tag{16.30}

\sum_{i=1}^k \eta_i = \frac{\displaystyle \sum_{i=1}^k \lambda_i}{\displaystyle \sum_{i=1}^m \lambda_i} \tag{16.31}

X' = \frac{1}{\sqrt{n - 1}} X^T

X' = U \Sigma V^T

Y = V^T X

y_k = \alpha_k^T x = \alpha_{1k} x_1 + \alpha_{2k} x_2 + \cdots + \alpha_{mk} x_m, \quad k =1,2,\cdots, m

\text{var}(y_k) = \alpha_k^T \Sigma \alpha_k = \lambda_k, \quad k = 1, 2, \cdots, m

\text{cov}(y) = \Lambda = \text{diag}(\lambda_1, \lambda_2, \cdots, \lambda_m)

\bar{x} = \frac{1}{n} \sum_{j=1}^n x_j \tag{16.40}

\begin{array}{l}
S = [s_{ij}]{m \times m} \
\displaystyle s{ij} = \frac{1}{n - 1} \sum_{k=1}^n (x_{ik} - \bar{x}i)(x{jk} - \bar{x}_j), \quad i,j=1,2,\cdots,m
\end{array} \tag{16.41}

\hat{\Sigma} = \frac{1}{n} A^d = \frac{1}{n}\sum_{\alpha=1}^{n-1}z_{\alpha} z'_{\alpha}

E(\hat{\Sigma}) = \frac{1}{n}\sum_{\alpha=1}^{n-1}z_{\alpha} z'_{\alpha} = \frac{n-1}{n} \Sigma

S = \frac{1}{n-1}A

S = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})((x_i - \bar{x}))'

A = \sum_{i=1}^n (x_i - \bar{x})((x_i - \bar{x}))'

E(S) = E(\frac{1}{n-1} A) = \frac{1}{n-1} E(A) = \Sigma

E(A) = (n-1) \Sigma

V(\bar{x}) = \frac{1}{n^2} \sum_{i=1}^n V(x_i) = \frac{1}{n^2} \sum_{i=1}^n \Sigma = \frac{1}{n} \Sigma

\begin{aligned}
E(\hat{\Sigma})
&= \frac{1}{n} E\Big[ \sum_{i=1}^n (x_i - \bar{x}) (x_i - \bar{x})' \Big] \
&= \frac{1}{n} E\Big { \sum_{i=1}^n \big[ (x_i - \mu) - (\bar{x} - \mu) \big ] \big[ (x_i - \mu ) - (\bar{x} - \mu) \big ]' \Big } \
&= \frac{1}{n} E \big [ \sum_{i=1}^n (x_i - \mu)(x_i - \mu)' - n(\bar{x} - u)(\bar{x} - u)' \big ] \
&= \frac{1}{n} \big[ \sum_{i=1}^n V(x_i) - nV(\bar{x}) \big] \
&= \frac{1}{n} \left(n \Sigma - n \cdot \frac{1}{n} \Sigma \right) \
&= \frac{n-1}{n} \Sigma
\end{aligned}

\begin{aligned}
E(A)
&= E \Big[ \sum_{i=1}^n (x_i - \bar{x}) (x_i-\bar{x})' \Big] \
&= n E(\hat{\Sigma}) \
&= n \cdot \frac{n-1}{n} \Sigma \
&= (n-1) \Sigma
\end{aligned}

\begin{array}{cl}
\min \limits_L & |X - L|_F \
\text{s.t.} & \text{rank}(L) \leqslant k
\end{array}

|A|F = \left( \sum{i=1}^m \sum_{j=1}^n (a_{ij})^2 \right)^{\frac{1}{2}}

\big | A - X \big |F = \min{S \in \mathcal{M}} \big | A - S \big |_F

\big | A - X \big |F = (\sigma{k+1}^2 +\sigma_{k+2}^2 + \cdots + \sigma_n^2)^{\frac{1}{2}}

\Sigma' =
\left [ \begin{array}{cccccc}
\sigma_1 & & & & & \
& \ddots & & & 0 & \
& & \sigma_{k} & & & \
& & & 0 & & \
& 0 & & & \ddots & \
& & & & & 0
\end{array} \right] =
\left[ \begin{array}{cc}
\Sigma_k & 0 \
0 & 0
\end{array} \right]

\big | A - A' \big | = (\sigma_{k+1}^2 +\sigma_{k+2}^2 + \cdots + \sigma_n^2)^{\frac{1}{2}} = \min_{S \in \mathcal{M}} \big | A - S \big |_F

X' = \frac{1}{\sqrt{n-1}}X^T

\begin{array}{cl}
\min \limits_L &\displaystyle \left |\frac{1}{\sqrt{n-1}} X^T - \frac{1}{\sqrt{n-1}} L^T \right |_F \
\text{s.t.} & \text{rank}(L) \leqslant k
\end{array}

X' = \frac{1}{\sqrt{n-1}}X^T, L' = \frac{1}{\sqrt{n-1}}L^T

\begin{array}{cl}
\min \limits_L & |X' - L' |_F \
\text{s.t.} & \text{rank}(L) \leqslant k \
& \displaystyle X' = \frac{1}{\sqrt{n-1}}X^T \
& \displaystyle L' = \frac{1}{\sqrt{n-1}}L^T
\end{array}

X' = U' \Sigma' V'^T

A \sim X' \
S \sim L'