Kernel Principal Component Analysis¶

Kernel Principal Component Analysis (kernel PCA) is an extension of principal component analysis (PCA) using techniques of kernel methods. Using a kernel, the originally linear operations of PCA are performed in a reproducing kernel Hilbert space.

This package defines a KernelPCA type to represent a kernel PCA model, and provides a set of methods to access the properties.

Properties¶

Let M be an instance of KernelPCA, d be the dimension of observations, and p be the output dimension (i.e the dimension of the principal subspace)

indim(M)¶: Get the input dimension d, i.e the dimension of the observation space.

outdim(M)¶: Get the output dimension p, i.e the dimension of the principal subspace.

projection(M)¶

Get the projection matrix (of size (n, p)). Each column of the projection matrix corresponds to an eigenvector, and n is a number of observations.

The principal components are arranged in descending order of the corresponding eigenvalues.

principalvars(M)¶: The variances of principal components.

Transformation and Construction¶

The package provides methods to do so:

transform(M, x)¶

Transform observations x into principal components.

Here, x can be either a vector of length d or a matrix where each column is an observation.

reconstruct(M, y)¶

Approximately reconstruct observations from the principal components given in y.

Here, y can be either a vector of length p or a matrix where each column gives the principal components for an observation.

Data Analysis¶

One can use the fit method to perform kernel PCA over a given dataset.

fit(KernelPCA, X; ...)¶

Perform kernel PCA over the data given in a matrix X. Each column of X is an observation.

This method returns an instance of KernelPCA.

Keyword arguments:

Let (d, n) = size(X) be respectively the input dimension and the number of observations:

name	description	default
kernel	The kernel function: This functions accepts two vector arguments `x` and `y`, and returns a scalar value.	`(x,y)->x'y`
solver	The choice of solver: `:eig`: uses `eigfact` `:eigs`: uses `eigs` (always used for sparse data)	`:eig`
maxoutdim	Maximum output dimension.	`min(d, n)`
inverse	Whether to perform calculation for inverse transform for non-precomputed kernels.	`false`
β	Hyperparameter of the ridge regression that learns the inverse transform (when `inverse` is `true`).	`1.0`
tol	Convergence tolerance for `eigs` solver	`0.0`
maxiter	Maximum number of iterations for `eigs` solver	`300`

Kernels¶

List of the commonly used kernels:

function description

(x,y)->x'y Linear

(x,y)->(x'y+c)^d Polynomial

(x,y)->exp(-γ*norm(x-y)^2.0) Radial basis function (RBF)

Example:

using MultivariateStats

# suppose Xtr and Xte are training and testing data matrix,
# with each observation in a column

# train a kernel PCA model
M = fit(KernelPCA, Xtr; maxoutdim=100, inverse=true)

# apply kernel PCA model to testing set
Yte = transform(M, Xte)

# reconstruct testing observations (approximately)
Xr = reconstruct(M, Yte)