Kernel Principal Component Analysis¶
Kernel Principal Component Analysis (kernel PCA) is an extension of principal component analysis (PCA) using techniques of kernel methods. Using a kernel, the originally linear operations of PCA are performed in a reproducing kernel Hilbert space.
This package defines a KernelPCA
type to represent a kernel PCA model, and provides a set of methods to access the properties.
Properties¶
Let M
be an instance of KernelPCA
, d
be the dimension of observations, and p
be the output dimension (i.e the dimension of the principal subspace)

indim
(M)¶ Get the input dimension
d
, i.e the dimension of the observation space.

outdim
(M)¶ Get the output dimension
p
, i.e the dimension of the principal subspace.

projection
(M)¶ Get the projection matrix (of size
(n, p)
). Each column of the projection matrix corresponds to an eigenvector, andn
is a number of observations.The principal components are arranged in descending order of the corresponding eigenvalues.

principalvars
(M)¶ The variances of principal components.
Transformation and Construction¶
The package provides methods to do so:

transform
(M, x)¶ Transform observations
x
into principal components.Here,
x
can be either a vector of lengthd
or a matrix where each column is an observation.

reconstruct
(M, y)¶ Approximately reconstruct observations from the principal components given in
y
.Here,
y
can be either a vector of lengthp
or a matrix where each column gives the principal components for an observation.
Data Analysis¶
One can use the fit
method to perform kernel PCA over a given dataset.

fit
(KernelPCA, X; ...)¶ Perform kernel PCA over the data given in a matrix
X
. Each column ofX
is an observation.This method returns an instance of
KernelPCA
.Keyword arguments:
Let
(d, n) = size(X)
be respectively the input dimension and the number of observations:name description default kernel The kernel function:
This functions accepts two vector arguments
x
andy
, and returns a scalar value.(x,y)>x'y
solver The choice of solver:
:eig
: useseigfact
:eigs
: useseigs
(always used for sparse data)
:eig
maxoutdim Maximum output dimension. min(d, n)
inverse Whether to perform calculation for inverse transform for nonprecomputed kernels. false
β Hyperparameter of the ridge regression that learns the inverse transform (when inverse
istrue
).1.0
tol Convergence tolerance for eigs
solver0.0
maxiter Maximum number of iterations for eigs
solver300
Kernels¶
List of the commonly used kernels:
function description (x,y)>x'y
Linear (x,y)>(x'y+c)^d
Polynomial (x,y)>exp(γ*norm(xy)^2.0)
Radial basis function (RBF)
Example:
using MultivariateStats
# suppose Xtr and Xte are training and testing data matrix,
# with each observation in a column
# train a kernel PCA model
M = fit(KernelPCA, Xtr; maxoutdim=100, inverse=true)
# apply kernel PCA model to testing set
Yte = transform(M, Xte)
# reconstruct testing observations (approximately)
Xr = reconstruct(M, Yte)