# Kernel Principal Component Analysis¶

Kernel Principal Component Analysis (kernel PCA) is an extension of principal component analysis (PCA) using techniques of kernel methods. Using a kernel, the originally linear operations of PCA are performed in a reproducing kernel Hilbert space.

This package defines a `KernelPCA` type to represent a kernel PCA model, and provides a set of methods to access the properties.

## Properties¶

Let `M` be an instance of `KernelPCA`, `d` be the dimension of observations, and `p` be the output dimension (i.e the dimension of the principal subspace)

`indim`(M)

Get the input dimension `d`, i.e the dimension of the observation space.

`outdim`(M)

Get the output dimension `p`, i.e the dimension of the principal subspace.

`projection`(M)

Get the projection matrix (of size `(n, p)`). Each column of the projection matrix corresponds to an eigenvector, and `n` is a number of observations.

The principal components are arranged in descending order of the corresponding eigenvalues.

`principalvars`(M)

The variances of principal components.

## Transformation and Construction¶

The package provides methods to do so:

`transform`(M, x)

Transform observations `x` into principal components.

Here, `x` can be either a vector of length `d` or a matrix where each column is an observation.

`reconstruct`(M, y)

Approximately reconstruct observations from the principal components given in `y`.

Here, `y` can be either a vector of length `p` or a matrix where each column gives the principal components for an observation.

## Data Analysis¶

One can use the `fit` method to perform kernel PCA over a given dataset.

`fit`(KernelPCA, X; ...)

Perform kernel PCA over the data given in a matrix `X`. Each column of `X` is an observation.

This method returns an instance of `KernelPCA`.

Keyword arguments:

Let `(d, n) = size(X)` be respectively the input dimension and the number of observations:

name description default
kernel

The kernel function:

This functions accepts two vector arguments `x` and `y`, and returns a scalar value.

`(x,y)->x'y`
solver

The choice of solver:

• `:eig`: uses `eigfact`
• `:eigs`: uses `eigs` (always used for sparse data)
`:eig`
maxoutdim Maximum output dimension. `min(d, n)`
inverse Whether to perform calculation for inverse transform for non-precomputed kernels. `false`
β Hyperparameter of the ridge regression that learns the inverse transform (when `inverse` is `true`). `1.0`
tol Convergence tolerance for `eigs` solver `0.0`
maxiter Maximum number of iterations for `eigs` solver `300`

## Kernels¶

List of the commonly used kernels:

function description
`(x,y)->x'y` Linear
`(x,y)->(x'y+c)^d` Polynomial
`(x,y)->exp(-γ*norm(x-y)^2.0)` Radial basis function (RBF)

Example:

```using MultivariateStats

# suppose Xtr and Xte are training and testing data matrix,
# with each observation in a column

# train a kernel PCA model
M = fit(KernelPCA, Xtr; maxoutdim=100, inverse=true)

# apply kernel PCA model to testing set
Yte = transform(M, Xte)

# reconstruct testing observations (approximately)
Xr = reconstruct(M, Yte)
```