scikit-learn: A Bug at the `inverse_transform` of the `KernelPCA`

I have a question about the following line of KernelPCA. This line appears in the reprojection of the sample points in KernelPCA but for me, this line seems unnecessary. Here, kernel ridge regression is used to (approximately) reproject the “transformed samples”(, which are coefficients of principal components), using precomputed dual coefficients of the kernel ridge regression. Though we need to add alpha to the diagonal elements of the kernel matrix when computing the dual coefficients (as in _fit_inverse_transform), we do not need to add alpha to the kernel in the prediction stage, as far as I understand.

https://github.com/scikit-learn/scikit-learn/blob/0fb307bf39bbdacd6ed713c00724f8f871d60370/sklearn/decomposition/_kernel_pca.py#L362

Is this line really necessary? If so, I would appreciate it if someone could explain why this line is needed.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 15 (12 by maintainers)

Most upvoted comments

The choice of the kernel for learning the inverse map could be different from the kernel you use in the KPCA! This is what I was missing, thanks! The sklearn implementation makes now sense to me. What was confusing me is that the same kernel is used for two different things: (1) KPCA (2) learning pre-image.

For the predefined kernels you will never encounter any issues with using the same kernel. But I was trying a callable kernel of the form:

 	exp(-1/2(x-y)V^{-1}(x-y))

where V is the covariance matrix of the input data. That gets harcoded and V^{-1}*(x-y) will only work if V^{-1} and (x-y) have matching dimensions. When this kernel gets called for the pre-image learning, as per the implementation, V will still be the covariance matrix wrt to the input data, but the x’s are now objects that live in some space with non-matching dimension. The code breaks.

One way to make this work is to have KernelPCA accept two kernels for these two different purposes. I’ve extended my local code to do that and everything works smoothly.

Quarlos on Sep 22, 2023