Bernhard Schölkopf

Research

Kernel methods

Schölkopf developed SVM methods achieving world record performance on the MNIST pattern recognition benchmark at the time. With the introduction of kernel PCA, Schölkopf and coauthors argued that SVMs are a special case of a much larger class of methods, and all algorithms that can be expressed in terms of dot products can be generalized to a nonlinear setting by means of what is known as reproducing kernels. Another significant observation was that the data on which the kernel is defined need not be vectorial, as long as the kernel Gram matrix is positive definite. Both insights together led to the foundation of the field of kernel methods, encompassing SVMs and many other algorithms. Kernel methods are now textbook knowledge and one of the major machine learning paradigms in research and applications.

Developing kernel PCA, Schölkopf extended it to extract invariant features and to design invariant kernels and showed how to view other major dimensionality reduction methods such as LLE and Isomap as special cases. In further work with Alex Smola and others, he extended the SVM method to regression and classification with pre-specified sparsity and quantile/support estimation. He proved a representer theorem implying that SVMs, kernel PCA, and most other kernel algorithms, regularized by a norm in a reproducing kernel Hilbert space, have solutions taking the form of kernel expansions on the training data, thus reducing an infinite dimensional optimization problem to a finite dimensional one. He co-developed kernel embeddings of distributions methods to represent probability distributions in Hilbert Spaces, with links to Fraunhofer diffraction as well as applications to independence testing.

Causality

Starting in 2005, Schölkopf turned his attention to causal inference. Causal mechanisms in the world give rise to statistical dependencies as epiphenomena, but only the latter are exploited by popular machine learning algorithms. Knowledge about causal structures and mechanisms is useful by letting us predict not only future data coming from the same source, but also the effect of interventions in a system, and by facilitating transfer of detected regularities to new situations.

Schölkopf and co-workers addressed (and in certain settings solved) the problem of causal discovery for the two-variable setting and connected causality to Kolmogorov complexity.

Around 2010, Schölkopf began to explore how to use causality for machine learning, exploiting assumptions of independence of mechanisms and invariance. His early work on causal learning was exposed to a wider machine learning audience during his Posner lecture at NeurIPS 2011, as well as in a keynote talk at ICML 2017. He assayed how to exploit underlying causal structures in order to make machine learning methods more robust with respect to distribution shifts and systematic errors, the latter leading to the discovery of a number of new exoplanets including K2-18b, which was subsequently found to contain water vapour in its atmosphere, a first for an exoplanet in the habitable zone.