Theoretical physics

Methods and approaches

Theoretical physicists do not all work the same way, and it is worth separating out the main approaches rather than treating them as a single undifferentiated activity.

Some theorists focus on phenomenology: building mathematical descriptions that reproduce experimental data and point toward further measurement, often without deriving the model from deeper principles. It is a bit like fitting a curve to data points before you understand what the curve means. The Bohr model of the hydrogen atom and the early analyses of spectral lines by Johann Balmer and Johannes Rydberg are good historical examples. They worked, and they were useful, even though the underlying mechanism was not yet understood.

Others construct model-based frameworks built around specific properties, such as internal consistency or compatibility with known symmetries, and then check whether the resulting theory matches observations. The Standard Model of particle physics was built this way during the 1960s and 1970s, combining quantum field theory with insights about the symmetries that govern the fundamental forces.

A related strategy involves effective theories: frameworks that describe the behaviour of a system accurately within a particular range of energy scales or length scales, without necessarily providing a complete account of what is happening at a deeper level. This is widespread in both particle physics and condensed matter physics, and it is a practical response to the fact that fully general theories are often impossible to solve in realistic settings.

Then there is the ambition of unification: finding frameworks that consolidate previously separate theories into a single coherent one. The electroweak unification of electromagnetism and the weak nuclear force, worked out by Sheldon Glashow, Abdus Salam, and Steven Weinberg in the 1960s, is the most recent fully successful example. The effort to incorporate gravity into a quantum framework has been going on for decades and remains, at the time of writing, unfinished.

Finally, theoretical physics sometimes advances simply because someone notices that a piece of mathematics developed for entirely different reasons happens to describe a physical situation perfectly. This happens more often than one might expect, and it is genuinely strange.

History

Ancient and medieval foundations

The attempt to account for natural phenomena in terms of physical principles, without invoking the supernatural, appears in ancient Greece at least as early as the sixth century BC. What we now call pre-Socratic philosophy was, in important respects, theoretical physics in its earliest form: an effort to identify general principles from which the behaviour of the natural world could be understood. Aristotle later synthesised much of this into a comprehensive account of motion, matter, and the cosmos. He was often wrong, but he was systematic, and his framework remained the dominant reference point in European and Islamic natural philosophy for well over a thousand years.

During the medieval period, the eleventh-century scholar Ibn al-Haytham made advances in optics and articulated, earlier than most accounts acknowledge, the importance of checking theoretical claims against controlled observations. His work on the behaviour of light and the mechanism of vision was influential across Europe for centuries after his death.

The Scientific Revolution

Things began to change rapidly in the sixteenth and seventeenth centuries. Nicolaus Copernicus proposed a heliocentric model of the solar system, which cut against centuries of received opinion. Johannes Kepler, working from Tycho Brahe's careful observational records, derived three precise mathematical laws describing planetary orbits. What is striking about Kepler's laws is not just that they fit the data, but that they gave theorists something concrete to explain. That invitation was taken up by Isaac Newton.

Newton's Philosophiae Naturalis Principia Mathematica of 1687 is arguably the founding document of theoretical physics in the modern sense. From a small number of laws of motion and a universal law of gravitation, Newton derived the orbits of the planets, the behaviour of falling bodies, and the shape of the tides. More than that, he established a template: a compact set of mathematical principles from which a vast range of phenomena could be derived and, crucially, quantitatively predicted.

The nineteenth century

The nineteenth century extended and deepened classical physics along several fronts. Joseph-Louis Lagrange, Leonhard Euler, and William Rowan Hamilton reformulated Newtonian mechanics in increasingly general mathematical terms, producing what are now called Lagrangian mechanics and Hamiltonian mechanics. These frameworks proved essential for the later development of both statistical mechanics and quantum mechanics, so the investment in abstraction paid off very handsomely indeed.

The century's greatest theoretical achievement, in many physicists' view, was Maxwell's electromagnetic theory, which showed that electricity, magnetism, and light are aspects of a single field obeying four coupled differential equations. His equations predicted electromagnetic waves propagating at the speed of light, something that united optics with electromagnetism and, in retrospect, pointed directly toward special relativity. It is hard to overstate how large a conceptual step this was.

Thermodynamics developed alongside, with Rudolf Clausius introducing the concept of entropy and giving a general statement of the second law. The development of statistical mechanics by Maxwell, Ludwig Boltzmann, and Josiah Willard Gibbs then provided a microscopic account of thermodynamic behaviour, connecting the observable bulk properties of matter to the mechanics of individual molecules. This was theoretical physics at its most satisfying: a bridge between two levels of description that had previously seemed entirely separate.

The twentieth century

The early twentieth century brought two upheavals in the foundations of physics, and it is perhaps fair to say that neither has been fully absorbed even now, a hundred years later.

The special theory of relativity, published by Albert Einstein in 1905, resolved the conflict between Newtonian mechanics and electromagnetism by rethinking the concepts of space and time from scratch. The general theory of relativity, which followed in 1915, extended this to include gravity, describing it as the curvature of spacetime caused by the presence of mass and energy. The theory predicted that light would be deflected as it passed near a massive object, a prediction confirmed during the solar eclipse of 1919. It also predicted the existence of black holes and gravitational waves, both subsequently observed, the latter only a century after the prediction was made.

Quantum mechanics was developed during the 1920s, primarily through the work of Werner Heisenberg, Max Born, Pascual Jordan, Erwin Schrödinger, and Paul Dirac. It replaced the classical picture of particles with continuously defined positions and momenta with a framework in which physical quantities take discrete values and are described by probability amplitudes. Quantum mechanics accounts for the structure of atoms and molecules, the specific heat capacity of solids at low temperatures, the photoelectric effect, and an enormous range of other phenomena that classical physics could not explain.

The synthesis of quantum mechanics with special relativity produced quantum field theory, in which particles are understood as excitations of underlying fields. Quantum electrodynamics, developed by Richard Feynman, Julian Schwinger, and Sin-Itiro Tomonaga in the late 1940s, achieved predictions of extraordinary precision. The Standard Model, completed during the 1960s and 1970s, extended this to the strong and weak nuclear forces and remains the most comprehensive account of elementary particles and their interactions yet achieved.

Condensed matter physics saw equally important theoretical advances in the same period. The BCS theory of superconductivity, proposed by John Bardeen, Leon Cooper, and John Robert Schrieffer in 1957, explained the phenomenon in terms of electron pairing mediated by lattice vibrations and won its authors the Nobel Prize in Physics in 1972.

Thought experiments

Thought experiments have a long and respected history in theoretical physics. They are, in essence, arguments conducted entirely in the head: one specifies an idealised physical situation, applies the relevant principles, and sees what follows. They are particularly useful when direct experimentation is unavailable or practically out of reach, though the conclusions they generate must eventually answer to real observation.

Einstein used thought experiments throughout his career. His early reflection on what an observer would perceive while travelling alongside a beam of light helped him identify the fundamental inconsistency between Newtonian mechanics and Maxwell's electromagnetism, and set him on the path toward special relativity. The EPR paradox, put forward by Einstein, Boris Podolsky, and Nathan Rosen in 1935, was a thought experiment designed to argue that quantum mechanics must be incomplete. It provoked John Bell's derivation of inequalities that any locally realistic theory must satisfy, inequalities that subsequent experiments have consistently found quantum mechanics to violate. Schrödinger's famous thought experiment involving a cat placed in a superposition of states was intended to highlight what he saw as an absurdity in the standard interpretation of quantum mechanics. Whatever one makes of that, it has continued to motivate research into the measurement problem for nearly ninety years.

The pattern, roughly speaking, is this: a thought experiment reveals a tension or consequence that would not otherwise be obvious, someone works out the mathematics, and eventually a real experiment is designed to check whether nature agrees.

Relationship to mathematics

Theoretical physics and mathematics have been intertwined since at least the time of Newton, who had to invent the calculus in order to write down his mechanics. The relationship runs in both directions, which is why it is more interesting than a simple story of physics borrowing tools from mathematics.

Joseph Fourier's work on heat conduction in the early nineteenth century led him to develop the theory of Fourier series, a fundamental branch of mathematical analysis, as a direct byproduct of trying to solve a physical problem. The formulation of quantum mechanics in the 1920s drove the rigorous development of functional analysis and the theory of Hilbert spaces.

More recently, research in string theory and related areas has produced substantial new results in algebraic geometry and topology. An example is the discovery of mirror symmetry (string theory), which although nowadays is a major research topic in pure mathematics, originated from string theory compactifications of extra dimensions to Calabi–Yau manifolds. Mirror symmetry drew the attention of mathematicians when it was used to count the number of rational curves on a general quintic threefold. String theoretic tools, including the No-ghost theorem and the mathematics of vertex operator algebras, were also used in the proof of monstrous moonshine.

Perhaps the most philosophically striking aspect of this relationship is how often mathematics developed for purely internal reasons turns out to be exactly what physics needs. Bernhard Riemann's generalisation of geometry, worked out in the 1850s with no physical application in mind, turned out to be precisely the mathematical framework Einstein needed for general relativity half a century later. There was no particular reason it had to work out that way. Eugene Wigner described this pattern, in a celebrated 1960 paper, as "the unreasonable effectiveness of mathematics in the natural sciences," and the phrase has stuck because the phenomenon itself remains genuinely puzzling.

Open problems

Theoretical physics in the early twenty-first century is not short of new problems.

The most structurally significant is perhaps the incompatibility between general relativity and quantum mechanics. Both theories are extremely accurate within their respective domains, but they are mathematically incompatible in their standard formulations. A consistent theory of quantum gravity that reduces to both in the appropriate limits has not been achieved, despite sustained effort over many decades in approaches including string theory and loop quantum gravity.

The Standard Model does not account for dark matter, which is inferred from gravitational observations to make up a large fraction of the mass in galaxies and galaxy clusters, but has not yet been detected through any other means. It also has nothing to say about dark energy, the term given to the apparent acceleration of the expansion of the universe, which remains unexplained. The origin of the matter-antimatter asymmetry in the observable universe, the detailed mechanism of cosmic inflation, and the nature of neutrino mass are among the other significant theoretical questions currently under active investigation.