Physics-informed neural networks

Modeling and computation

A general nonlinear partial differential equation can be written as:

${\displaystyle u_t+\mathcal{N}[u;\lambda ]=0,x\in \mathrm{\Omega },t\in [0,T]}$

where ${\displaystyle u(t,x)}$ denotes the solution, ${\displaystyle \mathcal{N}[\cdot ;\lambda ]}$ is a nonlinear operator parameterized by ${\displaystyle \lambda }$, and ${\displaystyle \mathrm{\Omega }}$ is a subset of ${\displaystyle {\mathbb{R}}^D}$. This general form of governing equations summarizes a wide range of problems in mathematical physics, such as conservative laws, diffusion process, advection-diffusion systems, and kinetic equations. Given noisy measurements of a generic dynamic system described by the equation above, PINNs can be designed to solve two classes of problems:

• data-driven solutions of partial differential equations
• data-driven discovery of partial differential equations

Data-driven solution of partial differential equations

The data-driven solution of PDE computes the hidden state ${\displaystyle u(t,x)}$ of the system given boundary data and/or measurements ${\displaystyle z}$, and fixed model parameters ${\displaystyle \lambda }$. We solve:

${\displaystyle u_t+\mathcal{N}[u]=0,x\in \mathrm{\Omega },t\in [0,T]}$.

by defining the residual ${\displaystyle f(t,x)}$ as:

${\displaystyle f:=u_t+\mathcal{N}[u]}$,

and approximating ${\displaystyle u(t,x)}$ by a deep neural network. This network can be differentiated using automatic differentiation. The parameters of ${\displaystyle u(t,x)}$ and ${\displaystyle f(t,x)}$ can be then learned by minimizing the following loss function ${\displaystyle L_{\text{tot}}}$:

${\displaystyle L_{\text{tot}}=L_u+L_f}$

where:

• ${\displaystyle L_u=\Vert u-z{\Vert }_{\mathrm{\Gamma }}}$ is the error between the PINN ${\displaystyle u(t,x)}$ and the set of boundary conditions and measured data on the set of points ${\displaystyle \mathrm{\Gamma }}$ where the boundary conditions and data are defined.
• ${\displaystyle L_f=\Vert f{\Vert }_{\mathrm{\Gamma }}}$ is the mean-squared error of the residual function. This second term encourages the PINN to learn the structural information expressed by the PDE during the training process.

This approach has been used to yield computationally efficient physics-informed surrogate models with applications in the forecasting of physical processes, model predictive control, multi-physics and multi-scale modeling, and simulation. It has been shown to converge to the solution of the PDE.

Data-driven discovery of partial differential equations

Given noisy and incomplete measurements ${\displaystyle z}$ of the state of the system, the data-driven discovery of PDEs results in computing the unknown state ${\displaystyle u(t,x)}$ and learning model parameters ${\displaystyle \lambda }$ that best describe the observed data:

${\displaystyle u_t+\mathcal{N}[u;\lambda ]=0,x\in \mathrm{\Omega },t\in [0,T]}$

By defining ${\displaystyle f(t,x)}$ as:

${\displaystyle f:=u_t+\mathcal{N}[u;\lambda ]=0}$,

and approximating ${\displaystyle u(t,x)}$ by a deep neural network, ${\displaystyle f(t,x)}$ results in a PINN. This network can be derived using automatic differentiation. The parameters of ${\displaystyle u(t,x)}$ and ${\displaystyle f(t,x)}$, together with the parameter ${\displaystyle \lambda }$ of the differential operator can be then learned by minimizing the following loss function ${\displaystyle L_{\text{tot}}}$:

${\displaystyle L_{\text{tot}}=L_u+L_f}$

where:

• ${\displaystyle L_u=\Vert u-z{\Vert }_{\mathrm{\Gamma }}}$, with ${\displaystyle u}$ and ${\displaystyle z}$ state solutions and measurements at sparse location ${\displaystyle \mathrm{\Gamma }}$, respectively.
• ${\displaystyle L_f=\Vert f{\Vert }_{\mathrm{\Gamma }}}$ is the residual function. This second term requires the structured information represented by the partial differential equations to be satisfied in the training process.

This strategy allows for discovering dynamic models described by nonlinear PDEs assembling computationally efficient and fully differentiable surrogate models that may find application in predictive forecasting, control, and data assimilation.

Extensions and applications

For piece-wise function approximation

PINNs are unable to approximate PDEs that have strong non-linearity or sharp gradients (such as those that commonly occur in practical fluid flow problems). Piecewise approximation has been an old practice in the field of numerical approximation. With the capability of approximating strong non-linearity, extremely light-weight PINNs are used to solve PDEs in much larger discrete subdomains that increases accuracy substantially and decreases computational load as well. DPINNs (Distributed Physics-Informed Neural Networks) and DPIELMs (Distributed Physics-Informed Extreme Learning Machines) use generalizable space-time domain discretization for better approximation. DPIELM is an extremely fast and lightweight approximator with competitive accuracy. Domain scaling on the top has a special effect. Another school of thought is discretization for parallel computation to leverage usage of available computational resources.

XPINNs are a generalized space-time domain decomposition approach for the physics-informed neural networks (PINNs) to solve nonlinear partial differential equations on arbitrary complex-geometry domains. The XPINNs further push the boundaries of both PINNs and conservative PINNs (cPINNs), which are a spatial domain decomposition approach in the PINN framework tailored to conservation laws. Compared to PINNs, XPINNs have large representation and parallelization capacity due to the inherent property of deployment of multiple neural networks in the smaller subdomains. Unlike cPINNs, XPINNs can be extended to any type of PDEs. Moreover, the domain can be decomposed in any arbitrary way (in space and time), which is not possible with cPINNs. Thus, XPINNs offer both space and time parallelization, thereby reducing the training cost more effectively. The XPINN is particularly effective for the large-scale problems (involving large data sets) as well as for the high-dimensional problems where single network-based PINNs are not adequate. The rigorous bounds on the errors resulting from the approximation of the nonlinear PDEs (incompressible Navier–Stokes equations) with PINNs and XPINNs have been proven. However, DPINN debunks the use of residual (flux) matching at the domain interfaces as they hardly seem to improve the optimization.

Theory of functional connections

In the PINN framework, initial and boundary conditions are not analytically satisfied, thus they need to be included in the loss function of the network to be simultaneously learned with the differential equation (DE) unknown functions. Having competing objectives during the network's training can lead to unbalanced gradients while using gradient-based techniques, which causes PINNs to often struggle to accurately learn the underlying DE solution. This drawback is overcome by using functional interpolation techniques such as the theory of functional connections (TFC)'s constrained expression, in the Deep-TFC framework, which reduces the solution search space of constrained problems to the subspace of neural network that analytically satisfies the constraints. A further improvement of PINN and functional interpolation approach is given by the extreme theory of functional connections (X-TFC) framework, where a single-layer Neural Network and the extreme learning machine training algorithm are employed. X-TFC allows to improve the accuracy and performance of regular PINNs, and its robustness and reliability are proved for stiff problems, optimal control, aerospace, and rarefied gas dynamics applications.

Physics-informed PointNet (PIPN) for multiple sets of irregular geometries

Regular PINNs are only able to obtain the solution of a forward or inverse problem on a single geometry. It means that for any new geometry (computational domain), one must retrain a PINN. This limitation of regular PINNs imposes high computational costs, specifically for a comprehensive investigation of geometric parameters in industrial designs. Physics-informed PointNet (PIPN) is fundamentally the result of a combination of PINN's loss function with PointNet. In fact, instead of using a simple fully connected neural network, PIPN uses PointNet as the core of its neural network. PointNet has been primarily designed for deep learning of 3D object classification and segmentation by the research group of Leonidas J. Guibas. PointNet extracts geometric features of input computational domains in PIPN. Thus, PIPN is able to solve governing equations on multiple computational domains (rather than only a single domain) with irregular geometries, simultaneously. The effectiveness of PIPN has been shown for incompressible flow, heat transfer and linear elasticity.

Semi-Lagrangian formulation of PINNs

Semi-Lagrangian Physics-Informed Neural Networks (SL-PINNs) are a class of PINNs designed for hyperbolic and transport-dominated partial differential equations such as Vlasov-Poisson . Instead of directly approximating the solution field, SL-PINNs learn an inverse characteristic flow map that transports phase-space coordinates backward to their initial locations, allowing the solution to be reconstructed from the initial condition. The approach combines a Lagrangian representation of transport dynamics with an Eulerian treatment of field equations and may incorporate volume-preserving regularization to improve conservation of physical quantities such as mass, momentum, and energy. The method has been applied to kinetic systems including the Vlasov–Poisson equations and has been reported to improve long-time stability and reduce numerical diffusion relative to conventional PINN formulations .

For inverse computations

Physics-informed neural networks (PINNs) have proven particularly effective in solving inverse problems within differential equations, demonstrating their applicability across science, engineering, and economics. They have shown to be useful for solving inverse problems in a variety of fields, including nano-optics, topology optimization/characterization, multiphase flow in porous media, and high-speed fluid flow. PINNs have demonstrated flexibility when dealing with noisy and uncertain observation datasets. They also demonstrated clear advantages in the inverse calculation of parameters for multi-fidelity datasets, meaning datasets with different quality, quantity, and types of observations. Uncertainties in calculations can be evaluated using ensemble-based or Bayesian-based calculations.

PINNs can also be used in connection with symbolic regression for discovering the mathematical expression in connection with discovery of parameters and functions. One example of such application is the study on chemical ageing of cellulose insulation material, in this example PINNs are used to first discover a parameter for a set of ordinary differential equations (ODEs) and later a function solution, which is later used to find a more fitting expression using a symbolic regression with a combination of operators.

For elasticity problems

Ensemble of physics-informed neural networks is applied to solve plane elasticity problems. Surrogate networks are intended for the unknown functions, namely, the components of the strain and the stress tensors as well as the unknown displacement field, respectively. The residual network provides the residuals of the partial differential equations (PDEs) and of the boundary conditions. The computational approach is based on principles of artificial intelligence. This approach can be extended to nonlinear elasticity problems, where the constitutive equations are nonlinear. PINNs can also be used for Kirchhoff plate bending problems with transverse distributed loads and to contact models with elastic Winkler's foundations. A comparison of PINNs with classical approximation methods in mechanics, especially the Least Squares Finite Element Method, is outlined in .

For shunted piezoelectric materials

A multi-physics-informed neural network (multi-PINN) is used to effectively solve static and dynamic electromechanical vibration suppression problems with a shunted circuit. Electromechanical problems, such as those involving a cantilever piezoelectric beam, can be converted into the appropriate PINN formulation. A time partitioning approach reduces the computational cost of the dynamic problem. The resulting system of ordinary differential equations (ODEs) is resolved using a multi-PINN to accurately predict the dynamic response to principal mode resonance excitation.

With backward stochastic differential equation

Deep backward stochastic differential equation method is a numerical method that combines deep learning with backward stochastic differential equation (BSDE) to solve high-dimensional problems in financial mathematics. By leveraging the powerful function approximation capabilities of deep neural networks, deep BSDE addresses the computational challenges faced by traditional numerical methods like finite difference methods or Monte Carlo simulations, which struggle with the curse of dimensionality. Deep BSDE methods use neural networks to approximate solutions of high-dimensional partial differential equations (PDEs), effectively reducing the computational burden. Additionally, integrating Physics-informed neural networks (PINNs) into the deep BSDE framework enhances its capability by embedding the underlying physical laws into the neural network architecture, ensuring solutions adhere to governing stochastic differential equations, resulting in more accurate and reliable solutions.

For biology

An extension or adaptation of PINNs are Biologically-informed neural networks (BINNs). BINNs introduce two key adaptations to the typical PINN framework: (i) the mechanistic terms of the governing PDE are replaced by neural networks, and (ii) the loss function ${\displaystyle L_{tot}}$ is modified to include ${\displaystyle L_{constr}}$, a term used to incorporate domain-specific knowledge that helps enforce biological applicability. For (i), this adaptation has the advantage of relaxing the need to specify the governing differential equation a priori, either explicitly or by using a library of candidate terms. Additionally, this approach circumvents the potential issue of misspecifying regularization terms in stricter theory-informed cases.

A natural example of BINNs can be found in cell dynamics, where the cell density ${\displaystyle u(x,t)}$ is governed by a reaction-diffusion equation with diffusion and growth functions ${\displaystyle D(u)}$ and ${\displaystyle G(u)}$, respectively:

${\displaystyle u_t=\mathrm{\nabla }\cdot [D(u)\mathrm{\nabla }u]+G(u)u,x\in \mathrm{\Omega },t\in [0,T]}$

In this case, a component of ${\displaystyle L_{constr}}$ could be ${\displaystyle ||D|{|}_{\mathrm{\Gamma }}}$ for ${\displaystyle D<D_{min},D>D_{max}}$, which penalizes values of ${\displaystyle D}$ that fall outside a biologically relevant diffusion range defined by ${\displaystyle D_{min}\le D\le D_{max}}$. Furthermore, the BINN architecture, when utilizing multilayer-perceptrons (MLPs), would function as follows: an MLP is used to construct ${\displaystyle u_{MLP}(x,t)}$ from model inputs ${\displaystyle (x,t)}$, serving as a surrogate model for the cell density ${\displaystyle u(x,t)}$. This surrogate is then fed into the two additional MLPs, ${\displaystyle D_{MLP}(u_{MLP})}$ and ${\displaystyle G_{MLP}(u_{MLP})}$, which model the diffusion and growth functions. Automatic differentiation can then be applied to compute the necessary derivatives of ${\displaystyle u_{MLP}}$,${\displaystyle D_{MLP}}$ and ${\displaystyle G_{MLP}}$ to form the governing reaction-diffusion equation.

Note that since ${\displaystyle u_{MLP}}$ is a surrogate for the cell density, it may contain errors, particularly in regions where the PDE is not fully satisfied. Therefore, the reaction-diffusion equation may be solved numerically, for instance using a method-of-lines approach.

Limitations

Translation and discontinuous behavior are hard to approximate using PINNs. They fail when solving differential equations with slight advective dominance and hence asymptotic behaviour causes the method to fail. Such PDEs could be solved by scaling variables. This difficulty in training of PINNs in advection-dominated PDEs can be explained by the Kolmogorov n–width of the solution. They also fail to solve a system of dynamical systems and hence have not been a success in solving chaotic equations. One of the reasons behind the failure of regular PINNs is soft-constraining of Dirichlet and Neumann boundary conditions which pose a multi-objective optimization problem which requires manually weighing the loss terms to be able to optimize. More generally, posing the solution of a PDE as an optimization problem brings with it all the problems that are faced in the world of optimization, the major one being getting stuck in local optima. Specifically, it was shown that some of these local optima are connected to unstable solutions of a given PDE.