Value function

In a problem of optimal control, the value function is defined as the supremum of the objective function taken over the set of admissible controls. Given ${\displaystyle (t_0,x_0)\in [0,t_1]\times {\mathbb{R}}^d}$, a typical optimal control problem is to

${\displaystyle \text{maximize}J(t_0,x_0;u)={\int }_{t_0}^{t_1}I(t,x(t),u(t))\mathrm{d}t+\varphi (x(t_1))}$

subject to

${\displaystyle \frac{\mathrm{d}x(t)}{\mathrm{d}t}=f(t,x(t),u(t))}$

with initial state variable ${\displaystyle x(t_0)=x_0}$. The objective function ${\displaystyle J(t_0,x_0;u)}$ is to be maximized over all admissible controls ${\displaystyle u\in U[t_0,t_1]}$, where ${\displaystyle u}$ is a Lebesgue measurable function from ${\displaystyle [t_0,t_1]}$ to some prescribed arbitrary set in ${\displaystyle {\mathbb{R}}^m}$. The value function is then defined as

${\displaystyle V(t,x(t))=\underset{u\in U}{max}{\int }_t^{t_1}I(\tau ,x(\tau ),u(\tau ))\mathrm{d}\tau +\varphi (x(t_1))}$

with ${\displaystyle V(t_1,x(t_1))=\varphi (x(t_1))}$, where ${\displaystyle \varphi (x(t_1))}$ is the "scrap value". If the optimal pair of control and state trajectories is ${\displaystyle (x^{\ast },u^{\ast })}$, then ${\displaystyle V(t_0,x_0)=J(t_0,x_0;u^{\ast })}$. The function ${\displaystyle h}$ that gives the optimal control ${\displaystyle u^{\ast }}$ based on the current state ${\displaystyle x}$ is called a feedback control policy, or simply a policy function.

Bellman's principle of optimality roughly states that any optimal policy at time ${\displaystyle t}$, ${\displaystyle t_0\le t\le t_1}$ taking the current state ${\displaystyle x(t)}$ as "new" initial condition must be optimal for the remaining problem. If the value function happens to be continuously differentiable, this gives rise to an important partial differential equation known as Hamilton–Jacobi–Bellman equation,

${\displaystyle -\frac{\mathrm{\partial }V(t,x)}{\mathrm{\partial }t}=\underset{u}{max}\left\{I(t,x,u)+\frac{\mathrm{\partial }V(t,x)}{\mathrm{\partial }x}f(t,x,u)\right\}}$

where the maximand on the right-hand side can also be re-written as the Hamiltonian, ${\displaystyle H\left(t,x,u,\lambda \right)=I(t,x,u)+\lambda (t)f(t,x,u)}$, as

${\displaystyle -\frac{\mathrm{\partial }V(t,x)}{\mathrm{\partial }t}=\underset{u}{max}H(t,x,u,\lambda )}$

with ${\displaystyle \mathrm{\partial }V(t,x)/\mathrm{\partial }x=\lambda (t)}$ playing the role of the costate variables. Given this definition, we further have ${\displaystyle \mathrm{d}\lambda (t)/\mathrm{d}t={\mathrm{\partial }}^{2}V(t,x)/\mathrm{\partial }x\mathrm{\partial }t+{\mathrm{\partial }}^{2}V(t,x)/\mathrm{\partial }{x}^2\cdot f(x)}$, and after differentiating both sides of the HJB equation with respect to ${\displaystyle x}$,

${\displaystyle -\frac{{\mathrm{\partial }}^{2}V(t,x)}{\mathrm{\partial }t\mathrm{\partial }x}=\frac{\mathrm{\partial }I}{\mathrm{\partial }x}+\frac{{\mathrm{\partial }}^{2}V(t,x)}{\mathrm{\partial }{x}^2}f(x)+\frac{\mathrm{\partial }V(t,x)}{\mathrm{\partial }x}\frac{\mathrm{\partial }f(x)}{\mathrm{\partial }x}}$

which after replacing the appropriate terms recovers the costate equation

${\displaystyle -\dot{\lambda }(t)=\underset{=\frac{\mathrm{\partial }H}{\mathrm{\partial }x}}{\underbrace{\frac{\mathrm{\partial }I}{\mathrm{\partial }x}+\lambda (t)\frac{\mathrm{\partial }f(x)}{\mathrm{\partial }x}}}}$

where ${\displaystyle \dot{\lambda }(t)}$ is Newton notation for the derivative with respect to time.

The value function is the unique viscosity solution to the Hamilton–Jacobi–Bellman equation. In an online closed-loop approximate optimal control, the value function is also a Lyapunov function that establishes global asymptotic stability of the closed-loop system.