Partial Differential Equation Discovery

Resources

Messenger, Daniel A., and David M. Bortz. 2021. “Weak SINDy for Partial Differential Equations.” Journal of Computational Physics 443 (October):110525. https://doi.org/10.1016/J.JCP.2021.110525.
Raissi, Maziar. 2018. “Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations.” Journal of Machine Learning Research 19 (25): 1–24.
Rudy, Samuel H, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. 2016. “Data-Driven Discovery of Partial Differential Equations.” Scientific Advances, 123.
Long, Zichao, Yiping Lu, and Bin Dong. 2019. “PDE-Net 2.0: Learning PDEs from Data with a Numeric-Symbolic Hybrid Deep Network.” Journal of Computational Physics 399 (December):108925. https://doi.org/10.1016/j.jcp.2019.108925.

Main Idea

Within Inverse Problems, PDE Discovery aims to recover a Partial Differential Equation for some system, given data from the system. This is an extension of Parameter Estimation and Neural ODEs. Often, the problem is pitched as discovering $N$ such that $u_{t} = N (x, t, u, u_{x}, u_{x x}, \dots)$ . Note that this places stricter requirements on the function class for $u$ than may be needed with a Weak Form . For instance $u_{t} = (κ u_{x})_{x}$ is more general than $u_{t} = κ_{x} u_{x} + κ u_{x x}$ . This field can also be seen as a specific case of Operator Learning.

SINDy is one popular example of these methods, although there are other approaches based on numeric-symbolic hybrid deep networks (PDE-Net 2.0), and on Neural Networks (Deep Hidden Physics Models). The neural-network based methods may also represent the state as a neural network $u^{θ} (x, t)$ . While SINDy-type methods often use Convex optimization techniques, the neural-network based methods rely on standard machine learning Unconstrained Optimization methods (like Gradient Descent).

Derivations from Constrained Formulation

Consider the following Constrained Optimization problem

\begin{aligned} min_{ϕ} \sum_{i = 1}^{N_{u}} | | u_{i} - u (x_{i}, t_{i}) | |^{2}, \\ s.t. u_{t} = N^{ϕ} (u, u_{x}, u_{x x}), \forall (x, t) \in Ω \times [0, T] . \end{aligned}

Neural ODE Method

Consider a mesh grid of points in space, $x \in R^{n_{x}}$ . Let us loosen the constraints to hold at just these points in space, rather than for the whole domain. Denote $v (t) = u (x, t)$ , and consider finite difference approximations of $u_{x}$ and $u_{x x}$ for $x$ as $D_{1}$ and $D_{2}$ (which also include boundary conditions). Then the optimization problem is

\begin{aligned} min_{ϕ} \sum_{i = 1}^{N_{u}} | | u_{i} - u (x_{i}, t_{i}) | |^{2}, \\ s.t. v_{t} = \underset{f (v)}{\underset{⏟}{N^{ϕ} (v, D_{1} v, D_{2} v)}}, \forall t \in [0, T] . \end{aligned}

Let us consider the constraint separately. Under certain smoothness assumptions and through Fundamental Theorem of Calculus, the constraint is equivalent to

\int_{0}^{t^{'}} v_{t} d t = v (t^{'}) - v (0) = \int_{0}^{t^{'}} f (v) d t, \forall t^{'} \in [0, T] .

Rearranging and trading $t$ and $t^{'}$ , the rewritten constraint reads

v (t) = v (0) + \int_{0}^{t} f^{ϕ} (v) d t, \forall t \in [0, T] .

Here, we assume that this must hold for a finite set of times, and numerically approximate this integration. This is equivalent to taking a numerical solution to the method of lines ODE. Denote this $v^{ϕ} (t)$ :

v^{ϕ} (t) = ODESolve (t, v (0), f^{ϕ}) .

While $v^{ϕ} (t) \approx u (x, t)$ , let us consider an interpolation of $v^{ϕ} (t)$ over $x$ , $u^{ϕ} (x, t)$ . Said differently, let $u^{ϕ} (x, t)$ be the numerical solution (with an interpolant) to the PDE.

Now, by construction the constraints are (numerically / approximately) satisfied and we have the following Unconstrained Optimization problem:

min_{ϕ} \sum_{i = 1}^{N_{u}} | | u_{i} - u^{ϕ} (x_{i}, t_{i}) | |^{2} .

The state $u^{ϕ} (x, t)$ is an (implicit) function of the optimization variable $ϕ$ (i.e. we solve for $u^{ϕ} (x, t)$ in an inner-loop for each update of $ϕ$ ). Thus, this a reduced-space method.

PINNs Method

Consider the main optimization problem, but now loosen the constraints to hold only over a set of collocation points ${(x_{j}, t_{j})}_{j = 1}^{N_{r}}$ , and to within some tolerance $ε$ . Then we have

\begin{aligned} min_{ϕ} \sum_{i = 1}^{N_{u}} | | u_{i} - u (x_{i}, t_{i}) | |^{2}, \\ s.t. | {[u_{t} - N^{ϕ} (u, u_{x}, u_{x x})]}_{(x_{j}, t_{j})} | \leq ε, j = 1, \dots, N_{r} . \end{aligned}

Let us introduce a neural network representation of the state $u^{θ} (x, t)$ , which will we also find during the optimization problem. Through automatic differentiation, we do not need to approximate $u_{x}$ numerically, and we can use the strong form with $u_{t}$ , rather than integrating in time. Thus, the final constrained optimization problem is as follows:

\begin{aligned} min_{θ, ϕ} \sum_{i = 1}^{N_{u}} | | u_{i} - u^{θ} (x_{i}, t_{i}) | |^{2}, \\ s.t. | {[u_{t}^{θ} - N^{ϕ} (u^{θ}, u_{x}^{θ}, u_{x x}^{θ})]}_{(x_{j}, t_{j})} | \leq ε, j = 1, \dots, N_{r} . \end{aligned}

Here, the parameters $θ$ of the state representation $u^{θ} (x, t)$ are updated on each iteration, placing them on equal footing with $ϕ$ . Therefore, this approach is a full-space method.

SINDy

SINDy methods do not fit nicely into this formulation. We approximate $u (x, t)$ in a weird way, and we also try to enforce the PDE constraints. Compared to these two methods, our approximation of $u (x, t)$ (and $u_{x}, u_{x x}, u_{t}$ ) is entirely independent of the PDE we're discovering. We construct $u$ and derivatives through denoising and Numerical Differentiation applied to the data. We add in some Sparsity restriction to the form of $N^{ϕ},$ but this should be fine.

Todo

Is there a way to show that for any approximation of $u (x, t)$ independent of the PDE, there's an adversarial PDE such that SINDy fails? It seems like by choosing a (biased) denoising and numerical differentiation method, some PDE could take advantage of this.