On the Convergence of Physics Informed Neural Networks for Linear Second-Order Elliptic and Parabolic Type PDEs

Physics informed neural networks (PINNs) are deep learning based techniques for solving partial differential equations (PDEs) encountered in computational science and engineering. Guided by data and physical laws, PINNs find a neural network that approximates the solution to a system of PDEs. Such a neural network is obtained by minimizing a loss function in which any prior knowledge of PDEs and data are encoded. Despite its remarkable empirical success in one, two or three dimensional problems, there is little theoretical justification for PINNs.As the number of data grows, PINNs generate a sequence of minimizers which correspond to a sequence of neural networks. We want to answer the question: Does the sequence of minimizers converge to the solution to the PDE? We consider two classes of PDEs: linear second-order elliptic and parabolic. By adapting the Schauder approach and the maximum principle, we show that the sequence of minimizers strongly converges to the PDE solution in $C^0$. Furthermore, we show that if each minimizer satisfies the initial/boundary conditions, the convergence mode becomes $H^1$. Computational examples are provided to illustrate our theoretical findings. To the best of our knowledge, this is the first theoretical work that shows the consistency of PINNs.     

Solving Allen-Cahn and Cahn-Hilliard Equations Using the Adaptive Physics Informed Neural Networks

Phase field models, in particular, the Allen-Cahn type and Cahn-Hilliard type equations, have been widely used to investigate interfacial dynamic problems. Designing accurate, efficient, and stable numerical algorithms for solving the phase field models has been an active field for decades. In this paper, we focus on using the deep neural network to design an automatic numerical solver for the Allen-Cahn and Cahn-Hilliard equations by proposing an improved physics informed neural network (PINN). Though the PINN has been embraced to investigate many differential equation problems, we find a direct application of the PINN in solving phase-field equations won't provide accurate solutions in many cases. Thus, we propose various techniques that add to the approximation power of the PINN. As a major contribution of this paper, we propose to embrace the adaptive idea in both space and time and introduce various sampling strategies, such that we are able to improve the efficiency and accuracy of the PINN on solving phase field equations. In addition, the improved PINN has no restriction on the explicit form of the PDEs, making it applicable to a wider class of PDE problems, and shedding light on numerical approximations of other PDEs in general.     

Investigating and Mitigating Failure Modes in Physics-Informed Neural Networks (PINNs)

This paper explores the difficulties in solving partial differential equations (PDEs) using physics-informed neural networks (PINNs). PINNs use physics as a regularization term in the objective function. However, a drawback of this approach is the requirement for manual hyperparameter tuning, making it impractical in the absence of validation data or prior knowledge of the solution. Our investigations of the loss landscapes and backpropagated gradients in the presence of physics reveal that existing methods produce non-convex loss landscapes that are hard to navigate. Our findings demonstrate that high-order PDEs contaminate backpropagated gradients and hinder convergence. To address these challenges, we introduce a novel method that bypasses the calculation of high-order derivative operators and mitigates the contamination of backpropagated gradients. Consequently, we reduce the dimension of the search space and make learning PDEs with non-smooth solutions feasible. Our method also provides a mechanism to focus on complex regions of the domain. Besides, we present a dual unconstrained formulation based on Lagrange multiplier method to enforce equality constraints on the model’s prediction, with adaptive and independent learning rates inspired by adaptive subgradient methods. We apply our approach to solve various linear and non-linear PDEs.     

Convergence of Physics-Informed Neural Networks Applied to Linear Second-Order Elliptic Interface Problems

With the remarkable empirical success of neural networks across diverse scientific disciplines, rigorous error and convergence analysis are also being developed and enriched. However, there has been little theoretical work focusing on neural networks in solving interface problems. In this paper, we perform a convergence analysis of physics-informed neural networks (PINNs) for solving second-order elliptic interface problems. Specifically, we consider PINNs with domain decomposition technologies and introduce gradient-enhanced strategies on the interfaces to deal with boundary and interface jump conditions. It is shown that the neural network sequence obtained by minimizing a Lipschitz regularized loss function converges to the unique solution to the interface problem in $H^2$ as the number of samples increases. Numerical experiments are provided to demonstrate our theoretical analysis.     

A Rate of Convergence of Physics Informed Neural Networks for the Linear Second Order Elliptic PDEs

In recent years, physical informed neural networks (PINNs) have been shown to be a powerful tool for solving PDEs empirically. However, numerical analysis of PINNs is still missing. In this paper, we prove the convergence rate to PINNs for the second order elliptic equations with Dirichlet boundary condition, by establishing the upper bounds on the number of training samples, depth and width of the deep neural networks to achieve desired accuracy. The error of PINNs is decomposed into approximation error and statistical error, where the approximation error is given in $C^2$ norm with ReLU$^3$ networks (deep network with activation function max$\{0,x^3\}$) and the statistical error is estimated by Rademacher complexity. We derive the bound on the Rademacher complexity of the non-Lipschitz composition of gradient norm with ReLU$^3$ network, which is of immense independent interest.     

High Order Deep Neural Network for Solving High Frequency Partial Differential Equations

This paper proposes a high order deep neural network (HOrderDNN) for solving high frequency partial differential equations (PDEs), which incorporates the idea of "high order" from finite element methods (FEMs) into commonly-used deep neural networks (DNNs) to obtain greater approximation ability. The main idea of HOrderDNN is introducing a nonlinear transformation layer between the input layer and the first hidden layer to form a high order polynomial space with the degree not exceeding $p$, followed by a normal DNN. The order $p$ can be guided by the regularity of solutions of PDEs. The performance of HOrderDNN is evaluated on high frequency function fitting problems and high frequency Poisson and Helmholtz equations. The results demonstrate that: HOrderDNNs($p > 1$) can efficiently capture the high frequency information in target functions; and when compared to physics-informed neural network (PINN), HOrderDNNs($p > 1$) converge faster and achieve much smaller relative errors with same number of trainable parameters. In particular, when solving the high frequency Helmholtz equation in Section 3.5, the relative error of PINN stays around 1 with its depth and width increase, while the relative error can be reduced to around 0.02 as $p$ increases (see Table 5).     

Localized Exponential Time Differencing Method for Shallow Water Equations: Algorithms and Numerical Study

In this paper, we investigate the performance of the exponential time differencing (ETD) method applied to the rotating shallow water equations. Comparing with explicit time stepping of the same order accuracy in time, the ETD algorithms could reduce the computational time in many cases by allowing the use of large time step sizes while still maintaining numerical stability. To accelerate the ETD simulations, we propose a localized approach that synthesizes the ETD method and overlapping domain decomposition. By dividing the original problem into many subdomain problems of smaller sizes and solving them locally, the proposed approach could speed up the calculation of matrix exponential vector products. Several standard test cases for shallow water equations of one or multiple layers are considered. The results show great potential of the localized ETD method for high-performance computing because each subdomain problem can be naturally solved in parallel at every time step.     

A Conservative Parallel Iteration Scheme for Nonlinear Diffusion Equations on Unstructured Meshes

In this paper, a conservative parallel iteration scheme is constructed to solve nonlinear diffusion equations on unstructured polygonal meshes. The design is based on two main ingredients: the first is that the parallelized domain decomposition is embedded into the nonlinear iteration; the second is that prediction and correction steps are applied at subdomain interfaces in the parallelized domain decomposition method. A new prediction approach is proposed to obtain an efficient conservative parallel finite volume scheme. The numerical experiments show that our parallel scheme is second-order accurate, unconditionally stable, conservative and has linear parallel speed-up.     

MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs

In this paper, we propose a machine learning approach via model-operator-data network (MOD-Net) for solving PDEs. A MOD-Net is driven by a model to solve PDEs based on operator representation with regularization from data. For linear PDEs, we use a DNN to parameterize the Green’s function and obtain the neural operator to approximate the solution according to the Green’s method. To train the DNN, the empirical risk consists of the mean squared loss with the least square formulation or the variational formulation of the governing equation and boundary conditions. For complicated problems, the empirical risk also includes a few labels, which are computed on coarse grid points with cheap computation cost and significantly improves the model accuracy. Intuitively, the labeled dataset works as a regularization in addition to the model constraints. The MOD-Net solves a family of PDEs rather than a specific one and is much more efficient than original neural operator because few expensive labels are required. We numerically show MOD-Net is very efficient in solving Poisson equation and one-dimensional radiative transfer equation. For nonlinear PDEs, the nonlinear MOD-Net can be similarly used as an ansatz for solving nonlinear PDEs, exemplified by solving several nonlinear PDE problems, such as the Burgers equation.     

Explicit Computation of Robin Parameters in Optimized Schwarz Waveform Relaxation Methods for Schrödinger Equations Based on Pseudodifferential Operators

The Optimized Schwarz Waveform Relaxation algorithm, a domain decomposition method based on Robin transmission condition, is becoming a popular computational method for solving evolution partial differential equations in parallel. Along with well-posedness, it offers a good balance between convergence rate, efficient computational complexity and simplicity of the implementation. The fundamental question is the selection of the Robin parameter to optimize the convergence of the algorithm. In this paper, we propose an approach to explicitly estimate the Robin parameter which is based on the approximation of the transmission operators at the subdomain interfaces, for the linear/nonlinear Schrödinger equation. Some illustrating numerical experiments are proposed for the one- and two-dimensional problems.     

Learning to Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning

Conservation laws are considered to be fundamental laws of nature. It has broad applications in many fields, including physics, chemistry, biology, geology, and engineering. Solving the differential equations associated with conservation laws is a major branch in computational mathematics. The recent success of machine learning, especially deep learning in areas such as computer vision and natural language processing, has attracted a lot of attention from the community of computational mathematics and inspired many intriguing works in combining machine learning with traditional methods. In this paper, we are the first to view numerical PDE solvers as an MDP and to use (deep) RL to learn new solvers. As proof of concept, we focus on 1-dimensional scalar conservation laws. We deploy the machinery of deep reinforcement learning to train a policy network that can decide on how the numerical solutions should be approximated in a sequential and spatial-temporal adaptive manner. We will show that the problem of solving conservation laws can be naturally viewed as a sequential decision-making process, and the numerical schemes learned in such a way can easily enforce long-term accuracy. Furthermore, the learned policy network is carefully designed to determine a good local discrete approximation based on the current state of the solution, which essentially makes the proposed method a meta-learning approach. In other words, the proposed method is capable of learning how to discretize for a given situation mimicking human experts. Finally, we will provide details on how the policy network is trained, how well it performs compared with some state-of-the-art numerical solvers such as WENO schemes, and supervised learning based approach L3D and PINN, and how well it generalizes.     

Frame Invariance and Scalability of Neural Operators for Partial Differential Equations

Partial differential equations (PDEs) play a dominant role in the mathematical modeling of many complex dynamical processes. Solving these PDEs often requires prohibitively high computational costs, especially when multiple evaluations must be made for different parameters or conditions. After training, neural operators can provide PDEs solutions significantly faster than traditional PDE solvers. In this work, invariance properties and computational complexity of two neural operators are examined for transport PDE of a scalar quantity. Neural operator based on graph kernel network (GKN) operates on graph-structured data to incorporate nonlocal dependencies. Here we propose a modified formulation of GKN to achieve frame invariance. Vector cloud neural network (VCNN) is an alternate neural operator with embedded frame invariance which operates on point cloud data. GKN-based neural operator demonstrates slightly better predictive performance compared to VCNN. However, GKN requires an excessively high computational cost that increases quadratically with the increasing number of discretized objects as compared to a linear increase for VCNN.     

Effective Two-Level Domain Decomposition Preconditioners for Elastic Crack Problems Modeled by Extended Finite Element Method

In this paper, we propose some effective one- and two-level domain decomposition preconditioners for elastic crack problems modeled by extended finite element method. To construct the preconditioners, the physical domain is decomposed into the "crack tip" subdomain, which contains all the degrees of freedom (dofs) of the branch enrichment functions, and the "regular" subdomains, which contain the standard dofs and the dofs of the Heaviside enrichment function. In the one-level additive Schwarz and restricted additive Schwarz preconditioners, the "crack tip" subproblem is solved directly and the "regular" subproblems are solved by some inexact solvers, such as ILU. In the two-level domain decomposition preconditioners, traditional interpolations between the coarse and the fine meshes destroy the good convergence property. Therefore, we propose an unconventional approach in which the coarse mesh is exactly the same as the fine mesh along the crack line, and adopt the technique of a non-matching grid interpolation between the fine and the coarse meshes. Numerical experiments demonstrate the effectiveness of the two-level domain decomposition preconditioners applied to elastic crack problems.     

Numerical Methods for Solving the Hartree-Fock Equations of Diatomic Molecules II

In order to solve the partial differential equations that arise in the Hartree-Fock theory for diatomic molecules and in molecular theories that include electron correlation, one needs efficient methods for solving partial differential equations. In this article, we present numerical results for a two-variable model problem of the kind that arises when one solves the Hartree-Fock equations for a diatomic molecule. We compare results obtained using the spline collocation and domain decomposition methods with third-order Hermite splines to results obtained using the more-established finite difference approximation and the successive over-relaxation method. The theory of domain decomposition presented earlier is extended to treat regions that are divided into an arbitrary number of subregions by families of lines parallel to the two coordinate axes. While the domain decomposition method and the finite difference approach both yield results at the micro-Hartree level, the finite difference approach with a 9-point difference formula produces the same level of accuracy with fewer points. The domain decomposition method has the strength that it can be applied to problems with a large number of grid points. The time required to solve a partial differential equation for a fine grid with a large number of points goes down as the number of partitions increases. The reason for this is that the length of time necessary for solving a set of linear equations in each subregion is very much dependent upon the number of equations. Even though a finer partition of the region has more subregions, the time for solving the set of linear equations in each subregion is very much smaller. This feature of the theory may well prove to be a decisive factor for solving the two-electron pair equation, which – for a diatomic molecule – involves solving partial differential equations with five independent variables. The domain decomposition theory also makes it possible to study complex molecules by dividing them into smaller fragments thatare calculated independently. Since the domain decomposition approach makes it possible to decompose the variable space into separate regions in which the equations are solved independently, this approach is well-suited to parallel computing.     

A Constrained Finite Element Method Based on Domain Decomposition Satisfying the Discrete Maximum Principle for Diffusion Problems

In this paper, we are concerned with the constrained finite element method based on domain decomposition satisfying the discrete maximum principle for diffusion problems with discontinuous coefficients on distorted meshes. The basic idea of domain decomposition methods is used to deal with the discontinuous coefficients. To get the information on the interface, we generalize the traditional Neumann-Neumann method to the discontinuous diffusion tensors case. Then, the constrained finite element method is used in each subdomain. Comparing with the method of using the constrained finite element method on the global domain, the numerical experiments show that not only the convergence order is improved, but also the nonlinear iteration time is reduced remarkably in our method.     

A Diagonal Sweeping Domain Decomposition Method with Source Transfer for the Helmholtz Equation

In this paper, we propose and test a novel diagonal sweeping domain decomposition method (DDM) with source transfer for solving the high-frequency Helmholtz equation in$\mathbb{R}^n$. In the method the computational domain is partitioned into overlapping checkerboard subdomains for source transfer with the perfectly matched layer (PML) technique, then a set of diagonal sweeps over the subdomains are specially designed to solve the system efficiently. The method improves the additive overlapping DDM [43] and the L-sweeps method [50] by employing a more efficient subdomain solving order. We show that the method achieves the exact solution of the global PML problem with $2^n$ sweeps in the constant medium case. Although the sweeping usually implies sequential subdomain solves, the number of sequential steps required for each sweep in the method is only proportional to the $n$-th root of the number of subdomains when the domain decomposition is quasi-uniform with respect to all directions, thus it is very suitable for parallel computing of the Helmholtz problem with multiple right-hand sides through the pipeline processing. Extensive numerical experiments in two and three dimensions are presented to demonstrate the effectiveness and efficiency of the proposed method.     

Optimal spatial field control for controlled release

Most distributed parameter control problems involve manipulation within the spatial domain. Such problems arise in a variety of applications including epidemiology, tissue engineering, and cancer treatment. This paper proposes an approach to solve a state‐constrained spatial field control problem that is motivated by a biomedical application. In particular, the considered manipulation over a spatial field is described by partial differential equations (PDEs) with spatial frequency constraints. The proposed optimization algorithm for tracking a reference spatial field combines three‐dimensional Fourier series, which are truncated to satisfy the spatial frequency constraints, with exploitation of structural characteristics of the PDEs. The computational efficiency and performance of the optimization algorithm are demonstrated in a numerical example. In the example, the spatial tracking error is shown to be almost entirely due to the limitation on the spatial frequency of the manipulated field. The numerical results suggest that the proposed optimal control approach has promise for controlling the release of macromolecules in tissue engineering applications.     

DL-PDE: Deep-Learning Based Data-Driven Discovery of Partial Differential Equations from Discrete and Noisy Data

In recent years, data-driven methods have been developed to learn dynamical systems and partial differential equations (PDE). The goal of such work is to discover unknown physics and corresponding equations. However, prior to achieving this goal, major challenges remain to be resolved, including learning PDE under noisy data and limited discrete data. To overcome these challenges, in this work, a deep-learning based data-driven method, called DL-PDE, is developed to discover the governing PDEs of underlying physical processes. The DL-PDE method combines deep learning via neural networks and data-driven discovery of PDE via sparse regressions. In the DL-PDE, a neural network is first trained, then a large amount of meta-data is generated, and the required derivatives are calculated by automatic differentiation. Finally, the form of PDE is discovered by sparse regression. The proposed method is tested with physical processes, governed by the diffusion equation, the convection-diffusion equation, the Burgers equation, and the Korteweg-de Vries (KdV) equation, for proof-of-concept and applications in real-world engineering settings. The proposed method achieves satisfactory results when data are noisy and limited.     

Two-Dimensional Multi-Domain Hybrid Spectral-WENO Methods for Conservation Laws

The multi-domain hybrid Spectral-WENO method (Hybrid) is introduced for the numerical solution of two-dimensional nonlinear hyperbolic systems in a Cartesian physical domain which is partitioned into a grid of rectangular subdomains. The main idea of the Hybrid scheme is to conjugate the spectral and WENO methods for solving problems with shock or high gradients such that the scheme adapts its solver spatially and temporally depending on the smoothness of the solution in a given subdomain. Built as a multi-domain method, an adaptive algorithm is used to keep the solutions parts exhibiting high gradients and discontinuities always inside WENO subdomains while the smooth parts of the solution are kept inside spectral ones, avoiding oscillations related to the well-known Gibbs phenomenon and increasing the numerical efficiency of the overall scheme. A higher order version of the multi-resolution analysis proposed by Harten is used to determine the smoothness of the solution in each subdomain. We also discuss interface conditions for the two-dimensional problem and the switching procedure between WENO and spectral subdomains. The Hybrid method is applied to the two-dimensional Shock-Vortex Interaction and the Richtmyer-Meshkov Instability (RMI) problems.     

Additive Schwarz Preconditioners with Minimal Overlap for Triangular Spectral Elements

The additive Schwarz preconditioner with minimal overlap is extended to triangular spectral elements (TSEM). The method is a generalization of the corresponding method in tensorial quadrilateral spectral elements (QSEM). The proposed preconditioners are based on partitioning the domain into overlapping subdomains, solving local problems on these subdomains and solving an additional coarse problem associated with the subdomain mesh. The results of numerical experiments show that the proposed preconditioner are robust with respect to the number of elements and are more efficient than the preconditioners with generous overlaps.