On the Convergence of Physics Informed Neural Networks for Linear Second-Order Elliptic and Parabolic Type PDEs

Physics informed neural networks (PINNs) are deep learning based techniques for solving partial differential equations (PDEs) encountered in computational science and engineering. Guided by data and physical laws, PINNs find a neural network that approximates the solution to a system of PDEs. Such a neural network is obtained by minimizing a loss function in which any prior knowledge of PDEs and data are encoded. Despite its remarkable empirical success in one, two or three dimensional problems, there is little theoretical justification for PINNs.As the number of data grows, PINNs generate a sequence of minimizers which correspond to a sequence of neural networks. We want to answer the question: Does the sequence of minimizers converge to the solution to the PDE? We consider two classes of PDEs: linear second-order elliptic and parabolic. By adapting the Schauder approach and the maximum principle, we show that the sequence of minimizers strongly converges to the PDE solution in $C^0$. Furthermore, we show that if each minimizer satisfies the initial/boundary conditions, the convergence mode becomes $H^1$. Computational examples are provided to illustrate our theoretical findings. To the best of our knowledge, this is the first theoretical work that shows the consistency of PINNs.     

MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs

In this paper, we propose a machine learning approach via model-operator-data network (MOD-Net) for solving PDEs. A MOD-Net is driven by a model to solve PDEs based on operator representation with regularization from data. For linear PDEs, we use a DNN to parameterize the Green’s function and obtain the neural operator to approximate the solution according to the Green’s method. To train the DNN, the empirical risk consists of the mean squared loss with the least square formulation or the variational formulation of the governing equation and boundary conditions. For complicated problems, the empirical risk also includes a few labels, which are computed on coarse grid points with cheap computation cost and significantly improves the model accuracy. Intuitively, the labeled dataset works as a regularization in addition to the model constraints. The MOD-Net solves a family of PDEs rather than a specific one and is much more efficient than original neural operator because few expensive labels are required. We numerically show MOD-Net is very efficient in solving Poisson equation and one-dimensional radiative transfer equation. For nonlinear PDEs, the nonlinear MOD-Net can be similarly used as an ansatz for solving nonlinear PDEs, exemplified by solving several nonlinear PDE problems, such as the Burgers equation.     

DL-PDE: Deep-Learning Based Data-Driven Discovery of Partial Differential Equations from Discrete and Noisy Data

In recent years, data-driven methods have been developed to learn dynamical systems and partial differential equations (PDE). The goal of such work is to discover unknown physics and corresponding equations. However, prior to achieving this goal, major challenges remain to be resolved, including learning PDE under noisy data and limited discrete data. To overcome these challenges, in this work, a deep-learning based data-driven method, called DL-PDE, is developed to discover the governing PDEs of underlying physical processes. The DL-PDE method combines deep learning via neural networks and data-driven discovery of PDE via sparse regressions. In the DL-PDE, a neural network is first trained, then a large amount of meta-data is generated, and the required derivatives are calculated by automatic differentiation. Finally, the form of PDE is discovered by sparse regression. The proposed method is tested with physical processes, governed by the diffusion equation, the convection-diffusion equation, the Burgers equation, and the Korteweg-de Vries (KdV) equation, for proof-of-concept and applications in real-world engineering settings. The proposed method achieves satisfactory results when data are noisy and limited.     

High Order Deep Neural Network for Solving High Frequency Partial Differential Equations

This paper proposes a high order deep neural network (HOrderDNN) for solving high frequency partial differential equations (PDEs), which incorporates the idea of "high order" from finite element methods (FEMs) into commonly-used deep neural networks (DNNs) to obtain greater approximation ability. The main idea of HOrderDNN is introducing a nonlinear transformation layer between the input layer and the first hidden layer to form a high order polynomial space with the degree not exceeding $p$, followed by a normal DNN. The order $p$ can be guided by the regularity of solutions of PDEs. The performance of HOrderDNN is evaluated on high frequency function fitting problems and high frequency Poisson and Helmholtz equations. The results demonstrate that: HOrderDNNs($p > 1$) can efficiently capture the high frequency information in target functions; and when compared to physics-informed neural network (PINN), HOrderDNNs($p > 1$) converge faster and achieve much smaller relative errors with same number of trainable parameters. In particular, when solving the high frequency Helmholtz equation in Section 3.5, the relative error of PINN stays around 1 with its depth and width increase, while the relative error can be reduced to around 0.02 as $p$ increases (see Table 5).     

Solving Allen-Cahn and Cahn-Hilliard Equations Using the Adaptive Physics Informed Neural Networks

Phase field models, in particular, the Allen-Cahn type and Cahn-Hilliard type equations, have been widely used to investigate interfacial dynamic problems. Designing accurate, efficient, and stable numerical algorithms for solving the phase field models has been an active field for decades. In this paper, we focus on using the deep neural network to design an automatic numerical solver for the Allen-Cahn and Cahn-Hilliard equations by proposing an improved physics informed neural network (PINN). Though the PINN has been embraced to investigate many differential equation problems, we find a direct application of the PINN in solving phase-field equations won't provide accurate solutions in many cases. Thus, we propose various techniques that add to the approximation power of the PINN. As a major contribution of this paper, we propose to embrace the adaptive idea in both space and time and introduce various sampling strategies, such that we are able to improve the efficiency and accuracy of the PINN on solving phase field equations. In addition, the improved PINN has no restriction on the explicit form of the PDEs, making it applicable to a wider class of PDE problems, and shedding light on numerical approximations of other PDEs in general.     

Simulation of Inviscid Compressible Flows Using PDE Transform

The solution of systems of hyperbolic conservation laws remains an interesting and challenging task due to the diversity of physical origins and complexity of the physical situations. The present work introduces the use of the partial differential equation (PDE) transform, paired with the Fourier pseudospectral method (FPM), as a new approach for hyperbolic conservation law problems. The PDE transform, based on the scheme of adaptive high order evolution PDEs, has recently been applied to decompose signals, images, surfaces and data to various target functional mode functions such as trend, edge, texture, feature, trait, noise, etc. Like wavelet transform, the PDE transform has controllable time-frequency localization and perfect reconstruction. A fast PDE transform implemented by the fast Fourier Transform (FFT) is introduced to avoid stability constraint of integrating high order PDEs. The parameters of the PDE transform are adaptively computed to optimize the weighted total variation during the time integration of conservation law equations. A variety of standard benchmark problems of hyperbolic conservation laws is employed to systematically validate the performance of the present PDE transform based FPM. The impact of two PDE transform parameters, i.e., the highest order and the propagation time, is carefully studied to deliver the best effect of suppressing Gibbs' oscillations. The PDE orders of 2-6 are used for hyperbolic conservation laws of low oscillatory solutions, while the PDE orders of 8-12 are often required for problems involving highly oscillatory solutions, such as shock-entropy wave interactions. The present results are compared with those in the literature. It is found that the present approach not only works well for problems that favor low order shock capturing schemes, but also exhibits superb behavior for problems that require the use of high order shock capturing methods.     

Extended Physics-Informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition Based Deep Learning Framework for Nonlinear Partial Differential Equations

We propose a generalized space-time domain decomposition approach for the physics-informed neural networks (PINNs) to solve nonlinear partial differential equations (PDEs) on arbitrary complex-geometry domains. The proposed framework, named eXtended PINNs ($XPINNs$), further pushes the boundaries of both PINNs as well as conservative PINNs (cPINNs), which is a recently proposed domain decomposition approach in the PINN framework tailored to conservation laws. Compared to PINN, the XPINN method has large representation and parallelization capacity due to the inherent property of deployment of multiple neural networks in the smaller subdomains. Unlike cPINN, XPINN can be extended to any type of PDEs. Moreover, the domain can be decomposed in any arbitrary way (in space and time), which is not possible in cPINN. Thus, XPINN offers both space and time parallelization, thereby reducing the training cost more effectively. In each subdomain, a separate neural network is employed with optimally selected hyperparameters, e.g., depth/width of the network, number and location of residual points, activation function, optimization method, etc. A deep network can be employed in a subdomain with complex solution, whereas a shallow neural network can be used in a subdomain with relatively simple and smooth solutions. We demonstrate the versatility of XPINN by solving both forward and inverse PDE problems, ranging from one-dimensional to three-dimensional problems, from time-dependent to time-independent problems, and from continuous to discontinuous problems, which clearly shows that the XPINN method is promising in many practical problems. The proposed XPINN method is the generalization of PINN and cPINN methods, both in terms of applicability as well as domain decomposition approach, which efficiently lends itself to parallelized computation. The XPINN code is available on $https://github.com/AmeyaJagtap/XPINNs$.     

The phosphodiesterase inhibitory selectivity and the in vitro and in vivo potency of the new PDE5 inhibitor vardenafil

We investigated the potency and the selectivity profile of vardenafil on phosphodiesterase (PDEs) enzymes, its ability to modify cGMP metabolism and cause relaxation of penile smooth muscle and its effect on erections in vivo under conditions of exogenous nitric oxide (NO) stimulation. PDE isozymes were extracted and purified from human platelets (PDE5) or bovine sources (PDEs 1, 2, 3, 4 and 6). The inhibition of these PDEs and of human recombinant PDEs by vardenafil was determined. The ability to potentiate NO-mediated relaxation and influence cGMP levels in human corpus cavernosum strips was measured in vitro, and erection-inducing activity was demonstrated in conscious rabbits after oral administration together with intravenous doses of sodium nitroprusside (SNP). The effects of vardenafil were compared with those of the well-recognized PDE5 inhibitor, sildenafil (values for sildenafil in brackets). Vardenafil specifically inhibited the hydrolysis of cGMP by PDE5 with an IC50 of 0.7 nM (6.6 nM). In contrast, the IC50 of vardenafil for PDE1 was 180 nM; for PDE6, 11 nM; for PDE2, PDE3 and PDE4, more than 1000 nM. Relative to PDE5, the ratios of the IC50 for PDE1 were 257 (60), for PDE6 16 (7.4). Vardenafil significantly enhanced the SNP-induced relaxation of human trabecular smooth muscle at 3 nM (10 nM). Vardenafil also significantly potentiated both ACh-induced and transmural electrical stimulation-induced relaxation of trabecular smooth muscle. The minimum concentration of vardenafil that significantly potentiated SNP-induced cGMP accumulation was 3 nM (30 nM). In vivo studies in rabbits showed that orally administered vardenafil dose-dependently potentiated erectile responses to intravenously administered SNP. The minimal effective dose that significantly potentiated erection was 0.1 mg/kg (1 mg/kg). The selectivity for PDE5, the potentiation of NO-induced relaxation and cGMP accumulation in human trabecular smooth muscle and the ability to enhance NO-induced erection in vivo indicate that vardenafil has the appropriate properties to be a potential compound for the treatment of erectile dysfunction. Vardenafil was more potent and selective than sildenafil on its inhibitory activity on PDE5.     

Implications of PDE4 structure on inhibitor selectivity across PDE families

Phosphodiesterases (PDEs) control cellular concentrations of cyclic adenosine monophosphate (cAMP) or cyclic guanosine monophosphate (cGMP). PDE4 and PDE5 selectively hydrolyze cAMP and cGMP, respectively. PDE family members share approximately 25% sequence identity within a conserved catalytic domain of about 300 amino acids. Crystal structure analysis of PDE4's catalytic domain identifies two metal-binding sites: a high-affinity site and a low-affinity site, which probably bind zinc (Zn2+) and magnesium (Mg2+), respectively. Absolute conservation among the PDEs of two histidine and two aspartic acid residues for divalent metal binding suggests the importance of these amino acids in catalysis. Although active sites of PDEs are apparently structurally similar, PDE4 is specifically inhibited by selective inhibitors such as rolipram, while PDE5 is preferentially blocked by sildenafil. Modeling interactions of the PDE5 inhibitor sildenafil with the PDE4 active site may help explain inhibitor selectivity and provide useful information for the design of new inhibitors.     

Potency, selectivity, and consequences of nonselectivity of PDE inhibition

Phosphodiesterases (PDEs) play a decisive role in cyclic nucleotide-mediated intracellular signaling. As PDEs are expressed in a variety of tissues, selectivity is a prerequisite for a therapeutically applicable PDE inhibitor. Sildenafil, vardenafil, and tadalafil are selective for PDE5, with vardenafil exhibiting the highest potency and minimal inhibition of other PDEs, with the exception of PDE6. Tadalafil is extremely selective for PDE5, but also potently inhibits PDE11, an enzyme with unknown physiological function. As PDE1 is expressed in the brain, myocardium, and vascular smooth muscle cells, nonselectivity with respect to this enzyme (selectivity: tadalafil>vardenafil>sildenafil) may result in vasodilation and tachycardia. Inhibition of PDE6 (selectivity: tadalafil>vardenafil congruent with sildenafil), which is expressed only in retina and functions in visual transduction, can transiently disturb vision. PDE5 inhibitors may also indirectly inhibit PDE3 by increasing cyclic guanosine monophospate levels, thereby elevating heart rate and vasodilation while inhibiting platelet aggregation.     

Potential therapeutic applications of phosphodiesterase inhibition in prostate cancer

Objective

Phosphodiesterases (PDEs) play a role in controlling cyclic nucleotide action, including cyclic guanosine monophosphate (cGMP). Previous studies have ascribed a protective role of cGMP signaling on hypoxia-mediated cancer progression. Herein, we determine their potential role in hypoxia-mediated chemoresistance and immune escape.

Materials and Methods

Phosphodiesterase assays were used to measure PDE activity in prostate cancer cell lines (DU145, PC3). Immunoblots were performed to determine the presence of PDEs in human prostate tissue samples. The effect of PDE inhibition on hypoxia-induced chemoresistance (compared to normoxic controls, 20% O₂) was determined using clonogenic assays. Flow cytometry was used to determine the effects of PDE inhibition on surface MHC class I-related chain A (MICA), a natural killer (NK) cell-activating ligand. A mouse model was used to evaluate the in vivo effects of PDE inhibition on the growth of human prostate cancer cells.

Results

PDE5 and PDE11 were the most prominent PDEs in the cell lines, representing between 86 and 95% of the total cGMP-specific PDE activity. Treatment of DU-145 cells with a PDE inhibitor significantly reduced the hypoxia-associated acquisition of resistance to doxorubicin, with a mean 51% reduction in surviving fraction compared to controls (p < 0.001, ANOVA). As well, PDE inhibition completely reversed (p = 0.02, ANOVA) hypoxia-induced shedding of the immune stimulatory molecule, MICA, and attenuated the growth of human prostate tumor xenografts in an NK cell-competent murine model (p = 0.03, Wilcoxon, Mann–Whitney).

Conclusions

These results suggest a rationale for future studies on the potential therapeutic applications of PDE inhibitors in men with prostate cancer.     

Investigating and Mitigating Failure Modes in Physics-Informed Neural Networks (PINNs)

This paper explores the difficulties in solving partial differential equations (PDEs) using physics-informed neural networks (PINNs). PINNs use physics as a regularization term in the objective function. However, a drawback of this approach is the requirement for manual hyperparameter tuning, making it impractical in the absence of validation data or prior knowledge of the solution. Our investigations of the loss landscapes and backpropagated gradients in the presence of physics reveal that existing methods produce non-convex loss landscapes that are hard to navigate. Our findings demonstrate that high-order PDEs contaminate backpropagated gradients and hinder convergence. To address these challenges, we introduce a novel method that bypasses the calculation of high-order derivative operators and mitigates the contamination of backpropagated gradients. Consequently, we reduce the dimension of the search space and make learning PDEs with non-smooth solutions feasible. Our method also provides a mechanism to focus on complex regions of the domain. Besides, we present a dual unconstrained formulation based on Lagrange multiplier method to enforce equality constraints on the model’s prediction, with adaptive and independent learning rates inspired by adaptive subgradient methods. We apply our approach to solve various linear and non-linear PDEs.     

Probabilistic High Order Numerical Schemes for Fully Nonlinear Parabolic PDEs

In this paper, we are concerned with probabilistic high order numerical schemes for Cauchy problems of fully nonlinear parabolic PDEs. For such parabolic PDEs, it is shown by Cheridito, Soner, Touzi and Victoir [4] that the associated exact solutions admit probabilistic interpretations, i.e., the solution of a fully nonlinear parabolic PDE solves a corresponding second order forward backward stochastic differential equation (2FBSDEs). Our numerical schemes rely on solving those 2FBSDEs, by extending our previous results [W. Zhao, Y. Fu and T. Zhou, SIAM J. Sci. Comput., 36 (2014), pp. A1731-A1751.]. Moreover, in our numerical schemes, one has the flexibility to choose the associated forward SDE, and a suitable choice can significantly reduce the computational complexity. Various numerical examples including the HJB equations are presented to show the effectiveness and accuracy of the proposed numerical schemes.     

Numerical Simulations for Full History Recursive Multilevel Picard Approximations for Systems of High-Dimensional Partial Differential Equations

One of the most challenging issues in applied mathematics is to develop and analyze algorithms which are able to approximately compute solutions of high-dimensional nonlinear partial differential equations (PDEs). In particular, it is very hard to develop approximation algorithms which do not suffer under the curse of dimensionality in the sense that the number of computational operations needed by the algorithm to compute an approximation of accuracy $ε$>0 grows at most polynomially in both the reciprocal 1/$ε$ of the required accuracy and the dimension $d∈\mathbb{N}$of the PDE. Recently, a new approximation method, the so-called full history recursive multilevel Picard (MLP) approximation method, has been introduced and, until today, this approximation scheme is the only approximation method in the scientific literature which has been proven to overcome the curse of dimensionality in the numerical approximation of semilinear PDEs with general time horizons. It is a key contribution of this article to extend the MLP approximation method to systems of semilinear PDEs and to numerically test it on several example PDEs. More specifically, we apply the proposed MLP approximation method in the case of Allen-Cahn PDEs, Sine-Gordon-type PDEs, systems of coupled semilinear heat PDEs, and semilinear Black-Scholes PDEs in up to 1000 dimensions. We also compare the performance of the proposed MLP approximation algorithm with a deep learning based approximation method from the scientific literature.     

Effects of phosphodiesterase 7 inhibition by RNA interference on the gene expression and differentiation of human mesenchymal stem cell-derived osteoblasts

The second messenger molecule cyclic adenosine monophosphate (cAMP) plays an important role in the hormonal regulation of bone metabolism. cAMP is inactivated by the cyclic nucleotide phosphodiesterases (PDEs), a superfamily of enzymes divided into 11 known families designated PDE 1-11. The aim of this study was to investigate the effect of PDE7 and PDE8 inhibition on the gene expression and differentiation of human osteoblasts. Osteoblasts differentiated from human mesenchymal stem cells (hMSC) were cultured and treated with short interfering RNAs (siRNAs) generated from PDE7 and PDE8 PCR products. Total RNA was isolated from the cells, and gene expression was assayed with cDNA microarray and quantitative real-time PCR. bALP measurements were assayed during differentiation, and mineralization was determined by quantitative Alizarin red S staining. PDE7 and PDE8 inhibition by RNA interference decreased the gene expression of PDE7A by 60-70%, PDE7B by 40-50%, and PDE8A by 30%. PDE7 silencing increased the expression of beta-catenin, osteocalcin, caspase-8, and cAMP-responsive element-binding protein 5 (CREB-5) genes and decreased the expression of the 1, 25-dihydroxyvitamin D3 receptor gene. PDE8A silencing increased the expression of anti-apoptotic genes, but decreased the expression of osteoglycin (osteoinductive factor) and bone morphogenetic protein 1 (BMP-1). PDE7 silencing increased bALP and mineralization up to three-fold compared to controls. Treatment with the PDE7-selective PDE inhibitor BRL-50481 had similar effects on mineralization as the gene silencing. The PDE7 silencing also increased forskolin stimulated cAMP response, but had no effect on the proliferation rate. Furthermore, osteocalcin expression was increased by PDE7 silencing by a mechanism dependent on protein kinase A. Our results show that specific gene silencing with the RNAi method is a useful tool for inhibiting the gene expression of specific PDEs and that PDE7 silencing upregulates several osteogenic genes and increases mineralization. PDE7 may play an important role in the regulation of osteoblastic differentiation.     

Phosphodiesterase 11 (PDE11) regulation of spermatozoa physiology

Fertilization is well correlated with sperm concentration, rate of forward motility, and percentage of live, uncapacitated ejaculated spermatozoa, which is regulated in part by cyclic adenosine monophosphate (cAMP) and cyclic guanosine monophosphate (cGMP). Phosphodiesterases (PDEs) hydrolyze cyclic nucleotides to their corresponding monophosphates, thereby counterbalancing the activities of cAMP and cGMP, and PDE11 is highly expressed in the testis, prostate, and developing spermatozoa. However, a physiological role of PDE11 is not known. We generated PDE11 knockout (PDE11-/-) mice to investigate the role of PDE11 in spermatozoa physiology. Ejaculated sperm from PDE11-/- mice displayed reduced sperm concentration, rate of forward progression, and percentage of live spermatozoa. Pre-ejaculated sperm from PDE11-/- mice displayed increased premature/spontaneous capacitance. These data are consistent with human data and suggest a role for PDE11 in spermatogenesis and fertilization potential. This is the first phenotype described for the PDE11-/- mouse and the first report of a physiological role for PDE11.     

Preface

Machine learning has been gaining recognition rapidly as a powerful computational technique to address some of the most challenging problems arising from scientific and engineering computations (SEC) with promising results in simulations of biological and quantum systems, fluid dynamics, wave scattering, high dimensional PDEs, and inverse problems, etc. This special issue contains 1 survey paper and 17 original research articles on recent developments in machine learning, especially deep neural networks, concerning both its theoretical and algorithmic aspects pertinent to SEC.     

Oral phosphodiesterase-5 inhibitors and sperm functions

This review aims to elucidate the possible effects of phosphodiesterase-5 (PDE5) inhibitors on sperm functions. PDEs hydrolyze cyclic nucleotides, and together with adenylyl and guanylyl cyclase, which catalyze the formation of cAMP and cGMP, regulate the levels of these second messengers in cells. cGMP-specific PDE5 is one of the PDEs that have been intensively studied because of its fundamental pharmacological relevance, as oral PDE5 inhibitors are used successfully in treating erectile dysfunction. In addition, they have shown diverse beneficial actions in different disease categories. Specific relevance of the cGMP system in reproductive functions has been recently proposed. Its use was shown to be devoid of effects on semen volume, concentration, sperm membrane integrity or sperm penetration assay. Most available studies demonstrated a significant increase in sperm motility and viability both in vivo and in vitro, which seems to be enhanced at low doses and reduced at high concentrations. Also, these molecules showed a role in capacitation and a debated one concerning acrosome reaction. However, due to the relative short period since the launching of oral PDE5 inhibitors, more investigations should be carried out in wider scales to assess their effect(s) on variant sperm function that could be beneficial as potential therapeutic approaches.     

An Adaptive Surrogate Modeling Based on Deep Neural Networks for Large-Scale Bayesian Inverse Problems

In Bayesian inverse problems, surrogate models are often constructed to speed up the computational procedure, as the parameter-to-data map can be very expensive to evaluate. However, due to the curse of dimensionality and the nonlinear concentration of the posterior, traditional surrogate approaches (such us the polynomial-based surrogates) are still not feasible for large scale problems. To this end, we present in this work an adaptive multi-fidelity surrogate modeling framework based on deep neural networks (DNNs), motivated by the facts that the DNNs can potentially handle functions with limited regularity and are powerful tools for high dimensional approximations. More precisely, we first construct offline a DNN-based surrogate according to the prior distribution, and then, this prior-based DNN-surrogate will be adaptively & locally refined online using only a few high-fidelity simulations. In particular, in the refine procedure, we construct a new shallow neural network that views the previous constructed surrogate as an input variable – yielding a composite multi-fidelity neural network approach. This makes the online computational procedure rather efficient. Numerical examples are presented to confirm that the proposed approach can obtain accurate posterior information with a limited number of forward simulations.