Ordinary and partial differential equations (DE) are used extensively in
scientific and mathematical domains to model physical systems. Current
literature has focused primarily on deep neural network (DNN) based methods for
solving a specific DE or a family of DEs. Research communities with a history
of using DE models may view DNN-based differential equation solvers (DNN-DEs)
as a faster and transferable alternative to current numerical methods. However,
there is a lack of systematic surveys detailing the use of DNN-DE methods
across physical application domains and a generalized taxonomy to guide future
research. This paper surveys and classifies previous works and provides an
educational tutorial for senior practitioners, professionals, and graduate
students in engineering and computer science. First, we propose a taxonomy to
navigate domains of DE systems studied under the umbrella of DNN-DE. Second, we
examine the theory and performance of the Physics Informed Neural Network
(PINN) to demonstrate how the influential DNN-DE architecture mathematically
solves a system of equations. Third, to reinforce the key ideas of solving and
discovery of DEs using DNN, we provide a tutorial using DeepXDE, a Python
package for developing PINNs, to develop DNN-DEs for solving and discovering a
classic DE, the linear transport equation.