SciML / SciMLBenchmarks.jl

SciMLBenchmarks.jl: Benchmarks for Scientific Machine Learning (SciML) and Equation Solvers SciMLBenchmarks.jl holds webpages, pdfs, and notebooks showing the benchmarks for the SciML Scientific Machine Learning Software ecosystem, including: • Benchmarks of equation solver implementations • Speed and robustness comparisons of methods for parameter estimation / inverse problems • Training universal differential equations (and subsets like neural ODEs) • Training of physics-informed neural networks (PINNs) • Surrogate comparisons, including radial basis functions, neural operators (DeepONets, Fourier Neural Operators), and more The SciML Bench suite is made to be a comprehensive open source benchmark from the ground up, covering the methods of computational science and scientific computing all the way to AI for science. Rules: Optimal, Fair, and Reproducible These benchmarks are meant to represent good optimized coding style. Benchmarks are preferred to be run on the provided open benchmarking hardware for full reproducibility (though in some cases, such as with language barriers, this can be difficult). Each benchmark is documented with the compute devices used along with package versions for necessary reproduction. These benchmarks attempt to measure in terms of work-precision efficiency, either timing with approximately matching error or building work-precision diagrams for direct comparison of speed at given error tolerances. **If any of the code from any of the languages can be improved, please open a pull request**. For critiques of benchmarks, please open a pull request that changes the code in the desired manner. Issues with recommended changes are generally vague and not actionable, while pull requests with code changes are exact. Thus if there is something you think should be changed in the code, please make the recommended change in the code! Results To view the results of the SciML Benchmarks, go to benchmarks.sciml.ai. By default, this will lead to the latest tagged version of the benchmarks. To see the in-development version of the benchmarks, go to https://benchmarks.sciml.ai/dev/. Static outputs in pdf, markdown, and html reside in SciMLBenchmarksOutput. Citing To cite the SciML Benchmarks, please cite the following: Current Summary The following is a quick summary of the benchmarks. These paint broad strokes over the set of tested equations and some specific examples may differ. Non-Stiff ODEs • OrdinaryDiffEq.jl's methods are the most efficient by a good amount • The methods tend to do the best in every benchmark of this category • At lower tolerances, does well consistently. • ARKODE and Hairer's / perform very similarly, but are both far less efficient than the methods. • The multistep methods, and , tend to not do very well. • The ODEInterface multistep method does not do as well as the other multistep methods. • ODE.jl's methods are not able to consistently solve the problems. • Fixed time step methods are less efficient than the adaptive methods. Stiff ODEs • In this category, the best methods are much more problem dependent. • For smaller problems: • , , and tend to be the most efficient at high tolerances. • and tend to be the most efficient at low tolerances. • For larger problems (Filament PDE): • and do the best at all normal tolerances. • The ESDIRK methods like and can come close. • is always the most efficient when tolerances go to the low extreme ( ) • Fixed time step methods tend to diverge on every tested problem because the high stiffness results in divergence of the Newton solvers. • ARKODE is very inconsistent and requires a lot of tweaking in order to not diverge on many of the tested problems. When it doesn't diverge, the similar algorithms in OrdinaryDiffEq.jl ( ) are much more efficient in most cases. • GeometricIntegrators.jl fails to converge on any of the tested problems. Dynamical ODEs • Higher order (generally order >=6) symplectic integrators are much more efficient than the lower order counterparts. • For high accuracy, using a symplectic integrator is not preferred. Their extra cost is not necessary since the other integrators are able to not drift simply due to having low enough error. • In this class, the methods are by far the most efficient. The methods do well for not being specific to the domain. Non-Stiff SDEs • For simple 1-dimensional SDEs at low accuracy, the and methods can do well. Beyond that, they are simply outclassed. • The and methods both are very similar within-class on the simple SDEs. • is the most efficient when applicable and the tolerances are low. • Generally, only low accuracy is necessary to get to sampling error of the mean. • The adaptive method is very conservative with error estimates. Stiff SDEs • The high order adaptive methods ( ) generally do well on stiff problems. • The "standard" low-order implicit methods, and , do not do well on all stiff problems. Some exceptions apply to well-behaved problems like the Stochastic Heat Equation. Non-Stiff DDEs • The efficiency ranking tends to match the ODE Tests, but the cutoff from low to high tolerance is lower. • does well in a large class of problems here. • The methods do well in low tolerance cases. Stiff DDEs • The Rosenbrock methods, specifically , perform well. Parameter Estimation • Broadly two different approaches have been used, Bayesian Inference and Optimisation algorithms. • In general it seems that the optimisation algorithms perform more accurately but that can be attributed to the larger number of data points being used in the optimisation cases, Bayesian approach tends to be slower of the two and hence lesser data points are used, accuracy can increase if proper data is used. • Within the different available optimisation algorithms, BBO from the BlackBoxOptim package and GN_CRS2_LM for the global case while LD_SLSQP,LN_BOBYQA and LN_NELDERMEAD for the local case from the NLopt package perform the best. •…