Benchmarks

[willtebbutt]

Benchmarking AD tools has come up a lot recently, and this seems like a good place to implement some benchmarks, in addition to "correctness" testing.

I was thinking that they should be micro-benchmarks, and the benchmarks themselves shouldn't depend on any functionality outside of Base and the standard libraries, with the possible exception of things that are needed to test supports for accelerators eg. CuArrays.jl. Equally these could be supported by typing things sufficiently abstractly 🤷

The first thing to do is figure out what it people actually care about the performance of. For example, I really care about broadcasting and operations involving linear algebra, but not so much about control flow, but I know that the Turing team has a different set of priorities. So perhaps if everyone could solicit what sorts of things they're interested in benchmarking, we can start to think about how to chop up tests. For example, there's a distinction between control-flow that depends on values and control-flow that doesn't from the perspective of reverse-mode AD, so we should probably be testing that kind of thing.

cc @vchuravy @yebai @oxinabox

[vchuravy]

When I asked the question people pointed me to various benchmarks/tests, they had flying around:

• https://github.com/GiggleLiu/nilangpaper/tree/master/benchmarks
• Forward mode AD benchmark suite FluxML/Zygote.jl#568
• https://github.com/JuliaDiff/ChainRules.jl/tree/master/test
• https://github.com/jrevels/MixedModeBroadcastAD.jl/blob/master/test/kernels.jl
• https://github.com/PumasAI/sensitivity_analysis_paper