Testing weird types

[willtebbutt]

Couldn't think of a better place to raise this, so 🤷‍♂ .

Proposal

A long time ago Jarrett made DiffTests.jl. That package is useful for testing whether your AD tool works well on problems that are tricky to differentiate. What we don't currently have is a package to test whether your package is compatible with the API that ChainRules specifies because, when DiffTests was written, the Julia AD world only really believed in numbers and arrays of numbers.

In short, I would like a package that tells me: "does my AD tool work with this weird type input and it's differential.", for increasingly complicated set of types. Bonus points for producing a nice human-readable report at the end.

IIRC e.g. ForwardDiff, ReverseDiff, Tracker etc only believe in numbers and dense arrays of numbers. ForwardDiff2 is a bit more flexible, and Zygote is more flexible still. FiniteDifferencing should really be almost as flexible as Zygote on this front, but it's not clear that it currently is, and it's hard to know what it should look like for this to be the case. FiniteDiff also only believes in numbers and arrays of numbers. This range of capabilities provides strong motivation for this package.

To summarise, this package would help

• everyone to better understand the landscape,
• package authors can get a feel for where they sit,
• any time we find a type that feels hard to work with, we have a package that we can encode this in and test everything else against.

[oxinabox]

Should we just resurrect DiffTests.jl and add that there?

I assume we are thinking of hard things like types that overload properties,
and things like FillArrays etc?

[willtebbutt]

Should we just resurrect DiffTests.jl and add that there?

Sounds reasonable.

I assume we are thinking of hard things like types that overload properties,
and things like FillArrays etc?

Precisely. I was thinking the main set of tests would just involve differentiating the identity function and checking that the AD in question doesn't break.