Improve performance of differentiable matrix calculations for Pauli Sentences

[albi3ro]

Context:

The new operator arithmetic now enabled provides a fully differentiable pathway for calculating the matrix that was not available before.

Unfortunately, a differentiable calculation of a large dense matrix can be quite slow, and users may not realize they should be using a sparse matrix formulation instead. Just because we offer it does not make it a universally good idea.

This PR hopes to close some of that performance gap, by taking advantage of the structure of pauli sentences to improve the efficiency of calculating the matrix.

Description of the Change:

Adds in a new pathway for calculating the matrix of a pauli sentence.

Benefits:

Some performance numbers for the H5 molecule follow:

While the differentiable matrix is still not as fast as sparse matrix calculation, this method makes up a lot of ground. Note that the dense matrix is now on the same order of magnitude as the sparse matrix calculation for the legacy hamiltonian.

This formulation is also jit-able, and jitting the calculation of the matrix allows us to pick up another order of magnitude improvement.

Possible Drawbacks:

Dense is still not as fast as the sparse matrix, and might not scale as well as the sparse formulation.

Also, this pathway is much less intuitive and involves some numerical black magic.

Related GitHub Issues:

[github-actions]

Hello. You may have forgotten to update the changelog!
Please edit doc/releases/changelog-dev.md with:

• A one-to-two sentence description of the change. You may include a small working example for new features.
• A link back to this PR.
• Your name (or GitHub username) in the contributors section.

[vincentmr]

Looks good to me.

[Qottmann]

[sc-58850]

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.69%. Comparing base (9151477) to head (94974a3).
Report is 13 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5578      +/-   ##
==========================================
- Coverage   99.69%   99.69%   -0.01%     
==========================================
  Files         410      412       +2     
  Lines       38278    38269       -9     
==========================================
- Hits        38161    38151      -10     
- Misses        117      118       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[trbromley]

Thanks @albi3ro! A changelog entry would be very appreciated as this is a nice perf improvement.

[trbromley]

Note that this supersedes #5392.

[mudit2812]

Looks good. I'm happy with the tests mostly, but looks like interface testing could be bumped up a bit as codecov is failing on the branches for casting for tensorflow