Import candidate: special remote tooling

[mih]

The datalad-next entry point to this code is https://github.com/datalad/datalad-next/tree/main/datalad_next/annexremotes and strong dependencies exist on the annexremote package, and pieces in https://github.com/datalad/datalad/tree/maint/datalad/customremotes: the base class SpecialRemote and the main() entrypoint for all special remotes.

This is a challenging piece (in a good way), because it will bring up a number of questions re encapsulation (how much datalad-logic should be in the resulting implementation?), and it should sharpen the concept of this library going through them.

Moreover, the special remote code in datalad-core is heavily patched already, and consolidating these changes into a reusable implementation would be of high value.

Discussion items:

• Future of the dependency on the DataLad CLI tooling in main()
• Future of the dependencu on the DataLad UI (ui.set_backend('annex')) -- is this even necessary still? can it be implemented within the datalad package via some kind of generic setup-hook mechanism?
• All new code should be fully tested outside a special remote process. The sole testing through git-annex is a major pain point of the current implementation. In addition we should have a toy special remote implementation, only for testing, that can be driven by git-annex as part of the tests (git annex testremote).

[christian-monch]

I agree with everything. Especially a proper test environment for special remote implementation would be a big leap forward.

Below are a few out-of-order thoughts and observations regarding special remotes and general code structure:

• Special remote tooling is a bold choice, I like it!

• You mention "remove dependencies on datalad.ui". This point highlights a general issue in the datalad code base. We often use one class and its subclasses, e.g. ui, to perform operations in different contexts. Different contexts are covered by using instances of other subclasses. This leads to tighter dependencies, for example on the class ui, and to complex class hierarchies and object-instantiation logic. I believe it would be beneficial to apply a more "delegate"-oriented approach, where an optional delegate is given to an object. This would align with the "hook mechanism" that you mentioned.

• A git-annex independent driver for special remote implementations, probably based on datalad_next.shell, might be helpful for high-level testing of special remote implementations. This driver would allow tests that fill a gap between unit-tests and git annex testremote.

[mih]

A git-annex independent driver for special remote implementations, probably based on datalad_next.shell, might be helpful for high-level testing of special remote implementations. This driver would allow tests that fill a gap between unit-tests and git annex testremote.

Ah, that is an excellent idea!!

[yarikoptic]

FWIW we had a flavor of "git-annex independent driver" for testing purposes in the core in the past, last seen around https://github.com/datalad/datalad/blob/0.15.2/datalad/customremotes/tests/test_base.py#L129 . So may be some ideas there could be reincarnated.