Build time A/B testing support

[mockitoguy]

Use case

Run build once, make some change, run build again, then compare build results. Sounds familiar? For certain kinds of changes, doing such A/B testing is super useful. It would be an excellent addition to release tools. After all, we want to really improve the lifecycle of software consumers and producers (easier, safer version bumps!). Example use cases:

• bumping version of a dependency and validating if the project still works ok (Automated e2e testing and pushing version upgrades to consumers #85). Bumped version can be of some compile dependency but also of a built-time dependency such as the version of a some Gradle plugin we use.
• making change in build logic and validating if stuff still works.

Example workflow

Example implementation of the build comparison workflow:

1. Clone a project from GitHub
2. Copy the clone to another directory so that we have 2 identical clones
3. Run a set of commands in clone 1, for example "./gradlew build -PsomeToolVersion=1.4"
4. Run a set of commands in clone 2, for example "./gradlew build -PsomeToolVersion=1.5"
5. Compare build directories
6. Analyze comparison results

Suggested design

Suggested starting point design - check out #114 (It's only a suggestion! Pick your own path if you want, just keep the team informed about the design decisions! :)

[mockitoguy]

@epeee, is this something you want to take a look? :) It's a very useful feature, not tangled with the internals of the project, and offers interesting challenge for you!

[epeee]

@szczepiq yes, I will take a look at this one. sounds really useful, great idea!

[mockitoguy]

@epeee, that's great! This is a big ticket, if you have any design / thoughts before / during implementation, let us know. THANKS!

[mockitoguy]

@epeee, do you think you will be able to work on this feature? No pressure - just asking!

[epeee]

@szczepiq yes, I will be able to work on this one (already started working on it). Sadly, I was more busy than usual during the last days.

[mockitoguy]

Thank you for confirmation! This makes it easier to plan our roadmap :)

[mockitoguy]

@epeee, I totally understand it's hard to find time for OSS at times! Do you mind if we hand this epic over to a different contributor? You can reengage whenever you have availability, just let us know and we will reorganize the team :)

[epeee]

@szczepiq sorry for the delay. What about reassigning it if I'm not able to come up with a solution on Fr (it looks like I will have some time to work on this one tomorrow)?

[mockitoguy]

Sounds good!

[epeee]

a quick update: I made quite some progress yesterday on the compareAB task and it looks like I will be able to push a very first version of it today.
cloneGitHubRepository, runA and runB tasks are already available in the abTesting branch.

[epeee]

I just pushed the compareAB task including serialization. Tests are still missing but I will work on them now.
Apart from that the only part which is missing now is the analyze part. It would be great if we could have some kind of discussion/brainstorming about how the analyze result could look like (in order to not go into a wrong direction).

[mockitoguy]

Hey @epeee, let's deprioritize this ticket for now and revisit when e2e testing is complete after 1.0. I'm finding it hard to focus on this at the moment. In the meantime, I've added 2 more "please contribute" tickets that would be really useful for 1.0 release!