CP-50475: parallelize device ops during VM lifecycle ops

[psafont]

Operations on different devices should be independent and therefore can be
parallelized. This both means parallelizing operations on different device
types and on devices for the same type.

An atom to serialize action has been introduced because the operation regarding
a single device must be kept serialized.

Also removes some unneeded parallel atoms, which cause some overhead as well as polluting the traces

This makes VM_starts to run about ~1 second faster on some tests where VMs have 5 VIFs. I'm testing it further, but it doesn't look like it's breaking any tests. I want to push it through some bootstorms and measure improvements

Please do suggest some further parallelization if you can find it!

BVT+BST are all green

[robhoes]

I have to look at the details of the PR, but the first thing I wondered is why we need to introduce a new Serial atomic operation, as clearly we are already able to serialise. The list returned by atomics_of_operation is implicitly serial and executed as such by perform_atomics. So now we have two ways of expression serial (atomic) operations.

I can see that nesting is easier with an explicit Serial atomic. Should we then as a next step make all serialisation explicit by turning atomics_of_operation into atomic_of_operation and wrapping all lists with Serial? Then we can merge perform_atomics and perform_atomics as well.

[psafont]

I introduced the serial atomic because it was convenient. At some point I also added a nop one, which removes the need to concatenate lists quite a bit, but I'd rather do the latter than the former, as it could end up in the traces.

The list returned by atomics_of_operation is implicitly serial and executed as such by perform_atomics. So now we have two ways of expression serial (atomic) operations.

I can see that nesting is easier with an explicit Serial atomic.

An option to avoid the serial atom would be to make the parallel one to receive a list of lists of atomics ops, instead of a list of atomic ops, although I'm not fond of the implicitness of it.

Should we then as a next step make all serialisation explicit by turning atomics_of_operation into atomic_of_operation and wrapping all lists with Serial? Then we can merge perform_atomics and perform_atomics as well.

I can try :)

[psafont]

Trying is a good exercise of yak-shaving: there's code to calculate progress of tasks. It's not clear to me that it works well currently, especially with regard to the parallel atomic op; and merging the functions needs to track progress into the Serial atomic op. The code needs unit-testing to be able to make it's not made even worse and the results are predictable.

The benefits of the current patch can be seen in the traces

Now the critical path doesn't contain the VIFs, and now it's dominated even more by the SM and the need to plug RW VBDs before RO ones

[minglumlu]

I thought there might be some dependencies between these operations. It's great to see that they actually can be performed in parallel like this. If the returned lists can be encoded with Parallel and Serial as well, as Rob suggested, the data structure would be an exact representation on the dependencies. I.E. the qemu and devices depending on it could be handled in parallel with other devices not-depending on qemu as well.

[lindig]

Can we merge this and it was only held back because we wanted to get some fixes in first?

[psafont]

This was held back because of the fixes as well as some performance tests being run. It's taking really long to get them scheduled :(

[psafont]

This can be merged whenever we feel like it, the performance tests won't be able to be run in weeks, so we will need to run benchmarks when this is merged to master