[EFM Recovery] Backward Compatibility for DKG Result Submission

[jordanschalm]

Context

We are deploying EFM Recovery as a Protocol HCU, so software changes need to be backward compatible with existing state and upgrade-safe. See Upgrade Plan (Notion) for details.

This issue is for implementing backward-compatibility for the DKG result submission process. The smart contract changes involved modifying the transaction structure to include the DKG index map as an argument. In the current software version, we always use this updated transaction structure. The upgraded software version must be able to handle both kinds of transaction, depending on the protocol version.

Implications of using protocol version to determine behaviour here

Which DKG result submission transaction we must use depends on the current smart contract version not the protocol state version. But we have no way intrinsic to the protocol to determine what that smart contract version is!

So, doing this means that we must coordinate the protocol upgrade and the smart contract upgrade within the same epoch phase. Previously we discussed how implementing backward-compatibility would allow us to decouple the protocol upgrade and the smart contract upgrade; because of the result submission, I think they will need to be coupled.

Links

• Result submission in flow-go
• Result submission transaction in flow-core-contracts

Definition of Done

• Modify Broker to send different transaction depending on protocol version
• I suspect we will need to change the default bootstrapping to use protocol v2 (and provide option to bootstrap with older version), otherwise the epoch integration tests are going to start failing with this change

[AlexHentschel]

summarizing a brief chat with Jordan here for future reference:

• The main challenge is upgrading the large cohort of consensus nodes in less than a week. Its a lot of people and different IT systems involved, which creates a large surface for things to go wrong occasionally, and some consensus participants being late with upgrade attempts or needing to be fixed thereafter. Therefore, we focusing on backwards-compatibility at the node level, so that we can have a rolling upgrade of the node software to a version that already implements the new protocol but still supports the old protocol.
• In comparison, the smart contract upgrade is a lot less risky to be delayed:
  • we will upgrade testnet prior to mainnet and the smart contract upgrade is exactly the same for both networks (except for testnet not needing a multi-sig of the governance committee)
  • We are only upgrading one platform, i.e. the execution state of mainnet for the smart contract upgrade as opposed to nearly 90 systems in case of node upgrades
  • the hum governance committee is drastically smaller compared to the consensus committee, and we have a lot of partners that can help with the necessary multisig
  • we have almost a week to do the smart contract upgrade on mainnet

Hence, we accept the marginal risk of the smart contract upgrade going wrong on mainnet, because this is much less likely to go wrong compared to attempting to upgrade a large fraction of consensus nodes all within a week. Therefore, we think the engineering investment into downwards compatible node software is warranted while extra engineering effort to do a downwards compatible smart contract upgrade is not.