[Upgrades] Document/address issues found during v0.0.11 upgrade on Alpha TestNet

[okdas]

Objective

Address issues and future-proof the network upgrade process to prevent panics and data incompatibility that were encountered during the v0.0.11 upgrade.

Origin Document

https://www.notion.so/buildwithgrove/Alpha-TestNet-v0-0-11-Postmortem-152a36edfff680d58411c8fa384f877c?pvs=4

Goals

• Ensure that future upgrades do not cause runtime panics due to missing parameters or breaking protobuf changes.
• Enhance configuration and upgrade planning to handle missing app.toml parameters gracefully.
• Strengthen the upgrade process to handle protobuf field changes more robustly.
• Improve cosmovisor instructions to facilitate rollback if upgrades fail.

Deliverables

• Configuration Propagation:
  Ensure default custom app.toml values are propagated automatically.

  • Add default values for poktroll.telemetry.
  • Change the pattern so newly added config fields do not cause panics if absent.

• Binary SHA Inclusion Decision:
  Exclude SHAs from upgrade transactions, but keep that option available.

  • Document pros and cons (security vs. ability to replace binaries mid-upgrade).
  • Default to NO SHAs for now.

• Upgrade Handler Practices:
  Avoid performing GetParams in the upgrade handler where protobuf structures might have changed.

  • Explicitly set all parameters in code during upgrade to avoid decoding unknown fields, unless we have to get existing parameters.
  • Update code comments/examples/references and documentation to reflect this best practice.

• Cosmovisor Rollback Instructions:
  Improve documentation and instructions for operators to handle failed upgrades.

  • Instruct how to remove the new version and change symlinks back to the old binary.
  • Instruct how to remove upgrade-info.json to prevent automatic re-upgrade attempts.

Non-goals / Non-deliverables

• Re-running the v0.0.11 upgrade on the current testnet (already failed, rollback handled separately).

General deliverables

• Comments: Add TODOs and comments in code to guide future developers.
• Testing: Add unit/E2E tests for upgrade scenarios, especially for config and protobuf changes.
• Documentation: Update developer and operator docs to reflect new practices, including code and configuration examples.

---

Creator: @okdas
Co-owners: @bryanchriswhite @Olshansk