Need help on upgrade

[archana-elrod]

I'm new to both k8s and RabbitMQ, I wonder if someone can help me with the upgrade issues I'm having.
I've upgraded cluster-operator v1.12.1 to v1.13.0 by executing
kubectl apply -f cluster-operator.yaml
where cluster-operator.yaml is from Release v1.13.0. Next I ran following

kubectl apply -f pdb.yaml
kubectl apply -f rabbitmq.yaml
kubectl rollout status statefulset/rabbitmq-server

pdb.yaml

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: #{Cluster.Name}
  namespace: #{Cluster.Namespace}
spec:
  minAvailable: 4 # the minimum required quorum + 1 (what the rabbitmq prestop hook waits for on shutdown)
  selector:
    matchLabels:
      app.kubernetes.io/name: #{Cluster.Name}

rabbitmq.yaml

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: #{Cluster.Name}
  namespace: #{Cluster.Namespace}
spec:
  override:
    statefulSet:
      spec:
        podManagementPolicy: Parallel
        template:
          spec:
            containers: []
            topologySpreadConstraints:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/name: #{Cluster.Name}
              maxSkew: 2
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: DoNotSchedule
  persistence:
    storage: #{PersistenceStorage.Size}
    storageClassName: #{PersistenceStorage.StorageClassName}
  rabbitmq: #https://www.rabbitmq.com/memory.html#paging for vm_memory_high_watermark
    additionalConfig: |
      cluster_partition_handling = pause_minority
      vm_memory_high_watermark_paging_ratio = 0.9
      vm_memory_high_watermark.relative = 0.8
      disk_free_limit.relative = 1.0
      log.file.level = warning
      default_user = admin
      default_pass = admin
  replicas: #{Replicas.Count}       # num replicas should be at least quorum+2 since rabbitmq prestop hook waits for quorum+1 to be
  resources:                        # ready when shutting down, and must be at least 1 more than PDB's minAvailable or it would be
    limits:                         # unable to drain a node with rabbitmq on it
      cpu: #{Resources.CPU.Limit}
      memory: #{Resources.Memory.Limit}
    requests:
      cpu: #{Resources.CPU.Request}
      memory: #{Resources.Memory.Request}

When I check the rabbitmq-server-# nodes, I see RabbitMQ 3.8.19 and Erlang 24.0.5, which I believe should have been RabbitMQ 3.9.X as per cluster-operator release v.1.12.0. I am clueless on what am I missing such that RabbitMQ server is not upgrading?
My k8 server version is at v1.26.3, please let me know if you need more info and wonder if someone can help me on RabbitMQ server not being bumped up when bumping up cluster-operator.

[mkuratczyk]

Operator upgrades do not automatically upgrade RabbitMQ cluster instances. When you initially deploy a RabbitmqCluster definition with no image property, the Operator will set it to the default image version (a newer Operator version would default to a newer RabbitMQ version - you can check the release notes to know which RabbitMQ version is the default for a given Operator version).

However, if the image is set, either manually or filled in by the Operator during the initial deployment, it will no longer be changed automatically. You can set the image you want and kubectl apply or for a quick test, you can do kubectl edit rmq ... (you will see that the image property is there, even if it wasn't present in the YAML you applied).

Final note: kubectl rollout status statefulset/rabbitmq-server should not be necessary. If you changed something, then kubectl apply will also restart the cluster. If you didn't change anything, then restarting the cluster won't change anything.

[Zerpet]

The cluster operator does not upgrade RabbitMQ server version automatically. You can specify the RabbitMQ image to use in the field .spec.image (see docs). I recommend that you do this manually, or build your own automation to validate and deploy newer RabbitMQ server versions.

The reason to not auto-upgrade RabbitMQ alongside with the cluster operator is because we can't guarantee version upgradability, and some upgrades e.g. 3.10.x -> 3.11.x require some feature flags to be enabled.

Last but not least, you can add an experimental environment variable CONTROL_RABBITMQ_IMAGE to the Cluster Operator manifest, so that cluster operator takes control over the RabbitMQ image, and auto-upgrades the RabbitMQ server. Please note that this option is experimental, and if something breaks, you get to keep all the pieces.