RDS tags inconsistent with the resource created in the cloud during composition update

[kelvinwijaya]

What happened?

RDS User defined tags is not tally with the one created in the cloud

How can we reproduce it?

1. Existing RDSInstance objects has been created with some tags
2. Change the composition spec. Adding new tags there
3. RDSInstance objects managed to get the updated value from the composition file and status show as SYNC true
4. Verify in RDS Console, some RDS Instance still not reflecting the newly added tags (not every instances has this issue or intermittent), only 1 out of 5 RDS get proper Tags update

What environment did it happen in?

Crossplane version: 1.0.0-rc.0.26.gd8254833

• Cloud provider or hardware configuration: AWS
• Kubernetes version (use kubectl version)

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.8-eks-e16311", GitCommit:"e163110a04dcb2f39c3325af96d019b492541
9eb", GitTreeState:"clean", BuildDate:"2020-03-27T22:40:13Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95
ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes distribution (e.g. Tectonic, GKE, OpenShift) : EKS

Suspect:

Referencing the External Tags logic in the Initialize phase which includes the Tags sorting: https://github.com/crossplane/provider-aws/blob/cd5da906f3f1fc8e259d78e9351584e6e4cfddd2/pkg/controller/database/rdsinstance.go#L285

But not during the Update phase, patching of the Tags may required some kind of sorting before calling the : https://github.com/crossplane/provider-aws/blob/cd5da906f3f1fc8e259d78e9351584e6e4cfddd2/pkg/controller/database/rdsinstance.go#L223

[negz]

Thanks for raising this bug @kelvinwijaya! I've proposed that we work on this in the Crossplane 1.1 release (so during the next few weeks). If anyone is interested on fixing this sooner, please let us know!

[eloo]

@negz @muvaf is there any progress here?
we have discovered that this bug is still open and unfixed :(

[nicolasbelanger]

Got burned by this just now... tried to add tags on these crds:

dbinstance.rds.aws.crossplane.io
dbparametergroup.rds.aws.crossplane.io
dbsubnetgroup.database.aws.crossplane.io
securitygroup.ec2.aws.crossplane.io

It worked only on:

dbsubnetgroup.database.aws.crossplane.io
securitygroup.ec2.aws.crossplane.io

[bjeevan-ib]

While this issue is open, is there any workaround for dbinstance.rds.aws.crossplane.io tag update?

[fabioaraujopt]

Can you please investigate this?

[MisterMX]

It seems that RDSInstance does handle different tag orders correctly and some resources in rds.aws.crossplane.io are missing an implementation for tag comparison and update.