DBCluster masterUserPasswordSecretRef required, even when AWS can't take it (when replicating)

[johnsmclay]

What happened?

When creating a DBCluster.rds.aws.crossplane.io/v1alpha1 object, the value replicationSourceIdentifier can be set to an ARN like arn:aws:rds:us-west-2:123456789:db:funtimesdb to have the new Aurora cluster replicate from that specified instance. When you do, AWS will not accept a masterUsername, a master password, or databaseName because all of that information will come from the master the new cluster will replicate from.
The issue is that on the DBCluster object, masterUserPasswordSecretRef is always required, even in the case of a valid entry in replicationSourceIdentifier causing AWS to reject the request because it contains a password.
My current workaround is to specify a real secret in masterUserPasswordSecretRef, but in it to specify a key that does not exist:

masterUserPasswordSecretRef:
      namespace: crossplane
      name: newclustertst-aurora
      key: thisisnotreal

When I do so, the arbitrary requirement for masterUserPasswordSecretRef is satisfied, but because it can't find the value in that secret crossplane either doesn't specify the secret in the request or it is null, therefore AWS accepts the request.

The other, related issue is that of password generation on a new cluster. Unlike DBInstance.rds.aws.crossplane.io/v1alpha1 with it's nullable MasterUserPasswordSecretRef and the boolean autogeneratePassword and RDSInstance.database.aws.crossplane.io/v1beta1 with it's nullable MasterUserPasswordSecretRef, on DBCluster you MUST specify the password in an existing secret when creating the cluster. It would be great if DBCluster had the same nullable MasterUserPasswordSecretRef and boolean autogeneratePassword combo as DBInstance so that it fixes the main issue of this bug report AND allows for auto password generation making DBCluster behavior consistent with DBInstance (and moreor less consistent with RDSInstance.

We are upgrading Crossplane and the provider to the newest version to help with another issue (#895) but I don't think it will help here as this is documented even in the newer versions.

How can we reproduce it?

Apply this secret and DBCluster to get the AWS rejection message (same error as when you unnecessarily supply masterUsername or databaseName):
NOTE: replicationSourceIdentifier must point to a working and compatible database instance.

apiVersion: v1
kind: Secret
metadata:
  name: newclustertst-aurora
  namespace: crossplane
stringData:
  password: testPassword!123
---
apiVersion: rds.aws.crossplane.io/v1alpha1
kind: DBCluster
metadata:
  name: newclustertst-aurora
spec:
  forProvider:
    region: us-west-2
    engine: aurora-postgresql
    engineVersion: "13.4"
    replicationSourceIdentifier: arn:aws:rds:us-west-2:123456789:db:funtimesdb
    #masterUsername: dbadmin
    #databaseName: newclustertst
    masterUserPasswordSecretRef:
      namespace: crossplane
      name: newclustertst-aurora
      key: password
      #key: thisisnotreal
    backupRetentionPeriod: 14
    enableIAMDatabaseAuthentication: true
    preferredBackupWindow: "7:00-8:00"
    storageEncrypted: true
    dbSubnetGroupName: "sandbox_us-west-2"
  providerConfigRef:
    name: aws-provider

To get the error from Crossplane, comment out "masterUserPasswordSecretRef" or specify a secret that doesn't exist and it will either tell you masterUserPasswordSecretRef is required or that the one specified doesn't exist.
To get it to work, comment key: password and uncomment #key: thisisnotreal.

What environment did it happen in?

Crossplane version: 1.0.0
Provider version: 0.20.0 (but according to the documentation, this is still the case with 0.25.0
Kubernetes distro: AWS EKS
Kubernetes version: 1.20.11
kubectl version:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:33:37Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-eks-f17b81", GitCommit:"f17b810c9e5a82200d28b6210b458497ddfcf31b", GitTreeState:"clean", BuildDate:"2021-10-15T21:46:21Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

[haarchri]

can you try provider-aws version 0.25 ? we running aurora cluster without problems
please check before for breaking changes between 0.20 and 0.25 !!

[johnsmclay]

As I said:

We are upgrading Crossplane and the provider to the newest version to help with another issue (#895) but I don't think it will help here as this is documented even in the newer versions.

So, we plan to try it on the 0.25 soon. But, the issue isn't "running aurora cluster", it is specifically starting a new one with the option "replicationSourceIdentifier" specified. In that case, AWS will not accept a password value, but the provider (even in 0.25.0 still requires it:

Essentially, we are setting these new clusters to replicate from the old non-aurora instances, cutting over, and then killing the old instances as AWS suggests for migrating from vanilla to Aurora.

[haarchri]

okay got it think we can change this here https://github.com/crossplane/provider-aws/blob/master/apis/rds/v1alpha1/custom_types.go#L60 adding optional/,omitempty and check the controller here https://github.com/crossplane/provider-aws/blob/master/pkg/controller/rds/dbcluster/setup.go#L91

[haarchri]

#1413 fixed this issue