[FEATURE] Limit Roles Created/Needed & Provide Default Deployment Role

[a13zen]

Is your feature request related to a problem? Please describe.
Currently, each module creates its own deployment role. If a large team of developers deploy many namespaces, each with many modules, they start to hit the IAM Role account limits within their accounts.

Describe the solution you'd like

1. I would like to avoid hitting the IAM Role account limits. To do so, I would need seedfarmer to create/need less roles. One way to achieve this is by have a "default" module deployment role. If I do not specify a modulestack.yaml, then use the default deployment role. If I do need module specific permissions and create the modulestack.yaml, then ensure to create a module specific role. This would allow me to drastically reduce the number of roles I need.
2. Ensure old/unused roles are cleaned up. This can be done by ensuring that module deployment roles for namespaces that don't exist any more are properly deleted and cleaned up. Ideally there would be a CLI command to allow cleaning up such roles.
3. Support metadata export/import across deployments. If users can import/export metadata across deployments, they can have shared resources such as EKS/Airflow/Buckets etc that don't need to be namespace specific in one deployment, and then they can import and use the shared resources in their namespace deployment. This would also allow reducing the overall Role footprint as users don't need to deploy copies of the shared resources.

Additional context
Current customers are hitting 5000 Role limits due to this already. These are hard, unadjustable limited from IAM.

[JunjieTang-D1]

endorse this feature to help the customers to scale

[dgraeber]

@a13zen Thanks for this FR request. I will address each item as you have ordered them.

1. IAM role limits -- I have also thought of this (if no modulestack.yaml, use a generic role) and agreed with this idea. The IAM role limit was raised in this ISSUE previously. But without any empirical data or anecdotal data, I could not deduce whether the proposed solution would be a benefit. It sounds like you @a13zen are vouching that a generic role for non-modulestack deployments would be a benefit, so we will begin vetting just to make sure there are no underlying issues that we are not seeing...but i agree with your proposal.

2. SF deletes roles upon a successful destroy of a module. I think you have lingering roles from failed destroys(???) . I cannot accommodate your request here: If the namespace does not exist, the manifest in SSM does not exist, thus we cannot create a deployment session to the account/region mapping to destroy any roles. Since SF looks across account/region mappings and IAM roles are global, SF assures that the proper role is leveraged for each module. Perhaps you could write a script that can determine what roles are not used (based on non-existent namespaces?) and use AWS credentials to execute the removal...but SF cannot do this. @a13zen I am sure we will discuss this in detail and come up with a viable solution, but your current suggestion cannot be met here.

3. This request MUST be in a different FR. We currently have a POC working where we can pass in metadata to deploy a single module for development efforts. The problem with sharing metadata across deployments is that the dependency management assurances do not exist. Dependency management is currently scoped to single deployments. For example: I have FoundationDeployment (FD), Deployment1 (D1), Deployment2 (D2). D1 and D2 reference FD metadata and are dependent on it. There is nothing (currently) stopping anyone from deleting FD....in effect hosing D1 and D2 at the same time. We will not implement metadata sharing until we determine the proper means of dependency management across accounts/regions and deployments in a project (in effect, we are scoping dependency management to the project level). There is a large LOE associated with that change - thus this item needs to be in its own FR. We will not address it as we look at the generic IAM role request.

We will move forward with vetting the generic IAM role usage for non modulestack deployments.

[a13zen]

Thanks for the feedback @dgraeber. I think it makes sense. Can open a different FR for 3. For point 2, if you can provide some guidance to our customers on how to find such roles it would be fantastic.

For 1. Let's do it.

FR for 3 created #668.

[dgraeber]

@a13zen Can do!

[kukushking]

@a13zen @JunjieTang-D1 the related PR #681 is merged and available in 5.0.0.