Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ID-897 filter resources endpoint #1236

Merged
merged 9 commits into from
Oct 31, 2023
Merged

Conversation

tlangs
Copy link
Contributor

@tlangs tlangs commented Oct 31, 2023

Ticket: https://broadworkbench.atlassian.net/browse/ID-897

GET /api/resources/v2/{resource_type_name} is just too darn slow. The DB query to generate the response is one of the most expensive queries in Sam, and it recently caused us to DDoS ourselves when a change was made in Terra-UI that ultimately hit that endpoint millions of times.

The current endpoint returns all the resources of a resource type that the calling user has access to, including direct and inherited roles and actions. It also returns all public resources of that type. The query backing this runs extraordinarily slowly, often .5-1 seconds, and takes a lot of DB CPU resources.

I’m adding a new endpoint: GET /api/resources/v3/filter?resourceTypes=resourceType1,resourceType2&policies=policyName1,policyName2&roles=roleName1,roleName2&actions=actionName1,actionName2&includePublic=false

This “filter” query runs an order-of-magnitude faster through some specific optimizations. The filter parameters are all lists (except includePublic). Results will be the intersection of all filters, excluding empty ones. For example:

  • GET /api/resources/v3/filter?resourceTypes=workspace would get all the workspaces available to the user, and include all the policies, roles, and actions they have on those workspaces.

  • GET /api/resources/v3/filter?resourceTypes=workspace&roles=reader,writer would get all workspaces for the user where they are in either the reader OR the writer role.

The response will look like this:

{
	"resources": [
		{
			"resourceType": "workspace",
			"resourceName": "theName"
			"isPublic": Boolean,
			"policies": ["policyName1", "policyName2", ...],
			"roles": ["roleName1", "roleName2", ...],
			"actions": ["actionName1", "actionName2", ...],
		},
		...
	]
}

In the initial implementation, getting public resources for a user through this query will remain unimplemented, and throw an exception if includePublic=true is set in the request. Some special attention needs to be paid to how we get public resources for a user, since that actually makes up the bulk of the query latency in /api/resource/v2/{resource_type_name}. Ended up implementing it, and its 4x faster than the list resources endpoint

This endpoint is extremely useful for services like Leo, which only want to get the resources for a user for which they have a specific type of access. Currently, Leo is forced to get all resources of a type for a user and filter client-side, but this query will let Leo ask Sam only for what it needs. Efficiencies!

In initial testing, the DB query supporting the filtering runs 10x faster than the standard list resources query!


PR checklist

  • I've followed the instructions if I've made any changes to the API, especially if they're breaking changes
  • I've filled out the Security Risk Assessment (requires Broad Internal network access) and attached the result to the JIRA ticket

@tlangs
Copy link
Contributor Author

tlangs commented Oct 31, 2023

I'm going to change this from v3 to v2 endpoint version

Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
7.2% 7.2% Duplication

@tlangs tlangs merged commit a3a3622 into develop Oct 31, 2023
22 checks passed
@tlangs tlangs deleted the tl_ID-897_filter_resources_endpoint branch October 31, 2023 21:45
Copy link

@mlilynolting mlilynolting left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants