Skip to content

Commit

Permalink
Release Anonymous and Report Fixes (#245)
Browse files Browse the repository at this point in the history
* Adds status banner component

* Adds outlined variant to status banner

* Build storybook files

* Build storybook files

* Builds storybook

* Test Chromatic GitHub Action

* Tests branch

* Fix for chromatic github action

* Removes index.html from feature branch; adds workingdir params to chromatic workflow file

* Adds src to install depency command

* Removes src from run command

* Testing Chromatic regression testing

* Finish up status banner

* Adds status variant to Status Banner. Fixes colors, outline variant styling.

* Tests Chromatic build to feature branch; resizes status banner title as part of test

* Finishing touches to status banner

* Clean up status banner folder/package.json chromatic script command

* Enlargen, bold status banner title for demo

* revert back to original

* add await to signin

* add await to signin

* upd TokenService to use issuer token_endpoint

* add debug for token svc

* finish adding token endpoint to Token service

* add server-side rendering API ROOT

* refactor the keycloak services

* refactor the keycloak services

* refactor the keycloak services

* fix crashing bug with delete access

* refactor the issuer and uma2 services, updates to whitelist

* upd cicd deploy with SSR_API_ROOT

* Add queries to whitelist

* update whitelist

* update whitelist

* upd whitelist and permissions

* upd UI for environment plugins

* upd to env plugins for jwt-keycloak template

* upd whitelist

* better handling of my access requests

* refactor the access mgr and credential admin

* adj credential issuer retrieval

* upd whitelist

* cleanup poc manager pages

* upd auth profile failure messages

* fix error on environment detail

* adj authz matrix

* upd whitelist and upd business profile handling

* upd scope retrieval and consumer list

* fix service account list query

* upd whitelists

* updated data rules with Blob list type to link blob object to Activity

* create connectExclusiveOne function to return one record to connect. Updated data rules to use the new function to connect Blob record to Activity record

* fix the ci remove to remote feeder deployment

* added label list

* First pass at new layout

* Add new confirmation dialog and wire up deleting namespaces

* Handle redirects after delete

* Removes storybook build files

* Layout revisions.

* first shot at a Namespace report

* add icon to export report

* upd docker compose fix

* implement audience client mapper (#219)

Implement audience client mapper

* minor fix for publishing auth profile with client mapper

* add updating unit testing and ghaction for sonar

* add updating unit testing and ghaction for sonar

* upd testing

* upd readme

* upd readme

* upd readme

* upd readme

* upd badges

* upd services and tests for xlsx report

* Fix hardcoded value

* First pass at new layout

* Add new confirmation dialog and wire up deleting namespaces

* Handle redirects after delete

* Layout revisions.

* Fix hardcoded value

* Add new action menu component, expand card component.

* Tidy up menu, applications page.

* Restyle the header.

* Add disclosure rows

* Add custom table component with sorting

* Tidy up table hover, use empty pane.

* Update theme.

* Add footer and remove owner column.

* Update docs, add new application services component

* add hover state, better mock service access

* Tidy up new stories content

* Tidy up and improve accessibility support

* Layout revisions.

* Try removing server side props

* Fix type error

* Fix missing data error

* Add back original code.

* Update graphql queries

* Name query

* Check temp ids

* Update httplocalhost3000admingraphiql-70d2f3.gql

* Add whitespace after

* Update indentation

* Remove end whitespace

* Remove stray log

* Add application services query

* Fix query spacing

* UI tweaks.

* Tidy up ghost button focus/active states

* Minor button tweaks.

* upd reporting

* upd whitelist and services

* Update ci-feat-sonar.yaml

* upd ds api for namespace

* upd unit test and services

* upd unit test and services

* deal with a few code smells

* deal with a few code smells

* Hide delete if user is not api-owner.

* upd report

* fix delete namespace error

* Remove redunant text from Applications

* Remove test text

* draft directory changes

* drop products without dataset

* upd directory typescript error

* seed with dataset for gwa

* seed with dataset for gwa

* seed with dataset for gwa fix

* fix directory api error

* upd services supporting report

* upd kong admin url for feeder feature deploy

* fix services report

* fix formatting on hosts xls

* fix role placement in xls

* Add stubbed out initial layout

* Tidy up UI components to be wired up to data.

* Wire up new api product item component.

* report consumer request

* upd id reference and breadcrumb link

* adj authz matrix for access request page

* resolve consumer requests

* return to page on logout, better auth and new flow type

* return to page on logout, better auth and new flow type

* upd feeder context

* fix error on plugin validation

* fix ability to add pcontrols on pending request

* fix ability to add controls on pending request

* upd formatting and fix build error

* minor linking fix for service access

* upd readme coverage badge

* upd readme coverage badge

* upd build oauth proxy to fix report gen error

* upd authz matrix

* upd fix for report

* upd product item logic

* upd product item

* upd authz matrix for access manager

* upd authz matrix

* Fix bug with report and missing credissuer

* upd identifiers

* small tweak for service access report

* small adj for report

* fix the activity error

* upd yarn lock

* upd package/yarn lock

* cleanup failing build

* upd locks

* Re-render the manage namespace modal when selecting different namespace

* rollbacked react version to 17.0.1, removed redundant declarations and fixed issue with namespace delete dialog

* Update package.json

* fix logic for directory anonymous

* upd link on anonymous access

* upd product edit

* upd plugin validation for active env

* add dataset to matrix for ns manager

* upd plugin check

* Add about json

* remove the consumer metrics tab on report

* move the export-report to a component

Co-authored-by: Justin Tendeck <[email protected]>
Co-authored-by: Nithin Shekar Kuruba <[email protected]>
Co-authored-by: Joshua Jones <[email protected]>
  • Loading branch information
4 people authored Nov 25, 2021
1 parent 6002d43 commit 716693e
Showing 34 changed files with 603 additions and 362 deletions.
3 changes: 2 additions & 1 deletion .env.local
Original file line number Diff line number Diff line change
@@ -23,4 +23,5 @@ EXTERNAL_URL=http://oauth2proxy.localtest.me:4180
OIDC_ISSUER=http://keycloak.localtest.me:9080/auth/realms/master
LOCAL_ENV=true
WORKING_PATH=/tmp
DESTINATION_URL=
DESTINATION_URL=
SSR_API_ROOT=http://apsportal.localtest.me:3000
6 changes: 6 additions & 0 deletions .github/workflows/ci-build-deploy.yaml
Original file line number Diff line number Diff line change
@@ -180,6 +180,12 @@ jobs:
profile-url: ${{ secrets.OIDC_ISSUER }}/protocol/openid-connect/userinfo
insecure-oidc-allow-unverified-email: 'true'
oidc-email-claim: 'sub'
pass-basic-auth: 'false'
pass-access-token: 'true'
set-xauthrequest: 'true'
skip-jwt-bearer-tokens: 'false'
set-authorization-header: 'false'
pass-authorization-header: 'false'
env:
SESSION_SECRET:
2 changes: 1 addition & 1 deletion .github/workflows/ci-build-feeders.yaml
Original file line number Diff line number Diff line change
@@ -129,7 +129,7 @@ jobs:
WORKING_PATH:
value: '/tmp'
KONG_ADMIN_URL:
value: 'http://kong-kong-admin:8001'
value: '${{ secrets.KONG_URL_DEV}}'
CKAN_URL:
value: 'https://catalog.data.gov.bc.ca'
PROM_URL:
34 changes: 34 additions & 0 deletions .github/workflows/ci-feat-sonar.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Sonar Scanner

on:
push:
branches: [dev,feat/*]

env:
REGISTRY: ghcr.io
REGISTRY_USERNAME: ${{ secrets.CONTAINER_REGISTRY_USERNAME }}
REGISTRY_PASSWORD: ${{ secrets.CONTAINER_REGISTRY_PASSWORD }}

jobs:
build:
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v2

- uses: actions/setup-node@v2
with:
node-version: '14'

- name: Run Tests
run: |
cd src
npm i
npm test
- name: SonarCloud Scan
uses: sonarsource/sonarcloud-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
10 changes: 10 additions & 0 deletions .github/workflows/scripts/feeder-init/platform-dataset.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
entity: DraftDataset
record:
id: 'api-gateway-services'
title: 'API Gateway Services'
notes: 'API Gateway Services provide a way to configure services on the API Gateway, manage access to APIs and get insight into the use of them.'
tags: ['gateway', 'kong', 'openapi']
sector: 'Service'
license_title: 'Access Only'
view_audience: 'Government'
security_class: 'LOW-PUBLIC'
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@ record:
id: 748D98F1F56C
name: Gateway Services API
namespace: platform
dataset: api-gateway-services
environments:
- id: FB000000
name: prod
2 changes: 2 additions & 0 deletions .github/workflows/scripts/init.sh
Original file line number Diff line number Diff line change
@@ -2,6 +2,7 @@

python scripts/template.py scripts/feeder-init/legal.yaml legal.yaml
python scripts/template.py scripts/feeder-init/platform-authz-profile.yaml platform-authz-profile.yaml
python scripts/template.py scripts/feeder-init/platform-dataset.yaml platform-dataset.yaml
python scripts/template.py scripts/feeder-init/platform-gwa-api.yaml platform-gwa-api.yaml

while true; do
@@ -15,6 +16,7 @@ while true; do
sleep 5
curl --fail -v http://localhost:8080/push -F [email protected]
curl --fail -v http://localhost:8080/push -F [email protected]
curl --fail -v http://localhost:8080/push -F [email protected]
curl --fail -v http://localhost:8080/push -F [email protected]
kill $FWD_PID
break
89 changes: 5 additions & 84 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# API Services Portal

[![Lifecycle:Maturing](https://img.shields.io/badge/Lifecycle-Maturing-007EC6)](https://github.com/bcgov/repomountie/blob/master/doc/lifecycle-badges.md)
[![Lifecycle:Stable](https://img.shields.io/badge/Lifecycle-Stable-97ca00?style=for-the-badge)](https://github.com/bcgov/repomountie/blob/master/doc/lifecycle-badges.md)
[![GitHub Workflow Status (branch)](https://img.shields.io/github/workflow/status/bcgov/aps-portal/Build%20and%20Deploy/dev?style=for-the-badge)](https://github.com/bcgov/api-services-portal/actions/workflows/ci-build-deploy.yaml)
[![Coverage](https://img.shields.io/sonar/coverage/aps-portal/dev?server=https%3A%2F%2Fsonarcloud.io&style=for-the-badge)](https://sonarcloud.io/summary/new_code?id=aps-portal)
![GitHub](https://img.shields.io/github/license/bcgov/aps-portal?style=for-the-badge)
![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/bcgov/aps-portal?label=release&style=for-the-badge)

## Introduction

@@ -140,89 +144,6 @@ Currently support feeders:

Source: `feeds`

## User Journeys

Roles:

- **API Owner**: Does the technical deployment of the API on the Gateway under a particular Namespace - Gateway Services.
- **Developer**: A Developer discovers APIs, requests access if required and consumes them.
- **Credential Admin**: Application for authenticating with an OIDC Auth provider for the purposes of client registration. The Credential Issuer will generate the new credentials and provide a mechanism for the Developer to retrieve them.
- **API Manager**: The API Manager makes APIs available for consumption with supporting documentation. They approve requests for access.
- **Pilot Tester**: This role enables features that are still being reviewed and not quite ready for broader use.

Typical Flow:

- API Owner: Technical deployment to the Gateway by the API Owner
- API Owner: Create a new DataSet (API) (belongs to a namespace) which can have one or more Gateway Services. Choose the Credential Issuer and the Policies that will be enforced on the gateway for the Gateway Services. A Gateway Service can only belong to one DataSet. A DataSet belongs to one Organization Unit. Only certain Gateway Services will be part of a DataSet (i.e./ only Prod Services).
- API Owner: Publish DataSet to BC Data Catalog or keep private
- API Manager: If private, an API Manager can create an Access Request on behalf of a user - choose DataSet, enter Email; notification sent to Developer
- Developer: If public, a Developer can login, view available APIs and Request Access, Open Console, Launch
- API Manager approves Access Request; Credential Issuer Admin will be notified there is a pending approval request
- Credential Admin will view the Access Request, will authenticate with the Credential Provider and do Client Registration (or create a Registration Token) and update the Access Request with a temporary Token (or contact the Client), marking Access Request as Reg Token Generated or Complete if the Developer got the Token (Automated, Semi-automated, Manual)
- Developer: Go to Pending Requests and Generate Token, where the Developer will make a note of the Credential, Access Request marked Completed
- Developer: Go to Access APIs to view the Gateway Services that the user has access to (public or with Applications). Have available: Open Console, Launch App

Sandbox vs Production for same DataSet. Credential Issuer can be different for different Gateway Services (ie./ dev/test/prod). Should we introduce environment?

Gateway Services will have attributes that are read only and some editable by the API Owner.

(Import from Kong could backfill the Gateway Services, Plugins, Credential Issuer)

Other capabilities: API Manager to Revoke Access.

The Credential Issuer will be displayed as a Type: Public, OIDC, API Key. If OIDC it will have the Discovery URL and the Client ID.

| Entity | Credential Admin | API Manager | API Owner | Developer | APS Platform |
| ------------- | ------------------------------------- | ----------------------------- | --------- | --------- | ------------ |
| AccessRequest | If Approved then Edit only Cred field | If Pending then Mark Approved |

AccessRequest - most complicated as different people can edit at different times
Consumer (Kong) **
Content - TBD
CredentialIssuer editable by Owner or Admin only.
Dataset (BCDC) **
DatasetGroup only editable by API Owner - Admin of Namespace
Gateway (Keycloak) **
Group (Keycloak) **
Organization (BCDC) **
OrganizationUnit (BCDC) **
Plugin (Kong) **
ServiceRoute (Kong) (most fields - update with DataSet, CredentialIssuer) **
TemporaryIdentity (created at login so that it can be used in the KeystoneJS Authorization model)

\*\* Read Only because they are generated from other systems.

AccessRequest: Pending -> Approved -> Issued -> Complete; at any time it can be Cancelled

CredentialIssuer can be for an API Key as well - in this case the CredentialAdmin will generate the API Key and email it to the Deveoper after the API Owner approves.

Emails: AccessRequest flow (Pending Approval, Pending Creation, Credentials Issued - Body of message)

AccessRequest can have credentials issued in three ways:

- Manual - credentials are created and passed manually by the Credential Admin to the Developer
- Semi-automatic - similar to Automatic except that the Credential Admin can review before sending the Email
- Automatic (one or two step) - the CredentialIssuer uses the supplied Client ID/Client Secret (or Token) to mint new Credentials (one step) or Token to generate new Client Credentials (two step).

Maybe better as:

- One step - credentials are passed on via email to the user (or can be "picked up" in the Developer Portal)
- Two step - temporary token saved and an email is sent to tell user credentials are approved and ready to generate

Generating the credentials (manual, using client registration service, using temporary registration token)
Communicating the credentials to the Developer (manually add creds in email - creds not persisted, have developer pick them up on portal)

CredentialIssuer -> AuthenticationPolicy / SecurityPolicy

Have code that:

- Calls API
- Loops through each (in batch) and does a search for the IDs; calculate hash; if diff do update, if missing do insert.
- Insertions should add standard addt'l fields that relate to the SystemOfRecord
- Get full list of IDs, remove ones that were retrieved from API and do a deletion (full sync).

Update gwa-api to return the Keycloak details, Kong details and BCDC details. For now just go direct for Kong and BCDC.

## Development

#### TypeScript
6 changes: 3 additions & 3 deletions feeds/prometheus/index.js
Original file line number Diff line number Diff line change
@@ -31,9 +31,9 @@ const queries = [
},
{
query:
'sum(increase(konglog_service_consumer_counter[1d])) by (consumer,service)',
step: 60 * 60 * 24,
id: 'konglog_service_consumer_daily',
'sum(increase(konglog_service_consumer_counter[60m])) by (consumer,service)',
step: 60 * 60,
id: 'konglog_service_consumer_hourly',
},
];

9 changes: 4 additions & 5 deletions oauth2-proxy/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@


# Oauth Proxy

```
export COOKIE_SECRET=""
export OIDC_CLIENT_ID=""
@@ -29,9 +28,9 @@ docker run -ti --rm --name proxy -p 4180:4180 \
--pass-access-token=true \
--set-xauthrequest=true \
--skip-jwt-bearer-tokens=false \
--set-authorization-header=true \
--pass-authorization-header=true \
--skip-auth-regex="/home|/public|/docs|/_next|/images|/devportal|/manager|/signout" \
--set-authorization-header=false \
--pass-authorization-header=false \
--skip-auth-regex="/home|/public|/docs|/_next|/images|/devportal|/manager|/ds/api|/signout" \
--whitelist-domain="${OIDC_ISSUER_HOSTNAME}" \
--upstream="http://${hostip}:3000"
```
9 changes: 5 additions & 4 deletions sonar-project.properties
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
sonar.organization=bcgov-oss
sonar.projectKey=aps-portal

# relative paths to source directories. More details and properties are described
# in https://sonarcloud.io/documentation/project-administration/narrowing-the-focus/
sonar.sources=aps-portal
sonar.host.url=https://sonarcloud.io

sonar.sources=src/auth,src/authz,src/batch,src/services

sonar.javascript.lcov.reportPaths=./src/__coverage__/lcov.info

3 changes: 2 additions & 1 deletion src/.eslintrc.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
module.exports = {
root: true,
parser: '@typescript-eslint/parser',
plugins: ['@typescript-eslint', 'react', 'jest'],
plugins: ['@typescript-eslint', 'react', 'jest', 'jsx-a11y'],
env: {
commonjs: true,
es6: true,
@@ -15,6 +15,7 @@ module.exports = {
'plugin:jest/recommended',
'plugin:react/recommended',
'plugin:react-hooks/recommended',
'plugin:jsx-a11y/recommended',
],
settings: {
react: {
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@

query GET($search: String!, $first: int) {
allDatasets(search: $search, first: $first) {
id
name
title
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@

query GET {
allProductsByNamespace {
id
name
description
organization {
id
title
}
organizationUnit {
id
title
}
dataset {
id
name
title
notes
sector
license_title
}
environments {
id
name
active
flow
services {
id
name
host
}
credentialIssuer {
name
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@

query GET($search: String!, $first: Int) {
allDatasets(search: $search, first: $first) {
id
name
title
}
}
1 change: 1 addition & 0 deletions src/authz/matrix.csv
Original file line number Diff line number Diff line change
@@ -32,6 +32,7 @@ CREDENTIAL ADMIN,,,CredentialIssuer,,"create,read",namespace,,,credential-admin,
CREDENTIAL ADMIN,,OwnedCredentialIssuer,,,,,,,credential-admin,,,allow,
CREDENTIAL ADMIN,,allCredentialIssuersByNamespace,,,,,,,credential-admin,,,allow,filterByUserNS
CREDENTIAL ADMIN,,,User,read,,,,,credential-admin,,,allow,
NS MANAGER,,,Dataset,,"create,update,delete",,,,api-owner,,,allow,
NS MANAGER,,getPermissionTicketsForResource,,,,,,,api-owner,,,allow,
NS MANAGER,,getUmaPoliciesByResourceName,,,,,,,api-owner,,,allow,
NS MANAGER,,getResourceOwners,,,,,,,api-owner,,,allow,
17 changes: 14 additions & 3 deletions src/controllers/DirectoryController.ts
Original file line number Diff line number Diff line change
@@ -40,9 +40,9 @@ function transformSetAnonymous(products: Product[]) {
products.forEach((prod) => {
prod.environments.forEach((env) => {
env.services.forEach((svc) => {
svc.plugins &&
svc.plugins
.filter(
function setAnonymousIfApplicable(plugins: any[]) {
plugins
?.filter(
(plugin) =>
plugin.name == 'key-auth' || plugin.name == 'jwt-keycloak'
)
@@ -52,6 +52,11 @@ function transformSetAnonymous(products: Product[]) {
(env as any).anonymous = true;
}
});
}
setAnonymousIfApplicable(svc.plugins);
svc.routes?.forEach((route) => {
setAnonymousIfApplicable(route.plugins);
});
});
});
});
@@ -124,6 +129,12 @@ const item = gql`
name
config
}
routes {
plugins {
name
config
}
}
}
}
dataset {
4 changes: 3 additions & 1 deletion src/nextapp/components/api-product-item/api-product-item.tsx
Original file line number Diff line number Diff line change
@@ -90,7 +90,9 @@ const ApiProductItem: React.FC<ApiProductItemProps> = ({ data, id }) => {
<Text ml={8} fontSize="sm">
For elevated access, please{' '}
<NextLink passHref href={accessLink}>
<Link>Request Access</Link>
<Link color="bc-link" textDecor="underline">
Request Access
</Link>
</NextLink>
</Text>
</GridItem>
3 changes: 2 additions & 1 deletion src/nextapp/components/namespace-delete/namespace-delete.tsx
Original file line number Diff line number Diff line change
@@ -56,9 +56,10 @@ const NamespaceDelete: React.FC<NamespaceDeleteProps> = ({
});
}
}, [client, deleteMutation, name, onClose, router, toast, user.namespace]);

const handleCancel = React.useCallback(() => onCancel(), [onCancel]);

React.useEffect(() => onOpen(), [onOpen]);
React.useEffect(() => onOpen(), [name]);

return (
<>
Loading

0 comments on commit 716693e

Please sign in to comment.