Upgrade Guide
Major version can only be upgraded incrementally from immediate previous major version, i.e. from N to N+1.
v1 to v2
Upgrading NotifyBC from v1 to v2 involves two steps
Update your client code
NotifyBC v2 introduced backward incompatible API changes documented in the rest of this section. If your client code will be impacted by the changes, update your code to address the incompatibility first.
Query parameter array syntax
In v1 array can be specified in query parameter using two formats
- by enclosing array elements in square brackets such as
&additionalServices=["s1","s2]in one query parameter - by repeating the query parameters, for example
&additionalServices=s1&additionalServices=s2
In v2 only the latter format is supported.
Date-Time fields
In v1 date-time fields can be specified in date-only string such as 2020-01-01. In v2 the field must be specified in ISO 8601 extended format such as 2020-01-01T00:00:00Z.
Return status codes
HTTP response code of success calls to following APIs are changed from 200 to 204
',15),m=e("h4",{id:"administrator-api",tabindex:"-1"},[e("a",{class:"header-anchor",href:"#administrator-api","aria-hidden":"true"},"#"),a(" Administrator API")],-1),v=e("em",null,"Administrator",-1),h=e("em",null,"UserCredential",-1),b=o(`Upgrade NotifyBC server
The procedure to upgrade from v1 to v2 depends on how v1 was installed.
Source-code Installation
- Stop NotifyBC
- Backup app root and database!
- Make sure current branch is tracking correct remote branch
git remote set-url origin https://github.com/bcgov/NotifyBC.git -git branch -u origin/main - - Make a note of any extra packages added to package.json
- Run
git pull && git checkout tags/v2.x.x -b <branch_name>from app root, replace v2.x.x with a v2 release, preferably latest, found in GitHub such as v2.9.0. - Make sure version property in package.json is 2.x.x
- Add back extra packages noted in step 4
- Move server/config.(local|dev|production).(js|json) to src/ if exists
- Move server/datasources.(local|dev|production).(js|json) to src/datasources/db.datasource.(local|dev|production).(js|json) if exists. Notice the file name has changed.
- Move server/middleware.*.(js|json) to src/ if exists. Reorganize top level properties to all or apiOnly, where all applies to all requests including web console and apiOnly applies to API requests only. For example, given
module.exports = {
- initial: {
- compression: {},
- },
- 'routes:before': {
- morgan: {
- enabled: false,
- },
- },
-};
-if compression middleware will be applied to all requests and morgan will be applied to API requests only, then change the file to
module.exports = {
- all: {
- compression: {},
- },
- apiOnly: {
- morgan: {
- enabled: false,
- },
- },
-};
-- Run
yarn install && yarn build
-- Start server by running
yarn startor Windows Service
OpenShift Installation
`,11),g=o(`Run
git clone https://github.com/bcgov/NotifyBC.git
-cd NotifyBC
-Run
oc delete bc/notify-bc
-oc process -f .openshift-templates/notify-bc-build.yml | oc create -f-
-ignore AlreadyExists errors
run
oc project <yourprojectname-<env>>
-oc delete dc/notify-bc-app dc/notify-bc-cron
-oc process -f .openshift-templates/notify-bc.yml | oc create -f-
-ignore AlreadyExists errors
copy value of environment variable MONGODB_USER from mongodb deployment config to the same environment variable of deployment config notify-bc-app and notify-bc-cron, replacing existing value
remove middleware.local.json from configMap notify-bc
add middleware.local.js to configMap notify-bc with following content
module.exports = {
- apiOnly: {
- morgan: {
- enabled: false,
- },
- },
-};
-OpenShift template to Helm
Upgrading NotifyBC on OpenShift created from OpenShift template to Helm involves 2 steps
Customize and install Helm chart
',4),_=e("em",null,"helm/values.local.yaml",-1),x=o(`- notify-bc configMap
- web route host name and certificates
Then run helm install with documented arguments to install a release.
Migrate MongoDB data
backup data from source
oc exec -i <mongodb-pod> -- bash -c 'mongodump -u "$MONGODB_USER" \\ --p "$MONGODB_PASSWORD" -d $MONGODB_DATABASE --gzip --archive' > notify-bc.gz -replace <mongodb-pod> with the mongodb pod name.
restore backup to target
cat notify-bc.gz | oc exec -i <mongodb-pod-0> -- \\ -bash -c 'mongorestore -u "$MONGODB_USERNAME" -p"$MONGODB_PASSWORD" \\ ---uri="mongodb://$K8S_SERVICE_NAME" --db $MONGODB_DATABASE --gzip --drop --archive' -replace <mongodb-pod-0> with the first pod name in the mongodb stateful set.
If both source and target are in the same OpenShift cluster, the two operations can be combined into one
oc exec -i <mongodb-pod> -- bash -c 'mongodump -u "$MONGODB_USER" \\
--p "$MONGODB_PASSWORD" -d $MONGODB_DATABASE --gzip --archive' | \\
-oc exec -i <mongodb-pod-0> -- bash -c \\
-'mongorestore -u "$MONGODB_USERNAME" -p"$MONGODB_PASSWORD" \\
---uri="mongodb://$K8S_SERVICE_NAME" --db $MONGODB_DATABASE --gzip --drop --archive'
-v2 to v3
v3 introduced following backward incompatible changes
- Changed output-only fields failedDispatches and successDispatches to dispatch.failed and dispatch.successful respectively in Notification api. If your client app depends on the fields, change accordingly.
- Changed config notification.logSuccessfulBroadcastDispatches to notification.guaranteedBroadcastPushDispatchProcessing and reversed default value from false to true. If you don't want NotifyBC guarantees processing all subscriptions to a broadcast push notification in a node failure resilient way, perhaps for performance reason, set the value to false in file /src/config.local.js.
After above changes are addressed, upgrading to v3 is as simple as
git pull
-git checkout tags/v3.x.x -b <branch_name>
-yarn install && yarn build
-or, if NotifyBC is deployed to Kubernetes using Helm.
git pull
-git checkout tags/v3.x.x -b <branch_name>
-helm upgrade <release-name> -f helm/platform-specific/<platform>.yaml -f helm/values.local.yaml helm
-Replace v3.x.x with a v3 release, preferably latest, found in GitHub such as v3.1.2.
v3 to v4
v4 introduced following backward incompatible changes that need to be addressed in this order
`,16),q=o("The precedence of config, middleware and datasource files has been changed. Local file takes higher precedence than environment specific file. For example, for config file, the new precedence in ascending order is
- default file /src/config.ts
- environment specific file /src/config.<env>.js, where <env> is determined by environment variable NODE_ENV
- local file /src/config.local.js
To upgrade, if you have environment specific file, merge its content into the local file, then delete it.
# in file helm/values.local.yaml
-redis:
- auth:
- password: '<secret>'
-After above changes are addressed, upgrading to v4 is as simple as
git pull
-git checkout tags/v4.x.x -b <branch_name>
-yarn install && yarn build
-or, if NotifyBC is deployed to Kubernetes using Helm.
git pull
-git checkout tags/v4.x.x -b <branch_name>
-helm upgrade <release-name> -f helm/platform-specific/<platform>.yaml -f helm/values.local.yaml helm
-Replace v4.x.x with a v4 release, preferably latest, found in GitHub such as v4.0.0.
v4 to v5
v5 introduced following backward incompatible changes
`,8),L=e("li",null,[e("p",null,"Replica set is required for MongoDB. If you deployed NotifyBC using Helm, replica set is already enabled by default.")],-1),F=e("li",null,[e("p",null,[a("If you use default in-memory database, data in "),e("em",null,"server/database/data.json"),a(" will not be migrated automatically. Manually migrate if necessary.")])],-1),V=e("p",null,[a("Update file "),e("em",null,"src/datasources/db.datasource.local.[json|js|ts]")],-1),K=e("li",null,[a("rename "),e("em",null,"url"),a(" property to "),e("em",null,"uri")],-1),W={href:"https://loopback.io/doc/en/lb4/MongoDB-connector.html#creating-a-mongodb-data-source",target:"_blank",rel:"noopener noreferrer"},Q={href:"https://mongoosejs.com/docs/connections.html#options",target:"_blank",rel:"noopener noreferrer"},Z=e("em",null,"host",-1),J=e("em",null,"port",-1),X=e("em",null,"database",-1),Y=e("em",null,"uri",-1),ee=o(`For example, change
{
- "name": "db",
- "connector": "mongodb",
- "url": "mongodb://127.0.0.1:27017/notifyBC"
-}
-to
{
- "uri": "mongodb://127.0.0.1:27017/notifyBC"
-}
-If you deployed NotifyBC using Helm, this is taken care of.
`,5),ae={href:"https://loopback.io/doc/en/lb4/Where-filter.html#operators",target:"_blank",rel:"noopener noreferrer"},ne={href:"https://www.mongodb.com/docs/manual/reference/operator/query/",target:"_blank",rel:"noopener noreferrer"},se=o("| Loopback operators | MongoDB operators |
|---|---|
| eq | $eq |
| and | $and |
| or | $or |
| gt, gte | $gt, $gte |
| lt, lte | $lt, $lte |
| between | (no equivalent, replace with $gt, $and and $lt) |
| inq, nin | $in, $nin |
| near | $near |
| neq | $ne |
| like, nlike | (replace with $regexp) |
| like, nlike, options: i | (replace with $regexp) |
| regexp | $regex |
GET http://localhost:3000/api/configurations?filter={"order":["serviceName asc"]}
-change to either
GET http://localhost:3000/api/configurations?filter={"order":[["serviceName","asc"]]}
-or
GET http://localhost:3000/api/configurations?filter={"order":"serviceName"}
-If you deployed NotifyBC using Helm, change MongoDB password format in your local values yaml file from
# in file helm/values.local.yaml
-mongodb:
- auth:
- rootPassword: <secret>
- replicaSetKey: <secret>
- password: <secret>
-to
# in file helm/values.local.yaml
-mongodb:
- auth:
- rootPassword: <secret>
- replicaSetKey: <secret>
- passwords:
- - <secret>
-After above changes are addressed, to upgrade NotifyBC to v5,
if NotifyBC is deployed from source code, run
git pull -git checkout tags/v5.x.x -b <branch_name> -yarn install && yarn build -if NotifyBC is deployed to Kubernetes using Helm,
- backup MongoDB database
- runReplace <release-name> with installed helm release name
helm uninstall <release-name> - - delete PVCs used by MongoDB stateful set
- runReplace
git pull -git checkout tags/v5.x.x -b <branch_name> -helm install <release-name> -f helm/platform-specific/<platform>.yaml -f helm/values.local.yaml helm -- v5.x.x with a v5 release, preferably latest, found in GitHub such as v5.0.0.
- <release-name> with installed helm release name
- <platform> with openshift or aks depending on your platform
- restore MongoDB database
Welcome
This site aims to be a comprehensive guide to NotifyBC. We’ll cover topics such as getting your instance up and running, interacting with browser or other server components, deployment, and give you some advice on participating in the future development of NotifyBC itself.
Helpful Hints
Throughout this guide there are a number of small-but-handy pieces of information that can make using NotifyBC easier, more interesting, and less hazardous. Here’s what to look out for.
General information
These are tips and tricks that will help you become a NotifyBC wizard!
Important information
These are tidbits you might want to keep in mind.
Warnings
Be aware of these messages if you wish to avoid disaster.
Benchmarks
tl;dr
A NotifyBC server node can deliver 1 million emails in as little as 1 hour to a SMTP server node. SMTP server node's disk I/O is the bottleneck in such case. Throughput can be improved through horizontal scaling.
When NotifyBC is used to deliver broadcast push notifications to a large number of subscribers, probably the most important benchmark is throughput. The benchmark is especially critical if a latency cap is desired. To facilitate capacity planning, load testing on the email channel has been conducted. The test environment, procedure, results and performance tuning advices are provided hereafter.
Environment
Hardware
Two computers, connected by 1Gbps LAN, are used to host
- NotifyBC
- Mac Mini Late 2012 model
- Intel core i7-3615QM
- 16GB RAM
- 2TB HDD
- SMTP and mail delivery
- Lenovo ThinkCentre M Series 2015 model
- Intel core i5-3470
- 8GB RAM
- 256GB SSD
Software Stack
The test was performed in August 2017. Unless otherwise specified, the versions of all other software were reasonably up-to-date at the time of testing.
NotifyBC
- MacOS Sierra Version 10.12.6
- Virtualbox VM with 8vCPU, 10GB RAM, created using miniShift v1.3.1+f4900b07
- OpenShift 1.5.1+7b451fc with metrics
- default NotifyBC OpenShift installation, which contains following relevant pods
- 1 mongodb pod with 1 core, 1GiB RAM limit
- a variable number of Node.js app pods each with 1 core, 1GiB RAM limit. The number varies by test runs as indicated in result.
SMTP and mail delivery
- Windows 7 host
- Virtualbox VM with 4 vCPU, 3.5GB RAM, running Windows Server 2012
- added SMTP Server feature
- in SMTP Server properties dialog box, uncheck all of following boxes in Messages tab
- Limit message size to (KB)
- Limit session size to (KB)
- Limit number of messages per connection to
- Limit number of recipients per message to
Procedure
',11),h=n("em",null,"/src/config.local.js",-1),m=a(`var _ = require('lodash');
-module.exports = {
- smtp: {
- host: '<smtp-vm-ip-or-hostname>',
- secure: false,
- port: 25,
- pool: true,
- direct: false,
- maxMessages: Infinity,
- maxConnections: 50,
- },
- notification: {
- broadcastCustomFilterFunctions: {
- /*jshint camelcase: false */
- contains_ci: {
- _func: function (resolvedArgs) {
- if (!resolvedArgs[0] || !resolvedArgs[1]) {
- return false;
- }
- return (
- _.toLower(resolvedArgs[0]).indexOf(_.toLower(resolvedArgs[1])) >=
- 0
- );
- },
- _signature: [
- {
- types: [2],
- },
- {
- types: [2],
- },
- ],
- },
- },
- },
-};
-$ node dist/utils/load-testing/bulk-post-subs.js -h
-Usage: node bulk-post-subs.js [Options] <userChannelId>
-[Options]:
--a, --api-url-prefix=<string> api url prefix. default to http://localhost:3000/api
--c, --channel=<string> channel. default to email
--s, --service-name=<string> service name. default to load
--n, --number-of-subscribers=<int> number of subscribers. positive integer. default to 1000
--f, --broadcast-push-notification-filter=<string> broadcast push notification filter. default to "contains_ci(title,'vancouver') || contains_ci(title,'victoria')"
--h, --help display this help
-The generated subscriptions contain a filter, hence all load testing results below included time spent on filtering.
`,2),_={href:"https://github.com/bcgov/NotifyBC/blob/main/src/utils/load-testing/curl-ntf.sh",target:"_blank",rel:"noopener noreferrer"},x=n("div",{class:"language-text line-numbers-mode","data-ext":"text"},[n("pre",{class:"language-text"},[n("code",null,`dist/utils/load-testing/curl-ntf.shResults
| email count | time taken (min) | throughput (#/min) | app pod count | notes on bottleneck |
|---|---|---|---|---|
| 1,000,000 | 71.5 | 13,986 | 1 | app pod cpu capped |
| 100,000 | 5.8 | 17,241 | 2 | smtp vm disk queue length hits 1 frequently |
| 1,000,000 | 57 | 17,544 | 2 | smtp vm disk queue length hits 1 frequently |
| 1,000,000 | 57.8 | 17,301 | 3 | smtp vm disk queue length hits 1 frequently |
Test runs using other software or configurations described below have also been conducted. Because throughput is significantly lower, results are not shown
- using Linux sendmail SMTP. The throughput of a 4-vCPU Linux VM is about the same as a 1-vCPU Windows SMTP server. Bottleneck in such case is the CPU of SMTP server.
- Reducing NotifyBC app pod's resource limit to 100 millicore CPU and 512MiB RAM. Even when scaled up pod count to 15, throughput is still about 1/3 of a 1-core pod.
Here is a sample email saved onto the mail drop folder of SMTP server.
Comparison to Other Benchmarks
',6),S={href:"https://technet.microsoft.com/en-us/library/bb124213(v=exchg.65).aspx",target:"_blank",rel:"noopener noreferrer"},T=n("em",null,"NotifyBC",-1),P=n("ol",null,[n("li",null,"Email size in Microsoft's load test is 50k, as opposed to 1k used in this test"),n("li",null,"SSD storage is used in this test. It is unlikely the test conducted in 2005 used SSD.")],-1),B=n("h2",{id:"advices",tabindex:"-1"},[n("a",{class:"header-anchor",href:"#advices","aria-hidden":"true"},"#"),s(" Advices")],-1),C=n("li",null,"Avoid using default direct mode in production. Instead use SMTP server. Direct mode doesn't support connection pooling, resulting in port depletion quickly.",-1),A={href:"https://nodemailer.com/smtp/pooled/",target:"_blank",rel:"noopener noreferrer"},N=a("- use SSD for its storage
- create a load balanced cluster if possible, since SMTP server is more likely to be the bottleneck.
Worker Process Count
When NotifyBC runs on a host with multiple CPUs, by default it creates a cluster of worker processes of which the count matches CPU count. You can override the number with the environment variable NOTIFYBC_WORKER_PROCESS_COUNT.
A note about worker process count on OpenShift
It has been observed that on OpenShift Node.js returns incorrect CPU count. The template therefore sets NOTIFYBC_WORKER_PROCESS_COUNT to 1. After all, on OpenShift NotifyBC is expected to be horizontally scaled by pods rather by CPUs.
Overview
NotifyBC is a general purpose API Server to manage subscriptions and dispatch notifications. It aims to implement some common backend processes of a notification service without imposing any constraints on the UI frontend, nor impeding other server components' functionality. This is achieved by interacting with user browser and other server components through RESTful API and other standard protocols in a loosely coupled way.
Features
NotifyBC facilitates both anonymous and authentication-enabled secure webapps implementing notification feature. A NotifyBC server instance supports multiple notification services. A service is a topic of interest that user wants to receive updates. It is used as the partition of notification messages and user subscriptions. A user may subscribe to a service in multiple push delivery channels allowed. A user may subscribe to multiple services. In-app pull notification doesn't require subscription as it's not intrusive to user.
notification
- both in-app pull notifications (a.k.a. messages or alerts) and push notifications
- multiple push notifications delivery channels
- sms
- unicast and broadcast message types
- future-dated notifications
- for in-app pull notifications
- support read and deleted message states
- message expiration
- deleted messages are not deleted immediately for auditing and recovery purposes
- for broadcast push notifications
- allow both sync and async POST API calls. For async API call, an optional callback url is supported
- can be auto-generated from RSS feeds
- allow user to specify filter rules evaluated against data contained in the notification
- allow sender to specify filter rules evaluated against data contained in the subscription
- allow application developer to create custom filter functions used by the filter rules mentioned above
subscription and un-subscription
- verify the ownership of push notification subscription channel:
- generates confirmation code based on a regex input
- send confirmation request to unconfirmed subscription channel
- verify confirmation code
- generate random un-subscription code
- send acknowledgement message after un-subscription for anonymous subscribers
- bulk unsubscription
- list-unsubscribe by email
- track bounces and unsubscribe the recipient from all subscriptions when hard bounce count exceeds threshold
- sms user can unsubscribe by replying a shortcode keyword with Swift sms provider
mail merge
Strings in notification or subscription message that are enclosed between curly braces { } are called tokens, also known as placeholders. Tokens are replaced based on the context of notification or subscription when dispatching the message. To avoid treating a string between curly braces as a token, escape the curly braces with backslash \\. For example \\{i_am_not_a_token\\} is not a token. It will be rendered as {i_am_not_a_token}.
Tokens whose names are predetermined by NotifyBC are called static tokens; otherwise they are called dynamic tokens.
static tokens
NotifyBC recognizes following case-insensitive static tokens. Most of the names are self-explanatory.
- {subscription_confirmation_url}
- {subscription_confirmation_code}
- {service_name}
- {http_host} - http host in the form http(s): //<host_name>:<port>. The value is obtained from the http request that triggers the message
- {rest_api_root} - REST API URL path prefix
- {subscription_id}
- anonymous unsubscription related tokens
- {unsubscription_url}
- {unsubscription_all_url} - url to unsubscribe all services the user has subscribed on this NotifyBC instance
- {unsubscription_code}
- {unsubscription_reversion_url}
- {unsubscription_service_names} - includes {service_name} and additional services user has unsubscribed, prefixed with conditionally pluralized word service.
dynamic tokens
Dynamic tokens are replaced with correspondingly named sub-field of data field in the notification or subscription if exist. Qualify token name with notification:: or subscription:: to indicate the source of substitution. If token name is not qualified, then both notification and subscription are checked, with notification taking precedence. Nested and indexed sub-fields are supported.
Examples
- {notification::description} is replaced with field data.description of the notification if exist
- {subscription::gender} is replaced with field data.gender of the subscription if exist
- {addresses[0].city} is replaced with field data.addresses[0].city of the notification if exist; otherwise is replaced with field data.addresses[0].city of the subscription if exist
- {nonexistingDataField} is unreplaced if neither notification nor subscription contains data.nonexistingDataField
Notification by RSS feeds relies on dynamic token
A notification created by RSS feeds relies on dynamic token to supply the context to message template. In this case the data field contains the RSS item.
Architecture
Request Types
NotifyBC, designed to be a microservice, doesn't use full-blown ACL to secure API calls. Instead, it classifies incoming requests into admin and user types. The key difference is while both admin and user can subscribe to notifications, only admin can post notifications.
Each type has two subtypes based on following criteria
',5),b=e("p",null,"super-admin, if the request meets both of the following two requirements",-1),g=e("p",null,"The request carries one of the following two attributes",-1),y=e("li",null,"the source ip is in the admin ip list",-1),w=e("em",null,"NotifyBC",-1),v=e("li",null,[e("p",null,"The request doesn't contain any of following case insensitive HTTP headers, with the first three being SiteMinder headers"),e("ul",null,[e("li",null,"sm_universalid"),e("li",null,"sm_user"),e("li",null,"smgov_userdisplayname"),e("li",null,"is_anonymous")])],-1),_=s('admin, if the request is not super-admin and meets one of the following criteria
- has a valid access token associated with an builtin admin user created and logged in using the administrator api, and the request doesn't contain any HTTP headers listed above
- has a valid OIDC access token containing customizable admin profile attributes
access token disambiguation
Here the term access token has been used to refer two different things
- the token associated with a builtin admin user
- the token generated by OIDC provider.
To reduce confusion, throughout the documentation the former is called access token and the latter is called OIDC access token.
authenticated user, if the request is neither super-admin nor admin, and meets one fo the following criteria
- contains any of the 3 SiteMinder headers listed above, and comes from either trusted SiteMinder proxy or admin ip list
- contains a valid OIDC access token
anonymous user, if the request doesn't meet any of the above criteria
The only extra privileges that a super-admin has over admin are that super-admin can perform CRUD operations on configuration, bounce and administrator entities through REST API. In the remaining docs, when no further distinction is necessary, an admin request refers to both super-admin and admin request; a user request refers to both authenticated and anonymous request.
An admin request carries full authorization whereas user request has limited access. For example, a user request is not allowed to
- send notification
- bypass the delivery channel confirmation process when subscribing to a service
- retrieve push notifications through API (can only receive notification from push notification channel such as email)
- retrieve in-app notifications that is not targeted to the current user
The result of an API call to the same end point may differ depending on the request type. For example, the call GET /notifications without a filter will return all notifications to all users for an admin request, but only non-deleted, non-expired in-app notifications for authenticated user request, and forbidden for anonymous user request. Sometimes it is desirable for a request from admin ip list, which would normally be admin request, to be voluntarily downgraded to user request in order to take advantage of predefined filters such as the ones described above. This can be achieved by adding one of the HTTP headers listed above to the request. This is also why admin request is not determined by ip or token alone.
",4),x=e("em",null,"NotifyBC",-1),q=["src"],C=s('Authentication Strategies
API requests to NotifyBC can be either anonymous or authenticated. As alluded in Request Types above, NotifyBC supports following authentication strategies
- ip whitelisting
- client certificate
- access token associated with an builtin admin user
- OpenID Connect (OIDC)
- CA SiteMinder
Authentication is performed in above order. Once a request passed an authentication strategy, the rest strategies are skipped. A request that failed all authentication strategies is anonymous.
The mapping between authentication strategy and request type is
| Admin | User | |||
| Super-admin | admin | authenticated | anonymous | |
| ip whitelisting | ✔ | |||
| client certifcate | ✔ | |||
| access token | ✔ | |||
| OIDC | ✔ | ✔ | ||
| SiteMinder | ✔ | |||
Which authentication strategy to use?
Because ip whitelist doesn't expire and client certificate usually has a relatively long expiration period (say one year), they are suitable for long-running unattended server processes such as server-side code of web apps, cron jobs, IOT sensors etc. The server processes have to be trusted because once authenticated, they have full privilege to NotifyBC. Usually the server processes and NotifyBC instance are in the same administrative domain, i.e. managed by the same admin group of an organization.
By contrast, OIDC and SiteMinder use short-lived tokens or session cookies. Therefore they are only suitable for interactive user sessions.
Access token associated with an builtin admin user should be avoided whenever possible.
Here are some common scenarios and recommendations
For server-side code of web apps
- use OIDC if the web app is OIDC enabled and user requests can be proxied to NotifyBC by web app; otherwise
- use ip whitelisting if obtaining ip is feasible; otherwise
- use client certificate (requires a little more config than ip whitelisting)
For front-end browser-based web apps such as SPAs
- use OIDC
For server apps that send requests spontaneously such as IOT sensors, cron jobs
- use ip whitelisting if obtaining ip is feasible; otherwise
- client certificate
If NotifyBC is ued by a SiteMinder protected web apps and NotifyBC is also protected by SiteMinder
- use SiteMinder
Application Framework
',8),I=e("em",null,"NotifyBC",-1),S={href:"https://nestjs.com/",target:"_blank",rel:"noopener noreferrer"},T=e("em",null,"NotifyBC",-1),A={href:"https://docs.nestjs.com/",target:"_blank",rel:"noopener noreferrer"};function N(l,B){const n=r("RouterLink"),o=r("ExternalLinkIcon");return d(),u("div",null,[p,e("p",null,[i("As exception, in order to prevent spamming by unconfirmed subscribers, dynamic tokens in subscription "),t(n,{to:"/docs/config-subscription/#confirmation-request-message"},{default:a(()=>[i("confirmation request message")]),_:1}),i(" and "),t(n,{to:"/docs/config-subscription/#duplicated-subscription"},{default:a(()=>[i("duplicated subscription")]),_:1}),i(" message are not replaced with subscription data, for example "),m,i(" tokens are left unchanged.")]),f,e("ul",null,[e("li",null,[b,e("ol",null,[e("li",null,[g,e("ul",null,[y,e("li",null,[i("has a client certificate that is signed using "),w,i(" server certificate. See "),t(n,{to:"/docs/config/certificates.html#client-certificate-authentication"},{default:a(()=>[i("Client certificate authentication")]),_:1}),i(" on how to sign.")])])]),v])]),_]),k,e("p",null,[i("The way "),x,i(" interacts with other components is diagrammed below. "),e("img",{src:l.$withBase("/img/architecture.svg"),alt:"architecture diagram"},null,8,q)]),C,e("p",null,[I,i(" is created on "),e("a",S,[i("NestJS"),t(o)]),i(". Contributors to source code of "),T,i(" should be familiar with NestJS. "),e("a",A,[i("NestJS Docs"),t(o)]),i(" serves a good complement to this documentation.")])])}const O=c(h,[["render",N],["__file","index.html.vue"]]);export{O as default}; diff --git a/version/5.0/assets/index.html-1d87c569.js b/version/5.0/assets/index.html-1d87c569.js deleted file mode 100644 index 95965fb7d..000000000 --- a/version/5.0/assets/index.html-1d87c569.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse('{"key":"v-6768263b","path":"/docs/overview/","title":"Overview","lang":"en-US","frontmatter":{"permalink":"/docs/overview/"},"headers":[{"level":2,"title":"Features","slug":"features","link":"#features","children":[{"level":3,"title":"notification","slug":"notification","link":"#notification","children":[]},{"level":3,"title":"subscription and un-subscription","slug":"subscription-and-un-subscription","link":"#subscription-and-un-subscription","children":[]},{"level":3,"title":"mail merge","slug":"mail-merge","link":"#mail-merge","children":[]}]},{"level":2,"title":"Architecture","slug":"architecture","link":"#architecture","children":[{"level":3,"title":"Request Types","slug":"request-types","link":"#request-types","children":[]},{"level":3,"title":"Authentication Strategies","slug":"authentication-strategies","link":"#authentication-strategies","children":[]}]},{"level":2,"title":"Application Framework","slug":"application-framework","link":"#application-framework","children":[]}],"git":{},"filePathRelative":"docs/getting-started/overview.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-1faeb6c7.js b/version/5.0/assets/index.html-1faeb6c7.js deleted file mode 100644 index 07da582af..000000000 --- a/version/5.0/assets/index.html-1faeb6c7.js +++ /dev/null @@ -1,60 +0,0 @@ -import{_ as r,r as n,o as l,c as d,a as e,b as t,d as a,w as p,e as s}from"./app-5138f739.js";const c={},u=s(`Notification
The notification API encapsulates the backend workflow of staging and dispatching a message to targeted user after receiving the message from event source.
Depending on whether an API call comes from user browser as a user request or from an authorized server application as an admin request, NotifyBC applies different permissions. Admin request allows full CRUD operations. An authenticated user request, on the other hand, are only allowed to get a list of in-app pull notifications targeted to the current user and changing the state of the notifications. An unauthenticated user request can not access any API.
When a notification is created by the event source server application, the message is saved to database prior to responding to API caller. In addition, for push notification, the message is delivered immediately, i.e. the API call is synchronous. For in-app pull notification, the message, which by default is in state new, can be retrieved later on by browser user request. A user request can only get the list of in-app messages targeted to the current user. A user request can then change the message state to read or deleted depending on user action. A deleted message cannot be retrieved subsequently by user requests, but the state can be updated given the correct id.
Deleted message is still kept in database.
NotifyBC provides API for deleting a notification. For the purpose of auditing and recovery, this API only marks the state field as deleted rather than deleting the record from database.
undo in-app notification deletion within a session
Because "deleted" message is still kept in database, you can implement undo feature for in-app notification as long as the message id is retained prior to deletion within the current session. To undo, call update API to set desired state.
In-app pull notification also supports message expiration by setting a date in field validTill. An expired message cannot be retrieved by user requests.
A message, regardless of push or pull, can be unicast or broadcast. A unicast message is intended for an individual user whereas a broadcast message is intended for all confirmed subscribers of a service. A unicast message must have field userChannelId populated. The value of userChannelId is channel dependent. In the case of email for example, this would be user's email address. A broadcast message must set isBroadcast to true and leave userChannelId empty.
Why field isBroadcast?
Unicast and broadcast message can be distinguished by whether field userChannelId is empty or not alone. So why the extra field isBroadcast? This is in order to prevent inadvertent marking a unicast message broadcast by omitting userChannelId or populating it with empty value. The precaution is necessary because in-app notifications may contain personalized and confidential information.
NotifyBC ensures the state of an in-app broadcast message is isolated by user, so that for example, a message read by one user is still new to another user. To achieve this, NotifyBC maintains two internal fields of array type - readBy and deletedBy. When a user request updates the state field of an in-app broadcast message to read or deleted, instead of altering the state field, NotifyBC appends the current user to readBy or deletedBy list. When user request retrieving in-app messages, the state field of the broadcast message in HTTP response is updated based on whether the user exists in field deletedBy and readBy. If existing in both fields, deletedBy takes precedence (the message therefore is not returned). The record in database, meanwhile, is unchanged. Neither field deletedBy nor readBy is visible to user request.
Model Schema
The API operates on following notification data model fields:
| Name | Attributes | ||||||
|---|---|---|---|---|---|---|---|
id notification id |
| ||||||
serviceName name of the service |
| ||||||
channel name of the delivery channel. Valid values: inApp, email, sms. |
| ||||||
userChannelId user's delivery channel id, for example, email address. For unicast inApp notification, this is authenticated user id. When sending unicast push notification, either userChannelId or userId is required. |
| ||||||
userId authenticated user id. When sending unicast push notification, either userChannelId or userId is required. |
| ||||||
state state of notification. Valid values: new, read (inApp only), deleted (inApp only), sent (push only) or error. For inApp broadcast notification, if the user has read or deleted the message, the value of this field retrieved by admin request will still be new. The state for the user is tracked in fields readBy and deletedBy in such case. For user request, the value contains correct state. |
| ||||||
created date and time of creation |
| ||||||
updated date and time of last update |
| ||||||
isBroadcast whether it's a broadcast message. A broadcast message should omit userChannelId and userId, in addition to setting isBroadcast to true |
| ||||||
skipSubscriptionConfirmationCheck When sending unicast push notification, whether or not to verify if the recipient has a confirmed subscription. This field allows subscription information be kept elsewhere and NotifyBC be used as a unicast push notification gateway only. |
| ||||||
validTill expiration date-time of the message. Applicable to inApp notification only. |
| ||||||
invalidBefore date-time in the future after which the notification can be dispatched. |
| ||||||
an object whose child fields are channel dependent:
|
| ||||||
httpHost This field is used to replace token {http_host} in push notification message template during mail merge and overrides config httpHost. |
| ||||||
asyncBroadcastPushNotification this field determines if the API call to create an immediate (i.e. not future-dated) broadcast push notification is asynchronous or not. If omitted, the API call is synchronous, i.e. the API call blocks until notifications to all subscribers have been dispatched. If set, valid values and corresponding behaviors are
|
| ||||||
the event that triggers the notification, for example, a RSS feed item when the notification is generated automatically by RSS cron job. Field data serves two purposes
|
| ||||||
broadcastPushNotificationSubscriptionFilter a string conforming to jmespath filter expressions syntax after the question mark (?). The filter is matched against the data field of the subscription. Examples of filter
|
| ||||||
readBy this is an internal field to track the list of users who have read an inApp broadcast message. It's not visible to a user request. |
| ||||||
deletedBy this is an internal field to track the list of users who have marked an inApp broadcast message as deleted. It's not visible to a user request. |
| ||||||
dispatch this is an internal field to track the broadcast push notification dispatch outcome. It consists of up to four arrays
|
|
Get Notifications
GET /notifications
-a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
",3),b=e("li",null,"URL-encoded stringified JSON object (see example below); or",-1),g={href:"https://github.com/ljharb/qs",target:"_blank",rel:"noopener noreferrer"},v=e("code",null,'?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"',-1),y=s(`Regardless, the filter will have to be parsed into a JSON object conforming to
{
- "where": {...},
- "fields": ...,
- "order": ...,
- "skip": ...,
- "limit": ...,
-}
-All properties are optional. The syntax for each property is documented, respectively
`,3),k=e("em",null,"where",-1),q={href:"https://www.mongodb.com/docs/manual/tutorial/query-documents/",target:"_blank",rel:"noopener noreferrer"},_=e("em",null,"fields",-1),w={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.select()",target:"_blank",rel:"noopener noreferrer"},x=e("em",null,"order",-1),I={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.sort()",target:"_blank",rel:"noopener noreferrer"},B=e("em",null,"skip",-1),j={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.skip/",target:"_blank",rel:"noopener noreferrer"},A=e("em",null,"limit",-1),C={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.limit/",target:"_blank",rel:"noopener noreferrer"},T=s(`outcome
- for admin requests, returns unabridged array of notification data matching the filter
- for authenticated user requests, in addition to filter, following constraints are imposed on the returned array
- only inApp notifications
- only non-deleted notifications. For broadcast notification, non-deleted means not marked by current user as deleted
- only non-expired notifications
- for unicast notifications, only the ones targeted to current user
- if current user is in readBy, then the state is changed to read
- the internal field readBy and deletedBy are removed
- forbidden to anonymous user requests
example
to retrieve notifications created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/notifications?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D'
-the value of the filter query parameter is URL-encoded stringified JSON object
{
- "where": {
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
- }
-}
-Get Notification Count
GET /notifications/count
-outcome
Validations rules are the same as GET /notifications. If passed, the output is a count of notifications matching the query
{
- "count": <number>
-}
-example
to retrieve the count of notifications created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/notifications/count?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D'
-the value of the where query parameter is URL-encoded stringified JSON object
{
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
-}
-Create/Send Notifications
POST /notifications
-permissions required, one of
- super admin
- admin
inputs
- an object containing notification data model fields. At a minimum all required fields that don't have a default value must be supplied. Id field should be omitted since it's auto-generated. The API explorer only created an empty object for field message but you should populate the child fields according to model schema
- parameter name: data
- required: true
- parameter type: body
- data type: object
if it's a user request, error is returned
inputs are validated. If validation fails, error is returned. In particular, for unicast push notification, the recipient as identified by either userChannelId or userId must have a confirmed subscription if field skipSubscriptionConfirmationCheck is not set to true. If skipSubscriptionConfirmationCheck is set to true, then the subscription check is skipped, but in such case the request must contain userChannelId, not userId as subscription data is not queried to obtain userChannelId from userId.
for push notification, if field httpHost is empty, it is populated based on request's http protocol and host.
the notification request is saved to database
if the notification is future-dated, then all subsequent request processing is skipped and response is sent back to user. Steps 7-11 below will be carried out later on by the cron job when the notification becomes current.
if it's an async broadcast push notification, then response is sent back to user but steps 7-12 below is processed separately
when processing an individual subscription,
- if the subscription has filter rule defined in field broadcastPushNotificationFilter and notification contains field data, then the data is matched against the filter rule. Notification message is only dispatched if there is a match.
- if the notification has filter rule defined in field broadcastPushNotificationSubscriptionFilter and subscription contains field data, then the data is matched against the filter rule. Notification message is only dispatched if there is a match.
If the subscription failed to pass any of the two filters, and if both guaranteedBroadcastPushDispatchProcessing and logSkippedBroadcastPushDispatches are true, the subscription id is logged to dispatch.skipped
the state of push notification is updated to sent or error depending on sending status. For broadcast push notification, the dispatching could be failed only for a subset of users. In such case, the field dispatch.failed contains a list of objects of {userChannelId, subscriptionId, error} the message failed to deliver to, but the state will still be set to sent.
For broadcast push notifications, if guaranteedBroadcastPushDispatchProcessing is true, then field dispatch.successful is populated with a list of subscriptionId of the successful dispatches.
For push notifications, the bounce records of successful dispatches are updated
the updated notification is saved back to database
if it's an async broadcast push notification with a callback url, then the url is called with POST verb containing the notification with updated status as the request body
for synchronous notification, the saved record is returned unless there is an error saving to database, in which case error is returned
example
To send a unicast email push notification, copy and paste following json object to the data value box in API explorer, change email addresses as needed, and click Try it out! button:
{
- "serviceName": "education",
- "userChannelId": "foo@bar.com",
- "skipSubscriptionConfirmationCheck": true,
- "message": {
- "from": "no_reply@bar.com",
- "subject": "test",
- "textBody": "This is a test"
- },
- "channel": "email"
-}
-As the result, foo@bar.com should receive an email notification even if the user is not a confirmed subscriber, and following json object is returned to caller upon sending the email successfully:
{
- "serviceName": "education",
- "state": "sent",
- "userChannelId": "foo@bar.com",
- "skipSubscriptionConfirmationCheck": true,
- "message": {
- "from": "no_reply@bar.com",
- "subject": "test",
- "textBody": "This is a test"
- },
- "created": "2016-09-30T20:37:06.011Z",
- "updated": "2016-09-30T20:37:06.011Z",
- "channel": "email",
- "isBroadcast": false,
- "id": "57eeccf23427b61a4820775e"
-}
-Update a Notification
PATCH /notifications/{id}
-This API is mainly used for updating an inApp notification.
permissions required, one of
- super admin
- admin
- authenticated user
inputs
- notification id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- an object containing fields to be updated.
- parameter name: data
- required: true
- parameter type: body
- data type: object
- notification id
outcome
- for user requests, NotifyBC performs following actions in sequence
- for unicast notification, if the notification is not targeted to current user, error is returned
- all fields except for state are discarded from the input
- for broadcast notification, current user id in appended to array readBy or deletedBy, depending on whether state is read or deleted, unless the user id is already in the array. The state field itself is then discarded
- the notification identified by id is merged with the updates and saved to database
- HTTP response code 204 is returned, unless there is error.
- admin requests are allowed to update any field
- for user requests, NotifyBC performs following actions in sequence
Delete a Notification
This API is mainly used for marking an inApp notification deleted. It has the same effect as updating a notification with state set to deleted.
DELETE /notifications/{id}
-- permissions required, one of
- super admin
- admin
- authenticated user
- inputs
- notification id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- notification id
- outcome: same as the outcome of Update a Notification with state set to deleted.
Replace a Notification
PUT /notifications/{id}
-This API is intended to be only used by admin web console to modify a notification in new state. Notifications in such state are typically future-dated or of channel in-app.
permissions required, one of
- super admin
- admin
inputs
- notification id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- notification data
- parameter name: data
- required: true
- parameter type: body
- data type: object
- notification id
outcome
NotifyBC process the request same way as Create/Send Notifications except that notification data is saved with id supplied in the parameter, replacing existing one.
API Overview
NotifyBC's core function is implemented by two models - subscription and notification. Other models - configuration, administrator and bounces etc, are for administrative purposes. A model determines the underlying database schema and the API. The APIs displayed in the web console (by default http://localhost:3000) and API explorer are also grouped by models. Click on a model in API explorer, say notification, to explore the operations on that model. Model specific APIs are available here:
',3),r=[n];function l(s,c){return i(),a("div",null,r)}const h=e(t,[["render",l],["__file","index.html.vue"]]);export{h as default}; diff --git a/version/5.0/assets/index.html-2fb78bd6.js b/version/5.0/assets/index.html-2fb78bd6.js deleted file mode 100644 index 6de3975e4..000000000 --- a/version/5.0/assets/index.html-2fb78bd6.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse('{"key":"v-147825fb","path":"/docs/","title":"Welcome","lang":"en-US","frontmatter":{"permalink":"/docs/"},"headers":[{"level":2,"title":"Helpful Hints","slug":"helpful-hints","link":"#helpful-hints","children":[]}],"git":{},"filePathRelative":"docs/getting-started/index.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-31bb012a.js b/version/5.0/assets/index.html-31bb012a.js deleted file mode 100644 index c3255c25d..000000000 --- a/version/5.0/assets/index.html-31bb012a.js +++ /dev/null @@ -1 +0,0 @@ -import{_ as i,r,o as a,c as s,a as t,b as e,d as n,e as c}from"./app-5138f739.js";const l={},p=c('Code of Conduct
As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality.
Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery
- Personal attacks
- Trolling or insulting/derogatory comments
- Public or private harassment
- Publishing other's private information, such as physical or electronic addresses, without explicit permission
- Other unethical or unprofessional conduct
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect of managing this project. Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team.
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community.
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting a project maintainer. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. Maintainers are obligated to maintain confidentiality with regard to the reporter of an incident.
',9),d={href:"http://contributor-covenant.org",target:"_blank",rel:"noopener noreferrer"},h={href:"http://contributor-covenant.org/version/1/3/0/",target:"_blank",rel:"noopener noreferrer"};function u(m,f){const o=r("ExternalLinkIcon");return a(),s("div",null,[p,t("p",null,[e("This Code of Conduct is adapted from the "),t("a",d,[e("Contributor Covenant"),n(o)]),e(", version 1.3.0, available at "),t("a",h,[e("http://contributor-covenant.org/version/1/3/0/"),n(o)])])])}const b=i(l,[["render",u],["__file","index.html.vue"]]);export{b as default}; diff --git a/version/5.0/assets/index.html-34c15120.js b/version/5.0/assets/index.html-34c15120.js deleted file mode 100644 index e055ade3d..000000000 --- a/version/5.0/assets/index.html-34c15120.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse('{"key":"v-22e054a1","path":"/docs/config-workerProcessCount/","title":"Worker Process Count","lang":"en-US","frontmatter":{"permalink":"/docs/config-workerProcessCount/"},"headers":[],"git":{},"filePathRelative":"docs/config/workerProcessCount.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-37bb569a.js b/version/5.0/assets/index.html-37bb569a.js deleted file mode 100644 index 46d89c0b8..000000000 --- a/version/5.0/assets/index.html-37bb569a.js +++ /dev/null @@ -1,34 +0,0 @@ -import{_ as s,r as l,o as r,c as o,a as e,b as t,d as n,e as i}from"./app-5138f739.js";const d={},p=i(`Configuration
The configuration API, accessible by only super-admin requests, is used to define dynamic configurations. Dynamic configuration is needed in situations like
- RSA key pair generated automatically at boot time if not present
- service-specific subscription confirmation request message template
Model Schema
The API operates on following configuration data model fields:
| Name | Attributes | ||||
|---|---|---|---|---|---|
id config id |
| ||||
name config name |
| ||||
value config value. |
| ||||
serviceName name of the service the config applicable to |
|
Get Configurations
GET /configurations
-a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
",3),h=e("li",null,"URL-encoded stringified JSON object (see example below); or",-1),f={href:"https://github.com/ljharb/qs",target:"_blank",rel:"noopener noreferrer"},g=e("code",null,'?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"',-1),b=i(`Regardless, the filter will have to be parsed into a JSON object conforming to
{
- "where": {...},
- "fields": ...,
- "order": ...,
- "skip": ...,
- "limit": ...,
-}
-All properties are optional. The syntax for each property is documented, respectively
`,3),v=e("em",null,"where",-1),_={href:"https://www.mongodb.com/docs/manual/tutorial/query-documents/",target:"_blank",rel:"noopener noreferrer"},k=e("em",null,"fields",-1),q={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.select()",target:"_blank",rel:"noopener noreferrer"},y=e("em",null,"order",-1),x={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.sort()",target:"_blank",rel:"noopener noreferrer"},w=e("em",null,"skip",-1),j={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.skip/",target:"_blank",rel:"noopener noreferrer"},T=e("em",null,"limit",-1),A={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.limit/",target:"_blank",rel:"noopener noreferrer"},C=i(`outcome
For admin request, a list of config items matching the filter; forbidden for user request
example
to retrieve configs created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/configurations?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D'
-the value of the filter query parameter is URL-encoded stringified JSON object
{
- "where": {
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
- }
-}
-Create a Configuration
POST /configurations
-permissions required, one of
- super admin
- admin
inputs
- an object containing configuration data model fields. At a minimum all required fields that don't have a default value must be supplied. Id field should be omitted since it's auto-generated. The API explorer only created an empty object for field value but you should populate the child fields.
- parameter name: data
- required: true
- parameter type: body
- data type: object
- an object containing configuration data model fields. At a minimum all required fields that don't have a default value must be supplied. Id field should be omitted since it's auto-generated. The API explorer only created an empty object for field value but you should populate the child fields.
outcome
NotifyBC performs following actions in sequence
- if it’s a user request, error is returned
- inputs are validated. For example, required fields without default values must be populated. If validation fails, error is returned
- if config item is notification with field value.rss populated, and if the field value.httpHost is missing, it is generated using this request's HTTP protocol , host name and port.
- item is saved to database
example
see the cURL command on how to create a dynamic subscription config
Update a Configuration
PATCH /configurations/{id}
-permissions required, one of
- super admin
- admin
inputs
- configuration id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- an object containing fields to be updated.
- parameter name: data
- required: true
- parameter type: body
- data type: object
- configuration id
outcome
Similar to POST except field update is always updated with current timestamp.
Update Configurations
PATCH /configurations
-outcome
Similar to POST except field update is always updated with current timestamp.
example
to set serviceName to myService for all configs created in year 2023 , run
curl -X PATCH --header 'Content-Type: application/json' 'http://localhost:3000/api/configurations?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D' -d @- << EOF
-{
- "serviceName": "myService",
-}
-EOF
-the value of the where query parameter is URL-encoded stringified JSON object
{
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
-}
-Delete a Configuration
DELETE /configurations/{id}
-permissions required, one of
- super admin
- admin
inputs
- configuration id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- configuration id
outcome
For admin request, delete the config item requested; forbidden for user request
Replace a Configuration
PUT /configurations/{id}
-This API is intended to be only used by admin web console to modify a configuration.
permissions required, one of
- super admin
- admin
inputs
- configuration id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- configuration data
- parameter name: data
- required: true
- parameter type: body
- data type: object
- configuration id
outcome
For admin requests, replace configuration identified by id with parameter data and save to database.
To run the utility
git clone https://github.com/bcgov/NotifyBC.git
-cd NotifyBC
-npm i -g yarn && yarn install && yarn build
-node dist/utils/bulk-import/subscription.js -a <api-url-prefix> -c <concurrency> <csv-file-path>
-Here <csv-file-path> is the path to csv file and <api-url-prefix> is the NotifyBC api url prefix , default to http://localhost:3000/api.
The script parses the csv file and generates a HTTP post request for each row. The concurrency of HTTP request is controlled by option -c which is default to 10 if omitted. A successful run should output the number of rows imported without any error message
success row count = ***
-Field Parsers
`,6),g={href:"https://github.com/Keyang/node-csvtojson#custom-parsers",target:"_blank",rel:"noopener noreferrer"},y={href:"https://github.com/bcgov/NotifyBC/tree/main/src/utils/bulk-import/subscription.ts",target:"_blank",rel:"noopener noreferrer"},x=s("em",null,"src/utils/bulk-import/subscription.ts",-1),w=s("em",null,"myCustomIntegerField",-1),C=s("em",null,"colParser",-1),N=r(` colParser: {
- ...
- , myCustomIntegerField: (item, head, resultRow, row, colIdx) => {
- return parseInt(item)
- }
- }
-A example of complete OIDC configuration looks like
module.exports = {
- ...
- oidc: {
- discoveryUrl:
- 'https://op.example.com/auth/realms/foo/.well-known/openid-configuration',
- clientId: 'NotifyBC',
- isAdmin(user) {
- const roles = user.resource_access.NotifyBC.roles;
- if (!(roles instanceof Array) || roles.length === 0) return false;
- return roles.indexOf('admin') > -1;
- },
- isAuthorizedUser(user) {
- return user.realm_access.roles.indexOf('offline_access') > -1;
- },
- },
-};
-In NotifyBC web console and only in the web console, OIDC authentication takes precedence over built-in admin user, meaning if OIDC is configured, the login button goes to OIDC provider rather than the login form.
There is no default OIDC configuration in /src/config.ts.
`,4);function v(b,y){const e=o("ExternalLinkIcon");return p(),c("div",null,[r,u,d,n("ol",null,[n("li",null,[k,s(" - "),n("a",m,[s("OIDC discovery"),i(e)]),s(" url")]),f]),h])}const g=t(l,[["render",v],["__file","index.html.vue"]]);export{g as default}; diff --git a/version/5.0/assets/index.html-42466a4a.js b/version/5.0/assets/index.html-42466a4a.js deleted file mode 100644 index 110147f27..000000000 --- a/version/5.0/assets/index.html-42466a4a.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse('{"key":"v-02a19d2b","path":"/docs/config-rsaKeys/","title":"RSA Keys","lang":"en-US","frontmatter":{"permalink":"/docs/config-rsaKeys/"},"headers":[],"git":{},"filePathRelative":"docs/config/rsaKeys.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-44f40347.js b/version/5.0/assets/index.html-44f40347.js deleted file mode 100644 index e114faddb..000000000 --- a/version/5.0/assets/index.html-44f40347.js +++ /dev/null @@ -1,74 +0,0 @@ -import{_ as i,r as p,o as l,c as r,a as n,b as s,d as a,w as c,e as t}from"./app-5138f739.js";const u={},d=n("h1",{id:"sms",tabindex:"-1"},[n("a",{class:"header-anchor",href:"#sms","aria-hidden":"true"},"#"),s(" SMS")],-1),m=n("h2",{id:"provider",tabindex:"-1"},[n("a",{class:"header-anchor",href:"#provider","aria-hidden":"true"},"#"),s(" Provider")],-1),k=n("p",null,[n("em",null,"NotifyBC"),s(" depends on underlying SMS service providers to deliver SMS messages. The supported service providers are")],-1),v={href:"https://twilio.com/",target:"_blank",rel:"noopener noreferrer"},b={href:"https://www.swiftsmsgateway.com",target:"_blank",rel:"noopener noreferrer"},h=t(`Only one service provider can be chosen per installation. To change service provider, add following config to file /src/config.local.js
module.exports = {
- sms: {
- provider: 'swift',
- },
-};
-Provider Settings
Provider specific settings are defined in config sms.providerSettings. You should have an account with the chosen service provider before proceeding.
Twilio
Add sms.providerSettings.twilio config object to file /src/config.local.js
module.exports = {
- sms: {
- providerSettings: {
- twilio: {
- accountSid: '<AccountSid>',
- authToken: '<AuthToken>',
- fromNumber: '<FromNumber>',
- },
- },
- },
-};
-Obtain <AccountSid>, <AuthToken> and <FromNumber> from your Twilio account.
Swift
Add sms.providerSettings.swift config object to file /src/config.local.js
module.exports = {
- sms: {
- providerSettings: {
- swift: {
- accountKey: '<accountKey>',
- },
- },
- },
-};
-Obtain <accountKey> from your Swift account.
Unsubscription by replying a keyword
With Swift short code, sms user can unsubscribe by replying to a sms message with a keyword. The keyword must be pre-registered with Swift.
To enable this feature,
`,15),g=t(`Generate a random string, hereafter referred to as <randomly-generated-string>
Add it to sms.providerSettings.swift.notifyBCSwiftKey in file /src/config.local.js
module.exports = {
- sms: {
- providerSettings: {
- swift: {
- notifyBCSwiftKey: '<randomly-generated-string>',
- },
- },
- },
-};
-Go to Swift web admin console, click Number Management tab
Click Launch button next to Manage Short Code Keywords
Click Features button next to the registered keyword(s). A keyword may have multiple entries. In such case do this for each entry.
Click Redirect To Webpage tab in the popup window
Throttle
All supported SMS service providers impose request rate limit. NotifyBC by default throttles request rate to 4/sec. To adjust the rate, create following config in file /src/config.local.js
module.exports = {
- sms: {
- throttle: {
- // minimum request interval in ms
- minTime: 250,
- },
- },
-};
-When NotifyBC is deployed from source code, by default the rate limit applies to one Node.js process only. If there are multiple processes, i.e. a cluster, the aggregated rate limit is multiplied by the number of processes. To enforce the rate limit across entire cluster, install Redis and add Redis config to sms.throttle
module.exports = {
- sms: {
- throttle: {
- /* Redis clustering options */
- datastore: 'ioredis',
- clientOptions: {
- host: '127.0.0.1',
- port: 6379,
- },
- },
- },
-};
-If you installed Redis Sentinel,
module.exports = {
- sms: {
- throttle: {
- /* Redis clustering options */
- datastore: 'ioredis',
- clientOptions: {
- name: 'mymaster',
- sentinels: [{ host: '127.0.0.1', port: 26379 }],
- },
- },
- },
-};
-When NotifyBC is deployed to Kubernetes using Helm, by default throttle, if enabled, uses Redis Sentinel therefore rate limit applies to whole cluster.
To disable throttle, set sms.throttle.enabled to false in file /src/config.local.js
module.exports = {
- sms: {
- throttle: {
- enabled: false,
- },
- },
-};
-SMTP
By default NotifyBC acts as the SMTP server itself and connects directly to recipient's SMTP server. To setup SMTP relay to a host, say smtp.foo.com, add following smtp config object to /src/config.local.js
module.exports = {
- email: {
- smtp: {
- host: 'smtp.foo.com',
- port: 25,
- pool: true,
- tls: {
- rejectUnauthorized: false,
- },
- },
- },
-};
-Throttle
NotifyBC can throttle email requests if SMTP server imposes rate limit. To enable throttle and set rate limit, create following config in file /src/config.local.js
module.exports = {
- email: {
- throttle: {
- enabled: true,
- // minimum request interval in ms
- minTime: 250,
- },
- },
-};
-where
- enabled - whether to enable throttle or not. Default to false.
- minTime - minimum request interval in ms. Example value 250 throttles request rate to 4/sec.
When NotifyBC is deployed from source code, by default the rate limit applies to one Node.js process only. If there are multiple processes, i.e. a cluster, the aggregated rate limit is multiplied by the number of processes. To enforce the rate limit across entire cluster, install Redis and add Redis config to email.throttle
module.exports = {
- email: {
- throttle: {
- enabled: true,
- // minimum request interval in ms
- minTime: 250,
- /* Redis clustering options */
- datastore: 'ioredis',
- clientOptions: {
- host: '127.0.0.1',
- port: 6379,
- },
- },
- },
-};
-If you installed Redis Sentinel,
module.exports = {
- email: {
- throttle: {
- enabled: true,
- // minimum request interval in ms
- minTime: 250,
- /* Redis clustering options */
- datastore: 'ioredis',
- clientOptions: {
- name: 'mymaster',
- sentinels: [{ host: '127.0.0.1', port: 26379 }],
- },
- },
- },
-};
-When NotifyBC is deployed to Kubernetes using Helm, by default throttle, if enabled, uses Redis Sentinel therefore rate limit applies to whole cluster.
Inbound SMTP Server
NotifyBC implemented an inbound SMTP server to handle
In order for the emails from internet to reach the SMTP server, a host where one of the following servers should be listening on port 25 open to internet
- NotifyBC, if it can be installed on such internet-facing host directly; otherwise,
- a tcp proxy server, such as nginx with stream proxy module that can proxy tcp port 25 traffic to backend NotifyBC instances.
Regardless which above option is chosen, you need to config NotifyBC inbound SMTP server by adding following static config email.inboundSmtpServer to file /src/config.local.js
module.exports = {
- email: {
- inboundSmtpServer: {
- enabled: true,
- domain: 'host.foo.com',
- listeningSmtpPort: 25,
- options: {
- // ...
- },
- },
- },
-};
-where
`,9),S=t("stream {
- server {
- listen 25;
- proxy_pass \${INBOUND_SMTP_DOMAIN}:443;
- proxy_ssl on;
- proxy_ssl_verify off;
- proxy_ssl_server_name on;
- proxy_connect_timeout 10s;
- }
-}
-Replace \${INBOUND_SMTP_DOMAIN} with the inbound SMTP server route domain.
Bounce
Bounces, or Non-Delivery Reports (NDRs), are system-generated emails informing sender of failed delivery. NotifyBC can be configured to receive bounces, record bounces, and automatically unsubscribe all subscriptions of a recipient if the number of recorded hard bounces against the recipient exceeds threshold. A deemed successful notification delivery deletes the bounce record.
Although NotifyBC records all bounce emails, not all of them should count towards unsubscription threshold, but rather only the hard bounces - those which indicate permanent unrecoverable errors such as destination address no longer exists. In principle this can be distinguished using smtp response code. In practice, however, there are some challenges to make the distinction
- the smtp response code is not fully standardized and may vary by recipient's smtp server so it's unreliable
- there is no standard smtp header in bounce email to contain smtp response code. Often the response code is embedded in bounce email body.
- the bounce email template varies by sender's smtp server
To mitigate, NotifyBC defines several customizable string pattern filters in terms of regular expression. Only bounce emails matched the filters count towards unsubscription threshold. It's a matter of trial-and-error to get the correct filter suitable to your smtp server.
to improve hard bounce recognition
Send non-existing emails to several external email systems. Inspect the bounce messages for common string patterns. After gone live, review bounce records in web console from time to time to identify new bounce types and decide whether the bounce types qualify as hard bounce. To avoid false positives resulting in premature unsubscription, it is advisable to start with a high unsubscription threshold.
Bounce handling involves four actions
`,9),R={href:"https://en.wikipedia.org/wiki/Variable_envelope_return_path",target:"_blank",rel:"noopener noreferrer"},O=e("em",null,"bn-{subscriptionId}-{unsubscriptionCode}@{inboundSmtpServerDomain}",-1),D=e("em",null,"NotifyBC",-1),z=e("li",null,"when a notification finished dispatching, the dispatching start and end time is recorded to all bounce records matching affects recipient addresses",-1),U=e("li",null,"when inbound smtp server receives a bounce message, it updates the bounce record by saving the message and incrementing the hard bounce count when the message matches the filter criteria. The filter criteria are regular expressions matched against bounce email subject and body, as well as regular expression to extract recipient's email address from bounce email body. It also unsubscribes the user from all subscriptions when the hard bounce count exceeds a predefined threshold.",-1),E=e("li",null,"A cron job runs periodically to delete bounce records if the latest notification is deemed delivered successfully.",-1),q=e("p",null,"To setup bounce handling",-1),A=e("li",null,[e("p",null,[n("set up "),e("a",{href:"#inbound-smtp-server"},"inbound smtp server")])],-1),$=e("li",null,[e("p",null,[n("verify config "),e("em",null,"email.bounce.enabled"),n(" is set to true or absent in "),e("em",null,"/src/config.local.js")])],-1),L=e("em",null,"/src/config.ts",-1),F={href:"https://tools.ietf.org/html/rfc3464",target:"_blank",rel:"noopener noreferrer"},V=t(`module.exports = {
- email: {
- bounce: {
- enabled: true,
- unsubThreshold: 5,
- subjectRegex: '',
- smtpStatusCodeRegex: '5\\\\.\\\\d{1,3}\\\\.\\\\d{1,3}',
- failedRecipientRegex:
- '(?:[a-z0-9!#$%&\\'*+/=?^_\`{|}~-]+(?:\\\\.[a-z0-9!#$%&\\'*+/=?^_\`{|}~-]+)*|"(?:[\\\\x01-\\\\x08\\\\x0b\\\\x0c\\\\x0e-\\\\x1f\\\\x21\\\\x23-\\\\x5b\\\\x5d-\\\\x7f]|\\\\\\\\[\\\\x01-\\\\x09\\\\x0b\\\\x0c\\\\x0e-\\\\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\\\x01-\\\\x08\\\\x0b\\\\x0c\\\\x0e-\\\\x1f\\\\x21-\\\\x5a\\\\x53-\\\\x7f]|\\\\\\\\[\\\\x01-\\\\x09\\\\x0b\\\\x0c\\\\x0e-\\\\x7f])+)\\\\])',
- },
- },
-};
-where
`,2),W=e("li",null,[e("p",null,[e("em",null,"unsubThreshold"),n(" is the threshold of hard bounce count above which the user is unsubscribed from all subscriptions")])],-1),G=e("li",null,[e("p",null,[e("em",null,"subjectRegex"),n(" is the regular expression that bounce message subject must match in order to count towards the threshold. If "),e("em",null,"subjectRegex"),n(" is set to empty string or "),e("em",null,"undefined"),n(", then this filter is disabled.")])],-1),H=e("em",null,"smtpStatusCodeRegex",-1),J={href:"https://tools.ietf.org/html/rfc3463",target:"_blank",rel:"noopener noreferrer"},K=e("ul",null,[e("li",null,[e("em",null,"message/delivery-status")]),e("li",null,"html"),e("li",null,"plain text")],-1),Q=e("em",null,"failedRecipientRegex",-1),X=e("em",null,"failedRecipientRegex",-1),Y=e("em",null,"undefined",-1),Z={href:"https://stackoverflow.com/questions/201323/how-to-validate-an-email-address-using-a-regular-expression",target:"_blank",rel:"noopener noreferrer"},ee=e("ul",null,[e("li",null,[e("em",null,"message/delivery-status")]),e("li",null,"html"),e("li",null,"plain text")],-1),ne=e("h2",{id:"list-unsubscribe-by-email",tabindex:"-1"},[e("a",{class:"header-anchor",href:"#list-unsubscribe-by-email","aria-hidden":"true"},"#"),n(" List-unsubscribe by Email")],-1),se=e("p",null,"Some email clients provide a consistent UI to unsubscribe if an unsubscription email address is supplied. For example, newer iOS built-in email app will display following banner",-1),ae=["src"],te=t(`To support this unsubscription method, NotifyBC implements a custom inbound SMTP server to transform received emails sent to address un-{subscriptionId}-{unsubscriptionCode}@{inboundSmtpServerDomain} to NotifyBC unsubscribing API calls. This unsubscription email address is generated by NotifyBC and set in header List-Unsubscribe of all notification emails.
To enable list-unsubscribe by email
- set up inbound smtp server
- verify config email.listUnsubscribeByEmail.enabled is set to true or absent in /src/config.local.js
To disable list-unsubscribe by email, set email.listUnsubscribeByEmail.enabled to false in /src/config.local.js
module.exports = {
- email: {
- listUnsubscribeByEmail: { enabled: false },
- },
-};
-There are two types of configurations - static and dynamic. Static configurations are defined in files or environment variables, requiring restarting NotifyBC to take effect, whereas dynamic configurations are defined in databases and updates take effect immediately.
Static Configurations
Most static configurations are specified in file /src/config.ts. If you need to change, instead of updating /src/config.ts file, create local file /src/config.local.js or environment specific file /src/config.<env>.js, which is only included when environment variable NODE_ENV equals <env>. Besides js, ts and json file extensions are also supported. The rest of the documentation assumes the file extension is js. Content in these files are deeply merged in following ascending precedence
- default file /src/config.ts
- environment specific file /src/config.<env>.js
- local file /src/config.local.js
Run build script whenever changing file in /src
Every time a file under /src, including config files, is updated, run yarn build before restarting NotifyBC to take effect.
Following configs should be customized per installation
',6),_=e("p",null,"In addition, if installing from source code",-1),y=e("p",null,"Customizing other configs only if needed.",-1),v=e("h2",{id:"dynamic-configurations",tabindex:"-1"},[e("a",{class:"header-anchor",href:"#dynamic-configurations","aria-hidden":"true"},"#"),i(" Dynamic Configurations")],-1),b=e("div",{class:"custom-container tip"},[e("p",{class:"custom-container-title"},"Why Dynamic Configs?"),e("p",null,"Dynamic configs are needed in cases such as"),e("ul",null,[e("li",null,"to allow define service-specific configs such as message templates"),e("li",null,"in a multi-node deployment, configs can be generated by one node (typically master) and shared with other nodes")])],-1);function w(x,C){const n=a("RouterLink");return c(),l("div",null,[f,e("div",u,[m,e("p",null,[i("The document pages in this section cover "),g,i(" app level configurations only. If your "),h,i(" is deployed to Kubernetes using Helm, you can also "),t(n,{to:"/docs/getting-started/installation.html#customizations"},{default:o(()=>[i("customize")]),_:1}),i(" infrastructure level configurations.")])]),p,e("ul",null,[e("li",null,[t(n,{to:"/docs/config/adminIpList.html"},{default:o(()=>[i("Admin IP List")]),_:1})]),e("li",null,[t(n,{to:"/docs/config/reverseProxyIpLists.html"},{default:o(()=>[i("Reverse Proxy IP Lists")]),_:1})]),e("li",null,[t(n,{to:"/docs/config/httpHost.html"},{default:o(()=>[i("HTTP Host")]),_:1})]),e("li",null,[t(n,{to:"/docs/config/email.html#smtp"},{default:o(()=>[i("SMTP")]),_:1})])]),_,e("ul",null,[e("li",null,[t(n,{to:"/docs/config/database.html"},{default:o(()=>[i("Database")]),_:1})]),e("li",null,[t(n,{to:"/docs/config/internalHttpHost.html"},{default:o(()=>[i("Internal HTTP Host")]),_:1})])]),y,v,e("p",null,[i("Dynamic configs are managed using REST "),t(n,{to:"/docs/api-config/"},{default:o(()=>[i("configuration api")]),_:1}),i(".")]),b])}const T=s(d,[["render",w],["__file","index.html.vue"]]);export{T as default}; diff --git a/version/5.0/assets/index.html-4f848009.js b/version/5.0/assets/index.html-4f848009.js deleted file mode 100644 index 960d9c581..000000000 --- a/version/5.0/assets/index.html-4f848009.js +++ /dev/null @@ -1,110 +0,0 @@ -import{_ as l,r,o,c as d,a as e,b as s,d as a,e as t}from"./app-5138f739.js";const p={},u=t(`Administrator
The administrator API provides knowledge factor authentication to identify admin request by access token (aka API token in other literatures) associated with a registered administrator maintained in NotifyBC database. Because knowledge factor authentication is vulnerable to brute-force attack, administrator API based access token authentication is less favorable than admin ip list, client certificate, or OIDC authentication.
Avoid Administrator API
Administrator API was created to circumvent an OpenShift limitation - the source ip of a request initiated from an OpenShift pod cannot be exclusively allocated to the pod's project, rather it has to be shared by all OpenShift projects. Therefore it's difficult to impose granular access control based on source ip.
With the introduction client certificate in v2.4.0, most use cases, if not all, that need Administrator API including the OpenShift use case mentioned above can be addressed by client certificate. Therefore only use Administrator API sparingly as last resort.
To enable access token authentication,
a super-admin signs up an administrator
For example,
curl -X POST "http://localhost:3000/api/administrators" -H "accept: application/json" -H "Content-Type: application/json" -d "{\\"username\\":\\"Foo\\",\\"email\\":\\"user@example.com\\",\\"password\\":\\"secret\\"}" -The step can also be completed in web console using button in Administrators panel.
Either super-admin or the user login to generate an access token
For example,
curl -X POST "http://localhost:3000/api/administrators/login" -H "accept: application/json" -H "Content-Type: application/json" -d "{\\"email\\":\\"user@example.com\\",\\"password\\":\\"secret\\",\\"tokenName\\":\\"myApp\\"}" -The step can also be completed in web console GUI by an anonymous user using button at top right corner. Access token generated by GUI is valid for 12hrs.
Apply access token to either Authorization header or access_token query parameter to make authenticated requests. For example, to get a list of notifications
ACCESS_TOKEN=6Nb2ti5QEXIoDBS5FQGWIz4poRFiBCMMYJbYXSGHWuulOuy0GTEuGx2VCEVvbpBK - -# Authorization Header -curl -X GET -H "Authorization: $ACCESS_TOKEN" \\ -http://localhost:3000/api/notifications - -# Query Parameter -curl -X GET http://localhost:3000/api/notifications?access_token=$ACCESS_TOKEN -In web console, once login as administrator, the access token is automatically applied.
Model Schemas
The Administrator API operates on three related sub-models - Administrator, UserCredential and AccessToken. An administrator has one and only one user credential and zero or more access tokens. Their relationship is diagramed as
`,7),c=["src"],m=t(`Administrator
| Name | Attributes | ||||||
|---|---|---|---|---|---|---|---|
id |
| ||||||
| |||||||
username user name |
|
UserCredential
| Name | Attributes | ||||
|---|---|---|---|---|---|
id |
| ||||
password hashed password |
| ||||
userId foreign key to Administrator model |
|
AccessToken
| Name | Attributes | ||||
|---|---|---|---|---|---|
id 64-byte random alphanumeric characters |
| ||||
userId foreign key to Administrator model |
| ||||
ttl Time-to-live in seconds. If absent, access token never expires. |
| ||||
name Name of the access token. Can be used to identify applications that use the token. |
|
Sign Up
POST /administrators
-This API allows a super-admin to create an admin.
permissions required, one of
- super admin
inputs
user information
{ - "email": "string", - "password": "string", - "username": "string" -} -Password must meet following complexity rules:
- contains at least 10 characters
- contains at least one lower case character a-z
- contains at least one upper case character A-Z
- contains at least one numeric character 0-9
- contains at lease one special character in !_@#$&*
email must be unique. username is optional.
- required: true
- parameter type: request body
- data type: object
outcome
- for super-admin requests,
- an Administrator is generated, populated with email and username
- a UserCredential is generated, populated with hashed password
- Administrator is returned
- forbidden otherwise
- for super-admin requests,
Login
POST /administrators/login
-This API allows an admin to login and create an access token
inputs
user information
{ - "email": "user@example.com", - "password": "string", - "tokenName": "string", - "ttl": 0 -} -tokenName and ttl are optional. If ttl is absent, access token never expires.
- required: true
- parameter type: request body
- data type: object
outcome
- if login is successful
- a new AccessToken is generated with tokenName is saved to AccessToken.name and ttl is saved to AccessToken.ttl.
- the new access token is returned
{ - "token": "string" -} -
- forbidden otherwise
- if login is successful
Set Password
POST /administrators/{id}/user-credential
-This API allows a super-admin or admin to create or update password by id. An admin can only create/update own record.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
password
{ - "password": "string" -} -The password must meet complexity rules specified in Sign Up.
- required: true
- parameter type: request body
- data type: object
- outcome
- for super-admins or admin,
- hash the input password
- remove any existing UserCredential.password for the Administrator
- create a new UserCredential.password
- forbidden otherwise
- for super-admins or admin,
Get Administrators
GET /administrators
-This API allows a super-admin or admin to search for administrators. An admin can only search for own record
`,22),h=e("li",null,[e("p",null,"permissions required, one of"),e("ul",null,[e("li",null,"super admin"),e("li",null,"admin")])],-1),v=e("p",null,"inputs",-1),b=t("a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
",3),k=e("li",null,"URL-encoded stringified JSON object (see example below); or",-1),g={href:"https://github.com/ljharb/qs",target:"_blank",rel:"noopener noreferrer"},q=e("code",null,'?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"',-1),f=t(`Regardless, the filter will have to be parsed into a JSON object conforming to
{
- "where": {...},
- "fields": ...,
- "order": ...,
- "skip": ...,
- "limit": ...,
-}
-All properties are optional. The syntax for each property is documented, respectively
`,3),y=e("em",null,"where",-1),_={href:"https://www.mongodb.com/docs/manual/tutorial/query-documents/",target:"_blank",rel:"noopener noreferrer"},w=e("em",null,"fields",-1),x={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.select()",target:"_blank",rel:"noopener noreferrer"},A=e("em",null,"order",-1),j={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.sort()",target:"_blank",rel:"noopener noreferrer"},T=e("em",null,"skip",-1),P={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.skip/",target:"_blank",rel:"noopener noreferrer"},I=e("em",null,"limit",-1),S={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.limit/",target:"_blank",rel:"noopener noreferrer"},D=t(`outcome
- for super-admins, returns an array of Administrators matching the filter
- for admins, returns an array of one element - own record if the record matches the filter; empty array otherwise
- forbidden otherwise
example
to retrieve administrators created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/administrators?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D'
-the value of the filter query parameter is URL-encoded stringified JSON object
{
- "where": {
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
- }
-}
-Get Administrator Count
GET /administrators/count
-This API allows a super-admin or admin to count administrators by filter. An admin can only count own record therefore the number is at most 1.
`,3),E=e("li",null,[e("p",null,"permissions required, one of"),e("ul",null,[e("li",null,"super admin"),e("li",null,"admin")])],-1),B=e("p",null,"inputs",-1),N=e("em",null,"where",-1),O={href:"https://www.mongodb.com/docs/manual/tutorial/query-documents/",target:"_blank",rel:"noopener noreferrer"},U=e("ul",null,[e("li",null,"parameter name: where"),e("li",null,"required: false"),e("li",null,"parameter type: query"),e("li",null,"data type: object")],-1),$=e("p",null,"The value can be expressed as either",-1),G=e("li",null,"URL-encoded stringified JSON object (see example below); or",-1),L={href:"https://github.com/ljharb/qs",target:"_blank",rel:"noopener noreferrer"},M=e("code",null,'?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"',-1),R=t(`outcome
- for super-admins or admin, a count matching the filter
- forbidden otherwise
example
to retrieve the count of administrators created in year 2023 , run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/administrators/count?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D'
-the value of the where query parameter is URL-encoded stringified JSON object
{
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
-}
-Delete an Administrator
DELETE /administrators/{id}
-This API allows a super-admin or admin to delete administrator by id. An admin can only delete own record.
permissions required, one of
- super admin
- admin
inputs
- Administrator id
- required: true
- parameter type: path
- data type: string
- Administrator id
outcome
- for super-admins or admin
- all AccessToken of the Administrator are deleted
- the corresponding UserCredential is deleted
- the Administrator is deleted
- forbidden otherwise
- for super-admins or admin
Get an Administrator
GET /administrators/{id}
-This API allows a super-admin or admin to get administrator by id. An admin can only get own record.
permissions required, one of
- super admin
- admin
inputs
- Administrator id
- required: true
- parameter type: path
- data type: string
- Administrator id
outcome
- for super-admins or admin, returns the Administrator
- forbidden otherwise
Update an Administrator
PATCH /administrators/{id}
-This API allows a super-admin or admin to update administrator fields by id. An admin can only update own record.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
user information
{ - "username": "string", - "email": "string" -} -- required: true
- parameter type: request body
- data type: object
- outcome
- for super-admins or admin, updates the Administrator
- forbidden otherwise
Replace an Administrator
PUT /administrators/{id}
-This API allows a super-admin or admin to replace administrator records by id. An admin can only replace own record. This API is different from Update an Administrator in that update/patch needs only to contain fields that are changed, ie the delta, whereas replace/put needs to contain all fields to be saved.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
user information
{ - "username": "string", - "email": "string" -} -- required: true
- parameter type: request body
- data type: object
- outcome
- for super-admins or admin, updates the Administrator. If password is also supplied, the password is handled same way as Set Password API
- forbidden otherwise
Get an Administrator's AccessTokens
GET /administrators/{id}/access-tokens
-This API allows a super-admin or admin to get access tokens by Administrator id. An admin can only get own records.
`,21),Q=e("li",null,[e("p",null,"permissions required, one of"),e("ul",null,[e("li",null,"super admin"),e("li",null,"admin")])],-1),X=e("p",null,"inputs",-1),H=e("li",null,[e("p",null,[e("em",null,"Administrator"),s(" id")]),e("ul",null,[e("li",null,"required: true"),e("li",null,"parameter type: path"),e("li",null,"data type: string")])],-1),z=t("a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
",3),F=e("li",null,"URL-encoded stringified JSON object (see example below); or",-1),V={href:"https://github.com/ljharb/qs",target:"_blank",rel:"noopener noreferrer"},K=e("code",null,'?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"',-1),W=t(`Regardless, the filter will have to be parsed into a JSON object conforming to
{
- "where": {...},
- "fields": ...,
- "order": ...,
- "skip": ...,
- "limit": ...,
-}
-All properties are optional. The syntax for each property is documented, respectively
`,3),Y=e("em",null,"where",-1),Z={href:"https://www.mongodb.com/docs/manual/tutorial/query-documents/",target:"_blank",rel:"noopener noreferrer"},ee=e("em",null,"fields",-1),se={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.select()",target:"_blank",rel:"noopener noreferrer"},ne=e("em",null,"order",-1),ae={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.sort()",target:"_blank",rel:"noopener noreferrer"},te=e("em",null,"skip",-1),ie={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.skip/",target:"_blank",rel:"noopener noreferrer"},le=e("em",null,"limit",-1),re={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.limit/",target:"_blank",rel:"noopener noreferrer"},oe=t(`outcome
- for super-admins or admin, a list of AccessTokens matching the filter
- forbidden otherwise
example
to retrieve access tokens created in year 2023 for administrator with id of 1, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/administrators/1/access-tokens?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D'
-the value of the filter query parameter is URL-encoded stringified JSON object
{
- "where": {
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
- }
-}
-Update an Administrator's AccessTokens
PATCH /administrators/{id}/access-tokens
-This API allows a super-admin or admin to update access tokens by Administrator id. An admin can only update own records.
`,3),pe=e("li",null,[e("p",null,"permissions required, one of"),e("ul",null,[e("li",null,"super admin"),e("li",null,"admin")])],-1),ue=e("p",null,"inputs",-1),ce=e("li",null,[e("p",null,[e("em",null,"Administrator"),s(" id")]),e("ul",null,[e("li",null,"required: true"),e("li",null,"parameter type: path"),e("li",null,"data type: string")])],-1),me=e("em",null,"where",-1),he={href:"https://www.mongodb.com/docs/manual/tutorial/query-documents/",target:"_blank",rel:"noopener noreferrer"},ve=e("ul",null,[e("li",null,"parameter name: where"),e("li",null,"required: false"),e("li",null,"parameter type: query"),e("li",null,"data type: object")],-1),be=e("p",null,"The value can be expressed as either",-1),ke=e("li",null,"URL-encoded stringified JSON object (see example below); or",-1),ge={href:"https://github.com/ljharb/qs",target:"_blank",rel:"noopener noreferrer"},qe=e("code",null,'?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"',-1),fe=t(`AccessToken information
{
- "ttl": 0,
- "name": "string"
-}
-- required: true
- parameter type: request body
- data type: object
outcome
- for super-admins or admin, success count
- forbidden otherwise
example
to set ttl token to 0 for all access tokens created in year 2023 for administrator with id 1, run
curl -X PATCH --header 'Content-Type: application/json' 'http://localhost:3000/api/administrators/1/access-tokens?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D' -d '{"ttl":0}'
-the value of the where query parameter is URL-encoded stringified JSON object
{
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
-}
-Create an Administrator's AccessToken
POST /administrators/{id}/access-tokens
-This API allows a super-admin or admin to create an access token by Administrator id. An admin can only create own records.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
AccessToken information
{ - "ttl": 0, - "name": "string" -} -- required: true
- parameter type: request body
- data type: object
- outcome
- for super-admins or admin
- Create and save AccessToken
- return AccessToken created
- forbidden otherwise
- for super-admins or admin
Delete an Administrator's AccessTokens
DELETE /administrators/{id}/access-tokens
-This API allows a super-admin or admin to delete access tokens by Administrator id. An admin can only delete own records.
`,8),we=e("li",null,[e("p",null,"permissions required, one of"),e("ul",null,[e("li",null,"super admin"),e("li",null,"admin")])],-1),xe=e("p",null,"inputs",-1),Ae=e("li",null,[e("p",null,[e("em",null,"Administrator"),s(" id")]),e("ul",null,[e("li",null,"required: true"),e("li",null,"parameter type: path"),e("li",null,"data type: string")])],-1),je=e("em",null,"where",-1),Te={href:"https://www.mongodb.com/docs/manual/tutorial/query-documents/",target:"_blank",rel:"noopener noreferrer"},Pe=e("ul",null,[e("li",null,"parameter name: where"),e("li",null,"required: false"),e("li",null,"parameter type: query"),e("li",null,"data type: object")],-1),Ie=e("p",null,"The value can be expressed as either",-1),Se=e("li",null,"URL-encoded stringified JSON object (see example below); or",-1),De={href:"https://github.com/ljharb/qs",target:"_blank",rel:"noopener noreferrer"},Ce=e("code",null,'?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"',-1),Ee=t(`outcome
- for super-admins or admin
- delete all AccessToken under the Administrator matching the filter
- return success count
- forbidden otherwise
example
to delete all access tokens created in year 2023 for administrator with id 1, run
curl -X DELETE --header 'Accept: application/json' 'http://localhost:3000/api/administrators/1/access-tokens?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D'
-the value of the where query parameter is URL-encoded stringified JSON object
{
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
-}
-curl -X GET 'http://localhost:3000/api/configurations?filter=%7B%22where%22%3A%20%7B%22name%22%3A%20%22rsa%22%7D%7D'
-or you can open API explorer, expand GET /configurations and set filter to
{"where": {"name": "rsa"}}
-The response should be something like
[
- {
- "name": "rsa",
- "value": {
- "private": "-----BEGIN RSA PRIVATE KEY-----\\nMIIEpgIBAAKCAQEA8Hl+/cF3AOxKVRHtZpeSDM+LLGc2hkDkKxRXe72maUAzDUoO\\noNd6wd02Cf6iO7kj0RSDHXUyINxCgvXy2Q7gME4zRN5WG4ItWZ7FITeNgJJW1r+J\\nshDjTwKVpMvcKHy0vyUl25ah7hnwGK6PbJvFWMmtIBw6Rs5DaERAlmilgkuUgdri\\naA4YhhS4pCJLvO2p9wZd+dLWUT+tpsOZGeecC8If3fyShgrocMbd8pYYDzf65oCt\\nVaLaNdERaIJSDcmbHxFpeBdEQEzxw2qRPbUCnSgQb8cVFLJ2eOEn5LylWhU96A1S\\n3w1IlRm5N2zG0En58Vruo26gEtl5KFu0zivlawIDAQABAoIBAQCAawFsFcKtVYIk\\nh9xVax/tg2/5GG0/qKuwbb6CMDcMAeLBeAjzz96YZL+U+sw8RJRh9ShHtOw+LCHA\\nugMj8xO5+Cjc4DbvnccGEwmGwZnpTTzelY687tPUv7aWON+rJ12GrhnXeEulUWis\\nZZvmDhGHZrvzZ9+fLEtHBRvQtrWcLCN0G5l1Z1KEWUj23vn1HZpfNvqigIbC05Pq\\nWUewRZShHUklhzky6DwLklWUKv2951ypd5CHhYfXn0eXjeyqcoYeZzoCSGqtvZar\\nVVOCPBKPn3cLZVKzYd02WO4CV07SpHCBtYPWf4OvXbOY6wV1Vc92S0K+ijASDDc0\\nB7Vjgb8RAoGBAPg4dSbn9GWNHydveidi2Zt4kftEW18C9xHbJ3t+QkhpLjq2kwcY\\no0iOWkEd4d1l0lKAVanBQazrazKiSyq1PJSJDyL3osHItA7Twq+gCXOfXw/0LbJh\\napK5DH3S2ZTM42wOdZLYIHvSqRuYUmnzhy9+Ads87b/ICCctUMCLz1afAoGBAPgC\\n4/zE/Au/A3wb48AywfmJ5kqPO0V7lqLrn/aBwdF1H/DHQ95cSuKrTEIysZxz52bh\\n7mAHjnWnY4zFNaUvcruHw78NOxUJvje8cDIUsrTefh+qmctiGR119z7iso9FlsxR\\np/o5BVT/K8q76xtkpOln2A0rc8sBNwtCoeeUzfm1AoGBAInH/O99raF49iQTswCN\\n1DCCerW4uedBZBebSI06BlzfVXPtyCsWN/ycV+jxR2B3lomJBwPVbDkp7DUM9SBd\\nvaTNd4N3ZfafC6N3VAfck6KEgmX+qibsABY1dYOaOIBqQorGc+jw4wcYZhoVMRny\\nvcVU8n7ZkTb1N+FXPA3FDXANAoGBAOuSg0/TI71cgEjgjOJA1DLco1vq1NfY3mp9\\n+QFCmwEDiYVBINwTOiY3o0W1tTLwfLoinDOmudBTYKGTqLLwcMBj4rCUNqxzBrUW\\nTlOjiWN3esFFYLPoyAZNyL14wzaHWQdWAIISq1fi0IvPFzB71pDFTFimD2SiENCn\\nR/YaR9OJAoGBAM21MRvTEMHF/EvqZ/X6t2zm9dtA22L2LeVy68aEdo82F/1RFvCM\\nGBWjGS7G7fXk/tV/YHbjibhgktvLu3Rss1wlHfGEjtDAIdp9dqH0cNxMgy/eTfoy\\nFfzV3l7pNSdILn1bNqoMz9CaYK7CGIYpBWCbRJlRSYw2FHJwl5tzgmkk\\n-----END RSA PRIVATE KEY-----",
- "public": "-----BEGIN PUBLIC KEY-----\\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8Hl+/cF3AOxKVRHtZpeS\\nDM+LLGc2hkDkKxRXe72maUAzDUoOoNd6wd02Cf6iO7kj0RSDHXUyINxCgvXy2Q7g\\nME4zRN5WG4ItWZ7FITeNgJJW1r+JshDjTwKVpMvcKHy0vyUl25ah7hnwGK6PbJvF\\nWMmtIBw6Rs5DaERAlmilgkuUgdriaA4YhhS4pCJLvO2p9wZd+dLWUT+tpsOZGeec\\nC8If3fyShgrocMbd8pYYDzf65oCtVaLaNdERaIJSDcmbHxFpeBdEQEzxw2qRPbUC\\nnSgQb8cVFLJ2eOEn5LylWhU96A1S3w1IlRm5N2zG0En58Vruo26gEtl5KFu0zivl\\nawIDAQAB\\n-----END PUBLIC KEY-----"
- },
- "id": "591cda5d6c7adec42a1874bc",
- "updated": "2017-05-17T23:18:53.385Z"
- }
-]
-The public key is the string -----BEGIN PUBLIC KEY-----...-----END PUBLIC KEY-----
Model Schema
The API operates on following data model fields:
| Name | Attributes | ||||
|---|---|---|---|---|---|
channel name of the delivery channel. Valid values: email, sms. |
| ||||
userChannelId user's delivery channel id, for example, email address. |
| ||||
hardBounceCount number of hard bounces recorded so far |
| ||||
state bounce record state. Valid values: active, deleted. |
| ||||
bounceMessages array of recorded bounce messages. Each element is an object containing the date bounce message was received and the message itself. |
| ||||
latestNotificationStarted latest notification started date. |
| ||||
latestNotificationEnded latest notification ended date. |
| ||||
created date and time bounce record was created |
| ||||
updated date and time of bounce record was last updated |
| ||||
id config id |
|
- features such as GraphQL have been in experimental state for years
- recent commits are mostly chores rather than enhancements
- core developers have ceased to contribute
To pave the way for future growth, switching platform becomes necessary. NestJS was chosen because
- both NestJS and Loopback are server-side Node.js frameworks
- NestJS has the closest feature set as Loopback. To a large extent NestJS is a superset of Loopback
- NestJS incorporates more technologies
/src/middleware.ts contains following default middleware settings
import path from 'path';
-module.exports = {
- all: {
- compression: {},
- },
- apiOnly: {
- helmet: {},
- morgan: {
- params: [
- ':remote-addr - :remote-user [:date[clf]] ":method :url HTTP/:http-version" :status ":req[X-Forwarded-For]"',
- ],
- enabled: false,
- },
- },
-};
-/src/middleware.ts has following structure
module.exports = {
- all: {
- '<middlewareName>': {params: [], enabled: <boolean>},
- },
- apiOnly: {
- '<middlewareName>': {params: [], enabled: <boolean>},
- },
-};
-Middleware defined under all applies to both API and web console requests, as opposed to apiOnly, which applies to API requests only. params are passed to middleware function as arguments. enabled toggles the middleware on or off.
To change default settings defined in /src/middleware.ts, create file /src/middleware.local.ts or /src/middleware.<env>.ts to override. For example, to enable access log,
module.exports = {
- apiOnly: {
- morgan: {
- enabled: true,
- },
- },
-};
-- \\n
- Anonymous or authenticated subscriptions \\n
- Push and in-app pull notifications \\n
- Email and SMS push notification channels \\n
- Unicast and broadcast message types \\n
- Broadcast push notification filter rules specifiable by both sender and subscriber \\n
- Notification auto-gen from RSS \\n
- \\n
- Handles common backend business logic only, allowing site developer implement frontend UI using widgets native to the site\\n \\n
- Loose coupling - interacts with user browser or other server components through RESTful API\\n \\n
- \\n
- Support end-to-end encryption\\n \\n
- Multiple authentication strategies including client certificate for server-server and OIDC for user-server \\n
- Resilient to node failures \\n
© 2016-present under the terms of the Apache License, Version 2.0.\\n","footerHtml":true,"head":[["title",{},"NotifyBC | A versatile notification API server"]]},"headers":[],"git":{},"filePathRelative":"index.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-6b858d80.js b/version/5.0/assets/index.html-6b858d80.js deleted file mode 100644 index 34152cfae..000000000 --- a/version/5.0/assets/index.html-6b858d80.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse('{"key":"v-23f62a3a","path":"/docs/config-email/","title":"Email","lang":"en-US","frontmatter":{"permalink":"/docs/config-email/"},"headers":[{"level":2,"title":"SMTP","slug":"smtp","link":"#smtp","children":[]},{"level":2,"title":"Throttle","slug":"throttle","link":"#throttle","children":[]},{"level":2,"title":"Inbound SMTP Server","slug":"inbound-smtp-server","link":"#inbound-smtp-server","children":[{"level":3,"title":"TCP Proxy Server","slug":"tcp-proxy-server","link":"#tcp-proxy-server","children":[]}]},{"level":2,"title":"Bounce","slug":"bounce","link":"#bounce","children":[]},{"level":2,"title":"List-unsubscribe by Email","slug":"list-unsubscribe-by-email","link":"#list-unsubscribe-by-email","children":[]}],"git":{},"filePathRelative":"docs/config/email.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-6bb621c5.js b/version/5.0/assets/index.html-6bb621c5.js deleted file mode 100644 index 2f651abf3..000000000 --- a/version/5.0/assets/index.html-6bb621c5.js +++ /dev/null @@ -1 +0,0 @@ -const i=JSON.parse(`{"key":"v-391365f4","path":"/docs/config-overview/","title":"Configuration Overview","lang":"en-US","frontmatter":{"permalink":"/docs/config-overview/","prev":"/docs/what's-new/"},"headers":[{"level":2,"title":"Static Configurations","slug":"static-configurations","link":"#static-configurations","children":[]},{"level":2,"title":"Dynamic Configurations","slug":"dynamic-configurations","link":"#dynamic-configurations","children":[]}],"git":{},"filePathRelative":"docs/config/overview.md"}`);export{i as data}; diff --git a/version/5.0/assets/index.html-6ca7c7e1.js b/version/5.0/assets/index.html-6ca7c7e1.js deleted file mode 100644 index 84eecd9d1..000000000 --- a/version/5.0/assets/index.html-6ca7c7e1.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse('{"key":"v-3cf0fa66","path":"/docs/conduct/","title":"Code of Conduct","lang":"en-US","frontmatter":{"permalink":"/docs/conduct/","editable":false,"prev":"/docs/upgrade/"},"headers":[],"git":{},"filePathRelative":"docs/meta/conduct.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-6e94f439.js b/version/5.0/assets/index.html-6e94f439.js deleted file mode 100644 index 9ea70b0ce..000000000 --- a/version/5.0/assets/index.html-6e94f439.js +++ /dev/null @@ -1 +0,0 @@ -const t=JSON.parse('{"key":"v-4cf2565c","path":"/docs/config-internalHttpHost/","title":"Internal HTTP Host","lang":"en-US","frontmatter":{"permalink":"/docs/config-internalHttpHost/"},"headers":[],"git":{},"filePathRelative":"docs/config/internalHttpHost.md"}');export{t as data}; diff --git a/version/5.0/assets/index.html-70c61dd5.js b/version/5.0/assets/index.html-70c61dd5.js deleted file mode 100644 index b62385610..000000000 --- a/version/5.0/assets/index.html-70c61dd5.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse('{"key":"v-04b96fc8","path":"/docs/api-overview/","title":"API Overview","lang":"en-US","frontmatter":{"permalink":"/docs/api-overview/","prev":"/docs/config-certificates/"},"headers":[],"git":{},"filePathRelative":"docs/api/overview.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-7646913e.js b/version/5.0/assets/index.html-7646913e.js deleted file mode 100644 index 387675881..000000000 --- a/version/5.0/assets/index.html-7646913e.js +++ /dev/null @@ -1 +0,0 @@ -import{_ as n,r as l,o as s,c as a,a as e,b as r,d as o}from"./app-5138f739.js";const i={},_=e("h1",{id:"acknowledgments",tabindex:"-1"},[e("a",{class:"header-anchor",href:"#acknowledgments","aria-hidden":"true"},"#"),r(" Acknowledgments")],-1),c=e("p",null,[e("em",null,"NotifyBC"),r(" is built on a myriad of open source software. At runtime it also depends on a few services. Credit goes to their contributors. Notably")],-1),h={href:"https://nodejs.org/",target:"_blank",rel:"noopener noreferrer"},d={href:"https://nestjs.com/",target:"_blank",rel:"noopener noreferrer"},f={href:"https://www.mongodb.com/",target:"_blank",rel:"noopener noreferrer"},p={href:"https://nodemailer.com/",target:"_blank",rel:"noopener noreferrer"},u={href:"https://jmespath.org/",target:"_blank",rel:"noopener noreferrer"},m={href:"https://vuejs.org/",target:"_blank",rel:"noopener noreferrer"},g={href:"https://vuetifyjs.com/",target:"_blank",rel:"noopener noreferrer"},b={href:"https://github.com/json-editor/json-editor",target:"_blank",rel:"noopener noreferrer"},w={href:"https://vuepress.vuejs.org/",target:"_blank",rel:"noopener noreferrer"},k={href:"https://www.twilio.com/",target:"_blank",rel:"noopener noreferrer"},j={href:"https://www.swiftsmsgateway.com/",target:"_blank",rel:"noopener noreferrer"},x={href:"https://bitnami.com/",target:"_blank",rel:"noopener noreferrer"};function N(v,y){const t=l("ExternalLinkIcon");return s(),a("div",null,[_,c,e("ul",null,[e("li",null,[e("a",h,[r("Node.js"),o(t)])]),e("li",null,[e("a",d,[r("NestJS"),o(t)])]),e("li",null,[e("a",f,[r("MongoDB"),o(t)])]),e("li",null,[e("a",p,[r("NodeMailer"),o(t)])]),e("li",null,[e("a",u,[r("JMESPath"),o(t)])]),e("li",null,[e("a",m,[r("Vue"),o(t)])]),e("li",null,[e("a",g,[r("Vuetify"),o(t)])]),e("li",null,[e("a",b,[r("JSON Editor"),o(t)])]),e("li",null,[e("a",w,[r("VuePress"),o(t)])]),e("li",null,[e("a",k,[r("Twilio"),o(t)])]),e("li",null,[e("a",j,[r("Swift SMS Gateway"),o(t)])]),e("li",null,[e("a",x,[r("Bitnami"),o(t)])])])])}const S=n(i,[["render",N],["__file","index.html.vue"]]);export{S as default}; diff --git a/version/5.0/assets/index.html-76d114a5.js b/version/5.0/assets/index.html-76d114a5.js deleted file mode 100644 index 0cad61091..000000000 --- a/version/5.0/assets/index.html-76d114a5.js +++ /dev/null @@ -1 +0,0 @@ -const t=JSON.parse('{"key":"v-a20dfce8","path":"/docs/web-console/","title":"Web Console","lang":"en-US","frontmatter":{"permalink":"/docs/web-console/"},"headers":[{"level":2,"title":"Ip whitelisting authentication","slug":"ip-whitelisting-authentication","link":"#ip-whitelisting-authentication","children":[]},{"level":2,"title":"Client certificate authentication","slug":"client-certificate-authentication","link":"#client-certificate-authentication","children":[]},{"level":2,"title":"Anonymous","slug":"anonymous","link":"#anonymous","children":[]},{"level":2,"title":"Access token authentication","slug":"access-token-authentication","link":"#access-token-authentication","children":[]},{"level":2,"title":"OIDC authentication","slug":"oidc-authentication","link":"#oidc-authentication","children":[]},{"level":2,"title":"SiteMinder authentication","slug":"siteminder-authentication","link":"#siteminder-authentication","children":[]}],"git":{},"filePathRelative":"docs/getting-started/web-console.md"}');export{t as data}; diff --git a/version/5.0/assets/index.html-7d2ae716.js b/version/5.0/assets/index.html-7d2ae716.js deleted file mode 100644 index ecec1a9b2..000000000 --- a/version/5.0/assets/index.html-7d2ae716.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse('{"key":"v-14ac19b5","path":"/help/","title":"","lang":"en-US","frontmatter":{},"headers":[{"level":2,"title":"Getting Help","slug":"getting-help","link":"#getting-help","children":[{"level":3,"title":"Documentation","slug":"documentation","link":"#documentation","children":[]},{"level":3,"title":"View source","slug":"view-source","link":"#view-source","children":[]},{"level":3,"title":"Google","slug":"google","link":"#google","children":[]},{"level":3,"title":"Outstanding issues and requests","slug":"outstanding-issues-and-requests","link":"#outstanding-issues-and-requests","children":[]}]}],"git":{},"filePathRelative":"help/index.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-7dc4890d.js b/version/5.0/assets/index.html-7dc4890d.js deleted file mode 100644 index 62ad1ab05..000000000 --- a/version/5.0/assets/index.html-7dc4890d.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse(`{"key":"v-e5065f60","path":"/docs/api-administrator/","title":"Administrator","lang":"en-US","frontmatter":{"permalink":"/docs/api-administrator/"},"headers":[{"level":2,"title":"Model Schemas","slug":"model-schemas","link":"#model-schemas","children":[{"level":3,"title":"Administrator","slug":"administrator-1","link":"#administrator-1","children":[]},{"level":3,"title":"UserCredential","slug":"usercredential","link":"#usercredential","children":[]},{"level":3,"title":"AccessToken","slug":"accesstoken","link":"#accesstoken","children":[]}]},{"level":2,"title":"Sign Up","slug":"sign-up","link":"#sign-up","children":[]},{"level":2,"title":"Login","slug":"login","link":"#login","children":[]},{"level":2,"title":"Set Password","slug":"set-password","link":"#set-password","children":[]},{"level":2,"title":"Get Administrators","slug":"get-administrators","link":"#get-administrators","children":[]},{"level":2,"title":"Get Administrator Count","slug":"get-administrator-count","link":"#get-administrator-count","children":[]},{"level":2,"title":"Delete an Administrator","slug":"delete-an-administrator","link":"#delete-an-administrator","children":[]},{"level":2,"title":"Get an Administrator","slug":"get-an-administrator","link":"#get-an-administrator","children":[]},{"level":2,"title":"Update an Administrator","slug":"update-an-administrator","link":"#update-an-administrator","children":[]},{"level":2,"title":"Replace an Administrator","slug":"replace-an-administrator","link":"#replace-an-administrator","children":[]},{"level":2,"title":"Get an Administrator's AccessTokens","slug":"get-an-administrator-s-accesstokens","link":"#get-an-administrator-s-accesstokens","children":[]},{"level":2,"title":"Update an Administrator's AccessTokens","slug":"update-an-administrator-s-accesstokens","link":"#update-an-administrator-s-accesstokens","children":[]},{"level":2,"title":"Create an Administrator's AccessToken","slug":"create-an-administrator-s-accesstoken","link":"#create-an-administrator-s-accesstoken","children":[]},{"level":2,"title":"Delete an Administrator's AccessTokens","slug":"delete-an-administrator-s-accesstokens","link":"#delete-an-administrator-s-accesstokens","children":[]}],"git":{},"filePathRelative":"docs/api/administrator.md"}`);export{e as data}; diff --git a/version/5.0/assets/index.html-83738b5b.js b/version/5.0/assets/index.html-83738b5b.js deleted file mode 100644 index 39ae6c991..000000000 --- a/version/5.0/assets/index.html-83738b5b.js +++ /dev/null @@ -1,4 +0,0 @@ -import{_ as e,o as t,c as s,e as n}from"./app-5138f739.js";const a={},o=n(`
HTTP Host
httpHost config sets the fallback http host used by
- mail merge token substitution
- internal HTTP requests spawned by NotifyBC
httpHost can be overridden by other configs or data. For example
- internalHttpHost config
- httpHost field in a notification
There are contexts where there is no alternatives to httpHost. Therefore this config should be defined.
Define the config, which has no default value, in /src/config.local.js
module.exports = {
- httpHost: 'http://foo.com',
-};
-By default trustedReverseProxyIps is empty and siteMinderReverseProxyIps contains only localhost as defined in /src/config.ts
module.exports = {
- siteMinderReverseProxyIps: ['127.0.0.1'],
-};
-To modify, add following objects to file /src/config.local.js
module.exports = {
- siteMinderReverseProxyIps: ['130.32.12.0'],
- trustedReverseProxyIps: ['172.17.0.0/16'],
-};
-The rule to determine if the incoming request is authenticated by SiteMinder is
`,5),b={href:"https://expressjs.com/en/guide/behind-proxies.html",target:"_blank",rel:"noopener noreferrer"},g=e("li",null,[s("if the real client ip is contained in "),e("em",null,"siteMinderReverseProxyIps"),s(", then the request is from SiteMinder, and its SiteMinder headers are trusted; otherwise, the request is considered as directly from internet, and its SiteMinder headers are ignored.")],-1);function x(y,S){const n=i("ExternalLinkIcon");return o(),r("div",null,[c,e("p",null,[s("SiteMinder, being a gateway approached SSO solution, expects the backend HTTP access point of the web sites it protests to be firewall restricted, otherwise the SiteMinder injected HTTP headers can be easily spoofed. However, the restriction cannot be easily implemented on PAAS such as OpenShift. To mitigate, two configuration objects are introduced to create an application-level firewall, both are arrays of ip addresses in the format of "),e("a",d,[s("dot-decimal"),t(n)]),s(" or "),e("a",u,[s("CIDR"),t(n)]),s(" notation")]),e("ul",null,[h,e("li",null,[m,s(" contains a list of ips or ranges of trusted reverse proxies. If "),f,s(" is placed behind SiteMinder Web Agents, then trusted reverse proxies should include only those between SiteMinder Web Agents and "),_,s(" application. When running on OpenShift, this is usually the OpenShift router. Express.js "),e("a",v,[s("trust proxy"),t(n)]),s(" is set to this config object.")])]),k,e("ol",null,[e("li",null,[s("obtain the real client ip address by filtering out trusted proxy ips according to "),e("a",b,[s("Express behind proxies"),t(n)])]),g])])}const M=a(l,[["render",x],["__file","index.html.vue"]]);export{M as default}; diff --git a/version/5.0/assets/index.html-8b37e8b3.js b/version/5.0/assets/index.html-8b37e8b3.js deleted file mode 100644 index 7e400106e..000000000 --- a/version/5.0/assets/index.html-8b37e8b3.js +++ /dev/null @@ -1,16 +0,0 @@ -import{_ as c,r as t,o as l,c as p,a as e,d as s,w as i,b as a,e as r}from"./app-5138f739.js";const m={},d=r(`TLS Certificates
NotifyBC supports HTTPS TLS to achieve end-to-end encryption. In addition, both server and client can be authenticated using certificates.
To enable HTTPS for server authentication only, you need to create two files
- server/certs/key.pem - a PEM encoded private key
- server/certs/cert.pem - a PEM encoded X.509 certificate chain
Use ConfigMaps on Kubernetes
Create key.pem and cert.pem as items in ConfigMap notify-bc, then mount the items under /home/node/app/server/certs similar to how config.local.js and middleware.local.js are implemented.
For self-signed certificate, run
openssl req -x509 -newkey rsa:4096 -keyout server/certs/key.pem -out server/certs/cert.pem -nodes -days 365 -subj "/CN=NotifyBC"
-to generate both files in one shot.
Caution about self-signed cert
Self-signed cert is intended to be used in non-production environments only to authenticate server. In such environments to allow NotifyBC connecting to itself, environment variable NODE_TLS_REJECT_UNAUTHORIZED must be set to 0.
To create a CSR from the private key generated above, run
openssl req -new -key server/certs/key.pem -out server/certs/csr.pem
-Then bring your CSR to your CA to sign. Replace server/certs/cert.pem with the cert signed by CA. If your CA also supplied intermediate certificate in PEM encoded format, say in a file called intermediate.pem, append all of the content of intermediate.pem to file server/certs/cert.pem.
Make a copy of self-signed server/certs/cert.pem
If you want to enable client certificate authentication documented below, make sure to copy self-signed server/certs/cert.pem to server/certs/ca.pem before replacing the file with the cert signed by CA. You need the self-signed server/certs/cert.pem to sign client CSR.
In case you created server/certs/key.pem and server/certs/cert.pem but don't want to enable HTTPS, create following config in src/config.local.js
module.exports = {
- tls: {
- enabled: false,
- },
-};
-Client certificate authentication
After enabling HTTPS, you can further configure such that a client request can be authenticated using client certificate. To do so, copy self-signed server/certs/cert.pem to server/certs/ca.pem. You will use your server key to sign client certificate CSR, and advertise server/certs/ca.pem as acceptable CAs during TLS handshake.
Assuming a client's CSR file is named myClientApp_csr.pem, to sign the CSR
openssl x509 -req -in myClientApp_csr.pem -CA server/certs/ca.pem -CAkey server/certs/key.pem -out myClientApp_cert.pem -set_serial 01 -days 365
-Then give myClientApp_cert.pem to the client. How a client app supplies the client certificate when making a request to NotifyBC varies by client type. Usually the client first needs to bundle the signed client cert and client key into PKCS#12 format
openssl pkcs12 -export -clcerts -in myClientApp_cert.pem -inkey myClientApp_key.pem -out myClientApp.p12
-To use myClientApp.p12, for cURL,
curl --insecure --cert myClientApp.p12 --cert-type p12 https://localhost:3000/api/administrators/whoami
-For browsers, check browser's instruction how to import myClientApp.p12. When browser accessing NotifyBC API endpoints such as https://localhost:3000/api/administrators/whoami, the browser will prompt to choose from a list certificates that are signed by the server certificate.
In case you created server/certs/ca.pem but don't want to enable client certificate authentication, create following config in src/config.local.js
module.exports = {
- tls: {
- clientCertificateEnabled: false,
- },
-};
-module.exports = {
- adminIps: ['127.0.0.1'],
-};
-to modify, create config object adminIps with updated list in file /src/config.local.js instead. For example, to add ip range 192.168.0.0/24 to the list
module.exports = {
- adminIps: ['127.0.0.1', '192.168.0.0/24'],
-};
-It should be noted that NotifyBC may generate http requests sending to itself. These http requests are expected to be admin requests. If you have created an app cluster such as in Kubernetes, you should add the cluster ip range to adminIps. In Kubernetes, this ip range is a private ip range. For example, in BCGov's OpenShift cluster OCP4, the ip range starts with octet 10.
`,4);function _(f,g){const n=t("RouterLink");return i(),o("div",null,[d,e("p",null,[s("By "),p(n,{to:"/docs/overview/#architecture"},{default:c(()=>[s("design")]),_:1}),s(", "),u,s(" classifies incoming requests into four types. For a request to be classified as super-admin, the request's source ip must be in admin ip list. By default, the list contains "),m,s(" only as defined by "),h,s(" in "),v]),k])}const x=a(r,[["render",_],["__file","index.html.vue"]]);export{x as default}; diff --git a/version/5.0/assets/index.html-a9eb55a1.js b/version/5.0/assets/index.html-a9eb55a1.js deleted file mode 100644 index 7542f703d..000000000 --- a/version/5.0/assets/index.html-a9eb55a1.js +++ /dev/null @@ -1 +0,0 @@ -const i=JSON.parse('{"key":"v-b6a1f058","path":"/docs/config-notification/","title":"Notification","lang":"en-US","frontmatter":{"permalink":"/docs/config-notification/"},"headers":[{"level":2,"title":"RSS Feeds","slug":"rss-feeds","link":"#rss-feeds","children":[]},{"level":2,"title":"Broadcast Push Notification Task Concurrency","slug":"broadcast-push-notification-task-concurrency","link":"#broadcast-push-notification-task-concurrency","children":[]},{"level":2,"title":"Broadcast Push Notification Custom Filter Functions","slug":"broadcast-push-notification-custom-filter-functions","link":"#broadcast-push-notification-custom-filter-functions","children":[]},{"level":2,"title":"Guaranteed Broadcast Push Dispatch Processing","slug":"guaranteed-broadcast-push-dispatch-processing","link":"#guaranteed-broadcast-push-dispatch-processing","children":[{"level":3,"title":"Also log skipped dispatches for broadcast push notifications","slug":"also-log-skipped-dispatches-for-broadcast-push-notifications","link":"#also-log-skipped-dispatches-for-broadcast-push-notifications","children":[]}]}],"git":{},"filePathRelative":"docs/config/notification.md"}');export{i as data}; diff --git a/version/5.0/assets/index.html-b2f7f58b.js b/version/5.0/assets/index.html-b2f7f58b.js deleted file mode 100644 index edd81aa3f..000000000 --- a/version/5.0/assets/index.html-b2f7f58b.js +++ /dev/null @@ -1,72 +0,0 @@ -import{_ as c,r as p,o as r,c as l,a as n,b as s,d as e,w as i,e as t}from"./app-5138f739.js";const u={},d=t('Notification
Configs in this section customize the handling of notification request or generating notifications from RSS feeds. They are all sub-properties of config object notification. Service-agnostic configs are static and service-dependent configs are dynamic.
RSS Feeds
NotifyBC can generate broadcast push notifications automatically by polling RSS feeds periodically and detect changes by comparing with an internally maintained history list. The polling frequency, RSS url, RSS item change detection criteria, and message template can be defined in dynamic configs.
Only first page is retrieved for paginated RSS feeds
If a RSS feed is paginated, NotifyBC only retrieves the first page rather than auto-fetch subsequent pages. Hence paginated RSS feeds should be sorted descendingly by last modified timestamp. Refresh interval should be adjusted small enough such that all new or updated items are contained in first page.
{
- "name": "notification",
- "serviceName": "myService",
- "value": {
- "rss": {
- "url": "http://my-serivce/rss",
- "timeSpec": "* * * * *",
- "itemKeyField": "guid",
- "outdatedItemRetentionGenerations": 1,
- "includeUpdatedItems": true,
- "fieldsToCheckForUpdate": ["title", "pubDate", "description"]
- },
- "httpHost": "http://localhost:3000",
- "messageTemplates": {
- "email": {
- "from": "no_reply@invlid.local",
- "subject": "{title}",
- "textBody": "{description}",
- "htmlBody": "{description}"
- },
- "sms": {
- "textBody": "{description}"
- }
- }
- }
-}
-The config items in the value field are
`,2),f=n("li",null,"url: RSS url",-1),k=n("a",{name:"timeSpec"},null,-1),v={href:"https://www.freebsd.org/cgi/man.cgi?crontab(5)",target:"_blank",rel:"noopener noreferrer"},g={href:"https://github.com/kelektiv/node-cron#cron-ranges",target:"_blank",rel:"noopener noreferrer"},y=t("Broadcast Push Notification Task Concurrency
To achieve horizontal scaling, when a broadcast push notification request, hereby known as original request, is received, NotifyBC divides subscribers into chunks and generates a HTTP sub-request for each chunk. The original request supervises the execution of sub-requests. The chunk size is defined by config broadcastSubscriberChunkSize. All subscribers in a sub-request chunk are processed concurrently when the sub-requests are submitted.
The original request submits sub-requests back to (preferably load-balanced) NotifyBC server cluster for processing. Sub-request submission is throttled by config broadcastSubRequestBatchSize. broadcastSubRequestBatchSize defines the upper limit of the number of Sub-requests that can be processed at any given time.
As an example, assuming the total number of subscribers for a notification is 1,000,000, broadcastSubscriberChunkSize is 1,000 and broadcastSubRequestBatchSize is 10, NotifyBC will divide the 1M subscribers into 1,000 chunks and generates 1,000 sub-requests, one for each chunk. The 1,000 sub-requests will be submitted back to NotifyBC cluster to be processed. The original request will ensure at most 10 sub-requests are submitted and being processed at any given time. In fact, the only time concurrency is less than 10 is near the end of the task when remaining sub-requests is less than 10. When a sub-request is received by NotifyBC cluster, all 1,000 subscribers are processed concurrently. Suppose each sub-request (i.e. 1,000 subscribers) takes 1 minute to process on average, then the total time to dispatch notifications to 1M subscribers takes 1,000/10 = 100min, or 1hr40min.
The default value for broadcastSubscriberChunkSize and broadcastSubRequestBatchSize are defined in /src/config.ts
module.exports = {
- notification: {
- broadcastSubscriberChunkSize: 1000,
- broadcastSubRequestBatchSize: 10,
- },
-};
-To customize, create the config with updated value in file /src/config.local.js.
If total number of subscribers is less than broadcastSubscriberChunkSize, then no sub-requests are spawned. Instead, the main request dispatches all notifications.
How to determine the optimal value for broadcastSubscriberChunkSize and broadcastSubRequestBatchSize?
broadcastSubscriberChunkSize is determined by the concurrency capability of the downstream message handlers such as SMTP server or SMS service provider. broadcastSubRequestBatchSize is determined by the size of NotifyBC cluster. As a rule of thumb, set broadcastSubRequestBatchSize equal to the number of non-master nodes in NotifyBC cluster.
Broadcast Push Notification Custom Filter Functions
Advanced Topic
Defining custom function requires knowledge of JavaScript and understanding how external libraries are added and referenced in Node.js. Setting a development environment to test the custom function is also recommended.
const _ = require('lodash')
-module.exports = {
- ...
- notification: {
- broadcastCustomFilterFunctions: {
- contains_ci: {
- _func: async function(resolvedArgs) {
- if (!resolvedArgs[0] || !resolvedArgs[1]) {
- return false
- }
- return _.toLower(resolvedArgs[0]).indexOf(_.toLower(resolvedArgs[1])) >= 0
- },
- _signature: [
- {
- types: [2]
- },
- {
- types: [2]
- }
- ]
- }
- }
- }
-}
-install additional Node.js modules
The recommended way to install additional Node.js modules is by running command npm install <your_module> from the directory one level above NotifyBC root. For example, if NotifyBC is installed on /data/notifyBC, then run the command from directory /data. The command will then install the module to /data/node_modules/<your_module>.
Guaranteed Broadcast Push Dispatch Processing
As a major enhancement in v3, by default NotifyBC guarantees all subscribers of a broadcast push notification will be processed in spite of NotifyBC node failures during dispatching. Node failure is a concern because the time takes to dispatch broadcast push notification is proportional to number of subscribers, which is potentially large.
The guarantee is achieved by
- logging the dispatch result to database individually right after each dispatch
- when subscribers are divided into chunks and a chunk sub-request fails, the original request re-submits the sub-request
- the original request periodically updates the notification updated timestamp field as heartbeat during dispatching
- if original request fails,
- a cron job detects the failure from the stale timestamp, and re-submits the original request
- all chunk sub-requests detects the the failure from the socket error, and stop processing
If performance is a higher priority to you, disable both the guarantee and bounce handling by setting config notification.guaranteedBroadcastPushDispatchProcessing and email.bounce.enabled to false in file /src/config.local.js
module.exports = {
- notification: {
- guaranteedBroadcastPushDispatchProcessing: false,
- },
- email: {
- bounce: {enabled: false},
- },
-};
-In such case only failed dispatches are written to dispatch.failed field of the notification.
Also log skipped dispatches for broadcast push notifications
When guaranteedBroadcastPushDispatchProcessing is true, by default only successful and failed dispatches are logged, along with dispatch candidates. Dispatches that are skipped by filters defined at subscription (broadcastPushNotificationFilter) or notification (broadcastPushNotificationSubscriptionFilter) are not logged for performance reason. If you also want skipped dispatches to be logged to dispatch.skipped field of the notification, set logSkippedBroadcastPushDispatches to true in file /src/config.local.js
module.exports = {
- ...
- notification: {
- ...
- logSkippedBroadcastPushDispatches: true,
- }
-}
-Setting logSkippedBroadcastPushDispatches to true only has effect when guaranteedBroadcastPushDispatchProcessing is true.
`,7);function L(M,H){const o=p("RouterLink"),a=p("ExternalLinkIcon");return r(),l("div",null,[d,n("p",null,[s("For example, to notify subscribers of "),m,s(" on updates to feed "),h,s(", create following config item using "),e(o,{to:"/docs/api-config/#create-a-configuration"},{default:i(()=>[s("POST configuration API")]),_:1})]),b,n("ul",null,[n("li",null,[s("rss "),n("ul",null,[f,n("li",null,[k,s("timeSpec: RSS poll frequency, a space separated fields conformed to "),n("a",v,[s("unix crontab format"),e(a)]),s(" with an optional left-most seconds field. See "),n("a",g,[s("allowed ranges"),e(a)]),s(" of each field")]),y])]),n("li",null,[s("httpHost: the http protocol, host and port used by "),e(o,{to:"/docs/overview/#mail-merge"},{default:i(()=>[s("mail merge")]),_:1}),s(". If missing, the value is auto-populated based on the REST request that creates this config item.")]),n("li",null,[s("messageTemplates: channel-specific message templates with channel name as the key. "),q,s(" generates a notification for each channel specified in the message templates. Message template fields are the same as those in "),e(o,{to:"/docs/api-notification/#field-message"},{default:i(()=>[s("notification api")]),_:1}),s(". Message template fields support dynamic token.")])]),_,n("p",null,[s("To support rule-based notification event filtering, "),S,s(" uses a "),n("a",w,[s("modified version"),e(a)]),s(" of "),n("a",B,[s("jmespath"),e(a)]),s(" to implement json query. The modified version allows defining custom functions that can be used in "),j,s(" field of subscription API and "),x,s(" field of subscription API. The functions must be implemented using JavaScript in config "),N,s(". The functions can even be "),C,s(". For example, the case-insensitive string matching function "),T,s(" shown in the example of that field can be created in file "),P]),R,n("p",null,[s("Consult jmespath.js source code on the "),n("a",F,[s("functionTable syntax"),e(a)]),s(" and "),n("a",z,[s("type constants"),e(a)]),s(" used by above code. Note the function can use any Node.js modules ("),n("em",null,[n("a",I,[s("lodash"),e(a)])]),s(" in this case).")]),A,n("p",null,[s("Guaranteed processing doesn't mean notification will be dispatched to every intended subscriber, however. Dispatch can still be rejected by smtp/sms server. Furthermore, even if dispatch is successful, it only means the sending is successful. It doesn't guarantee the recipient receives the notification. "),e(o,{to:"/docs/config/email.html#bounce"},{default:i(()=>[s("Bounce")]),_:1}),s(" may occur for a successful dispatch, for instance; or the recipient may not read the message.")]),n("p",null,[s("The guarantee comes at a performance penalty because result of each dispatch is written to database one by one, taking a toll on the database. It should be noted that the "),e(o,{to:"/docs/miscellaneous/benchmarks.html"},{default:i(()=>[s("benchmarks")]),_:1}),s(" were conducted without the guarantee.")]),D])}const E=c(u,[["render",L],["__file","index.html.vue"]]);export{E as default}; diff --git a/version/5.0/assets/index.html-bc520f9b.js b/version/5.0/assets/index.html-bc520f9b.js deleted file mode 100644 index 222cf768d..000000000 --- a/version/5.0/assets/index.html-bc520f9b.js +++ /dev/null @@ -1 +0,0 @@ -const e=JSON.parse('{"key":"v-32b5e2dd","path":"/docs/config-reverseProxyIpLists/","title":"Reverse Proxy IP Lists","lang":"en-US","frontmatter":{"permalink":"/docs/config-reverseProxyIpLists/"},"headers":[],"git":{},"filePathRelative":"docs/config/reverseProxyIpLists.md"}');export{e as data}; diff --git a/version/5.0/assets/index.html-bfdc43cd.js b/version/5.0/assets/index.html-bfdc43cd.js deleted file mode 100644 index 66439d42f..000000000 --- a/version/5.0/assets/index.html-bfdc43cd.js +++ /dev/null @@ -1,4 +0,0 @@ -import{_ as r,r as o,o as l,c as d,a as e,b as t,d as n,w as c,e as a}from"./app-5138f739.js";const u={},h=a('Developer Notes
Setup development environment
Install Visual Studio Code and following extensions:
- Prettier
- ESLint
- Vetur
- Code Spell Checker
- Debugger for Chrome
Multiple run configs have been created to facilitate debugging server, client, test and docs.
Client certificate authentication doesn't work in client debugger
Because Vue cli webpack dev server cannot proxy passthrough HTTPS connections, client certificate authentication doesn't work in client debugger. If testing client certificate authentication in web console is needed, run yarn build to generate prod client distribution and launch server debugger on https://localhost:3000
Automated Testing
',7),p=e("em",null,"NotifyBC",-1),m={href:"https://jestjs.io/",target:"_blank",rel:"noopener noreferrer"},b=e("code",null,"yarn test:e2e",-1),f=e("em",null,"Test",-1),g=e("p",null,"Github Actions runs tests as part of the build. All test scripts should be able to run unattended, headless, quickly and depend only on local resources.",-1),_=e("h3",{id:"writing-test-specs",tabindex:"-1"},[e("a",{class:"header-anchor",href:"#writing-test-specs","aria-hidden":"true"},"#"),t(" Writing Test Specs")],-1),v={href:"https://github.com/visionmedia/supertest",target:"_blank",rel:"noopener noreferrer"},w={href:"https://github.com/nodkz/mongodb-memory-server",target:"_blank",rel:"noopener noreferrer"},k=e("em",null,"sendMail",-1),x=e("em",null,"sendSMS",-1),y=a(`- start at a processing phase as early as possible. For example, to test a REST end point, start with the HTTP user request.
- assert outcome of a processing phase as late and down below as possible - the HTTP response body/code, the database record created, for example.
- avoid asserting middleware function input/output to facilitate code refactoring.
- mock email/sms sending function (implemented by default). Inspect the input of the function, or at least assert the function has been called.
Install Docs Website
If you want to contribute to NotifyBC docs beyond simple fix ups, run
yarn --cwd docs install
-yarn --cwd docs dev
-If everything goes well, the last line of the output will be
> VuePress dev server listening at http://localhost:8080/NotifyBC/
-Quick Start
For the impatient, here's how to get a boilerplate NotifyBC instance up and running if you have git and node.js installed:
git clone https://github.com/bcgov/NotifyBC.git
-cd NotifyBC
-npm i -g yarn && yarn install && yarn build
-yarn start
-# => Now browse to http://localhost:3000
-Installation
NotifyBC can be installed in 3 ways:
For the purpose of evaluation, both source code and docker container will do. For production, the recommendation is one of
- deploying to Kubernetes
- setting up a load balanced app cluster from source code build, backed by MongoDB.
To setup a development environment in order to contribute to NotifyBC, installing from source code is preferred.
Deploy locally from Source Code
System Requirements
',8),v=n("li",null,"Git",-1),h={href:"https://nodejs.org",target:"_blank",rel:"noopener noreferrer"},k=n("li",null,"openssl (if enable HTTPS)",-1),b=n("li",null,"MongoDB with replica set, required for production",-1),g=n("li",null,"A standard SMTP server to deliver outgoing email, required for production if email is enabled.",-1),y={href:"http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html",target:"_blank",rel:"noopener noreferrer"},f=n("em",null,"NotifyBC",-1),_=n("li",null,[e("A SMS service provider if needs to enable SMS channel. The supported service providers are "),n("ul",null,[n("li",null,"Twilio (default)"),n("li",null,"Swift")])],-1),x=n("li",null,"Redis v6, required if email or sms throttling is enabled",-1),w=n("li",null,"SiteMinder, if needs SiteMinder authentication",-1),C=n("li",null,"An OIDC provider, if needs OIDC authentication",-1),B=t("- Minimum runtime firewall requirements:
- outbound to your ISP DNS server
- outbound to any on port 80 and 443 in order to run build scripts and send SMS messages
- outbound to any on SMTP port 25 if using direct mail; for SMTP relay, outbound to your configured SMTP server and port only
- inbound to listening port (3000 by default) from other authorized server ips
- if NotifyBC instance will handle anonymous subscription from client browser, the listening port should be open to internet either directly or indirectly through a reverse proxy; If NotifyBC instance will only handle SiteMinder authenticated webapp requests, the listening port should NOT be open to internet. Instead, it should only open to SiteMinder web agent reverse proxy.
- If list-unsubscribe by email is needed, then one of the following must be met
- NotifyBC can bind to port 25 opening to internet
- a tcp proxy server of which port 25 is open to internet. This proxy server can reach NotifyBC on a tcp port.
Installation
Run following commands
git clone https://github.com/bcgov/NotifyBC.git
-cd NotifyBC
-npm i -g yarn && yarn install && yarn build
-yarn start
-If successful, you will see following output
...
-Server is running at http://0.0.0.0:3000
-The above commands installs the main version, i.e. main branch tip of NotifyBC GitHub repository. To install a specific version, say v2.1.0, run
git checkout tags/v2.1.0 -b v2.1.0
-install from behind firewall
If you want to install on a server behind firewall which restricts internet connection, you can work around the firewall as long as you have access to a http(s) forward proxy server. Assuming the proxy server is http://my_proxy:8080 which proxies both http and https requests, to use it:
For Linux
export http_proxy=http://my_proxy:8080 -export https_proxy=http://my_proxy:8080 -git config --global url."https://".insteadOf git:// -For Windows
git config --global http.proxy http://my_proxy:8080 -git config --global url."https://".insteadOf git:// -npm config set proxy http://my_proxy:8080 -npm i -g yarn -yarn config set proxy http://my_proxy:8080 -
Install Windows Service
After get the app running interactively, if your server is Windows and you want to install the app as a Windows service, run
npm install -g node-windows
-npm link node-windows
-node windows-service.js
-To install,
Follow your platform's instruction to login to the platform. For AKS, run
az loginandaz aks get-credentials; for OpenShift, runoc loginRun
git clone https://github.com/bcgov/NotifyBC.git -cd NotifyBC -helm install -gf helm/platform-specific/<platform>.yaml helm -replace <platform> with openshift or aks depending on your platform.
The above commands create following artifacts:
- 1 stateful set of 3 pods running a MongoDB replicaset
- 1 stateful set of 3 pods running a Redis sentinel
- 2 deployments - notify-bc-app and notify-bc-cron
- 1 HPA - notify-bc-cron
- 5 services - notify-bc, notify-bc-smtp, mongodb-headless, redis and redis-headless
- 3 PVCs each for one MongoDB pod
- 3 service accounts - notify-bc, mongodb and redis
- a few config maps, most importantly notify-bc
- a few secrets, most importantly mongodb and redis, containing credentials for Mongodb and Redis respectively
- On AKS,
- a notify-bc ingress
- On OpenShift,
- 2 routes - notify-bc-web and notify-bc-smtp
To upgrade,
helm upgrade <release-name> -f helm/platform-specific/<platform>.yaml --set mongodb.auth.rootPassword=<mongodb-root-password> --set mongodb.auth.replicaSetKey=<mongodb-replica-set-key> --set mongodb.auth.password=<mongodb-password> helm
-replace <release-name> with installed helm release name and <platform> with openshift or aks depending on your platform. MongoDB credentials <mongodb-root-password>, <mongodb-replica-set-key> and <mongodb-password> can be found in secret <release-name>-mongodb. It is recommended to specify mongodb credentials in a file rather than command line. See Customizations below.
To uninstall,
helm uninstall <release-name>
-replace <release-name> with installed helm release name.
Customizations
`,9),J=n("em",null,".local.yaml",-1),U=n("em",null,"helm/values.local.yaml",-1),W=n("em",null,"helm/values.yaml",-1),Y={href:"https://github.com/bitnami/charts/tree/master/bitnami/mongodb",target:"_blank",rel:"noopener noreferrer"},$=n("em",null,"helm/values.local.yaml",-1),X=n("em",null,"helm/values.yaml",-1),Z=n("em",null,"mongodb",-1),Q=n("em",null,"helm/values.local.yaml",-1),nn=t(`To apply customizations, add -f helm/values.local.yaml to the helm command after -f helm/platform-specific/<platform>.yaml. For example, to install chart with customization on OpenShift,
helm install -gf helm/platform-specific/openshift.yaml -f helm/values.local.yaml helm
-to upgrade an existing release with customization on OpenShift,
helm upgrade <release-name> -f helm/platform-specific/openshift.yaml -f helm/values.local.yaml helm
-Backup helm/values.local.yaml
Backup helm/values.local.yaml to a private secured SCM is highly recommended, especially for production environment.
Following are some common customizations
`,6),en=t(`Update config.local.js in ConfigMap, for example to define httpHost
# in file helm/values.local.yaml
-configMap:
- config.local.js: |-
- module.exports = {
- httpHost: 'https://myNotifyBC.myOrg.com',
- }
-Set hostname on AKS,
# in file helm/values.local.yaml
-ingress:
- hosts:
- - host: myNotifyBC.myOrg.com
- paths:
- - path: /
-# in file helm/values.local.yaml
-ingress:
- annotations:
- cert-manager.io/cluster-issuer: letsencrypt
- tls:
- - secretName: tls-secret
- hosts:
- - notify-bc.local
-Route host names on Openshift are by default auto-generated. To set to fixed values
# in file helm/values.local.yaml
-route:
- web:
- host: 'myNotifyBC.myOrg.com'
- smtp:
- host: 'smtp.myNotifyBC.myOrg.com'
-Add certificates to OpenShift web route
# in file helm/values.local.yaml
-route:
- web:
- tls:
- caCertificate: |-
- -----BEGIN CERTIFICATE-----
- ...
- -----END CERTIFICATE-----
- certificate: |-
- -----BEGIN CERTIFICATE-----
- ...
- -----END CERTIFICATE-----
- insecureEdgeTerminationPolicy: Redirect
- key: |-
- -----BEGIN PRIVATE KEY-----
- ...
- -----END PRIVATE KEY-----
-# in file helm/values.local.yaml
-mongodb:
- architecture: standalone
-To set credentials,
# in file helm/values.local.yaml
-mongodb:
- auth:
- rootPassword: <secret>
- replicaSetKey: <secret>
- passwords:
- - <secret>
-To install a Helm chart, the above credentials can be randomly defined. To upgrade an existing release, they must match what's defined in secret <release-name>-mongodb.
`,4),vn=n("p",null,"Redis",-1),hn=n("em",null,"NotifyBC",-1),kn={href:"https://github.com/bitnami/charts/tree/master/bitnami/redis",target:"_blank",rel:"noopener noreferrer"},bn=n("em",null,"redis",-1),gn=t(`# in file helm/values.local.yaml
-redis:
- auth:
- password: <secret>
-To install a Helm chart, the above credential can be randomly defined. To upgrade an existing release, It must match what's defined in secret <release-name>-redis.
`,2),yn=n("li",null,[n("p",null,"Both Bitnami MongoDB and Redis use Docker Hub for docker registry. Rate limit imposed by Docker Hub can cause runtime problems. If your organization has JFrog artifactory, you can change the registry")],-1),fn=t(`# in file helm/values.local.yaml
-global:
- imageRegistry: <artifactory.myOrg.com>
- imagePullSecrets:
- - <docker-pull-secret>
-Enable scheduled MongoDB backup CronJob
# in file helm/values.local.yaml -cronJob: - enabled: true - schedule: '1 0 * * *' - retentionDays: 7 - timeZone: UTC - persistence: - size: 5Gi -where
- enabled: whether to enable the MongoDB backup CronJob or not; default to
false - schedule: the Unix crontab schedule; default to
'1 0 * * *'which runs daily at 12:01AM - retentionDays: how many days the backup is retained; default to
7 - timeZone: the Unix TZ environment variable; default to
UTC - persistence size: size of PVC; default to
5Gi
The CronJob backs up MongoDB to a PVC named after the chart with suffix -cronjob-mongodb-backup and purges backups that are older than retentionDays.
To facilitate restoration, mount the PVC to MongoDB pod
# in file helm/values.local.yaml -mongodb: - extraVolumes: - - name: export - persistentVolumeClaim: - claimName: <PVC_NAME> - extraVolumeMounts: - - name: export - mountPath: /export - readOnly: true -Restoration can then be achieved by running in MongoDB pod
mongorestore -u "$MONGODB_EXTRA_USERNAMES" -p"$MONGODB_EXTRA_PASSWORDS" \\ ---uri="mongodb://$K8S_SERVICE_NAME" --db $MONGODB_EXTRA_DATABASES --gzip --drop \\ ---archive=/export/<mongodb-backup-YYMMDD-hhmmss.gz> -- enabled: whether to enable the MongoDB backup CronJob or not; default to
NotifyBC image tag defaults to appVersion in file helm/Chart.yaml. To change to latest, i.e. tip of the main branch,
# in file helm/values.local.yaml -image: - tag: latest -Enable autoscaling for app pod
# in file helm/values.local.yaml -autoscaling: - enabled: true -
Deploy Docker Container
If you have git and Docker installed, you can run following command to deploy NotifyBC Docker container:
docker run --platform linux/amd64 --rm -dp 3000:3000 ghcr.io/bcgov/notify-bc
-# open http://localhost:3000
-If successful, similar output is displayed as in source code installation.
`,5),wn={__name:"index.html",setup(Cn){const l=d();return(Bn,Sn)=>{const a=i("ExternalLinkIcon");return r(),p("div",null,[m,n("ul",null,[n("li",null,[e("Software "),n("ul",null,[v,n("li",null,[n("a",h,[e("Node.js"),s(a)]),e("@"+c(u(l).packageJson.engines.node),1)]),k])]),n("li",null,[e("Services "),n("ul",null,[b,g,n("li",null,[e("A tcp proxy server such as "),n("a",y,[e("nginx stream proxy"),s(a)]),e(" if list-unsubscribe by email is needed and "),f,e(" server cannot expose port 25 to internet")]),_,x,w,C])]),B]),S,n("p",null,[e("Now open "),n("a",T,[e("http://localhost:3000"),s(a)]),e(". The page displays NotifyBC Web Console.")]),D,n("p",null,[e("after "),N,e(". A list of versions can be found "),n("a",M,[e("here"),s(a)]),e(".")]),I,n("p",null,[e("This will create and start service "),A,e(". To change service name, modify file "),E,e(" before running it. See "),n("a",z,[e("node-windows"),s(a)]),e(" for other operations such as uninstalling the service.")]),O,n("p",null,[R,e(" provides a "),n("a",P,[e("container package"),s(a)]),e(" in GitHub Container Registry and a "),n("a",q,[e("Helm"),s(a)]),e(" chart to facilitate Deploying to Kubernetes. Azure AKS and OpenShift are the two tested platforms. Other Kubernetes platforms are likely to work subject to customizations. Before deploying to AKS, "),n("a",V,[e("create an ingress controller "),s(a)]),e(".")]),K,n("ul",null,[F,n("li",null,[e("Platform-specific CLI such as "),n("a",H,[e("Azure CLI"),s(a)]),e(" or "),n("a",G,[e("OpenShift CLI"),s(a)])]),n("li",null,[n("a",j,[e("Helm CLI"),s(a)])])]),L,n("p",null,[e("Various customizations can be made to chart. Some are platform dependent. To customize, first create a file with extension "),J,e(". The rest of the document assumes the file is "),U,e(". Then add customized parameters to the file. See "),W,e(" and Bitnami MongoDB chart "),n("a",Y,[e("readme"),s(a)]),e(" for customizable parameters. Parameters in "),$,e(" overrides corresponding ones in "),X,e(". In particular, parameters under "),Z,e(" of "),Q,e(" overrides Bitnami MongoDB chart parameters.")]),nn,n("ul",null,[en,n("li",null,[n("p",null,[e("Use "),n("a",an,[e("Let's Encrypt on AKS"),s(a)]),e(". After following the instructions in the link, add following ingress customizations to file "),sn]),tn]),ln,n("li",null,[on,n("p",null,[rn,e(" chart depends on "),n("a",pn,[e("Bitnami MongoDB chart"),s(a)]),e(" for MongoDB database provisioning. All documented parameters are customizable under "),cn,e(". For example, to change "),un,e(" to "),dn]),mn]),n("li",null,[vn,n("p",null,[hn,e(" chart depends on "),n("a",kn,[e("Bitnami Redis chart"),s(a)]),e(" for Redis provisioning. All documented parameters are customizable under "),bn,e(". For example, to set credential")]),gn]),yn]),fn,n("p",null,[e("The above settings assume you have setup secretDatabase
By default NotifyBC uses in-memory database backed up by folder /server/database/ for local and docker deployment and MongoDB for Kubernetes deployment. To use MongoDB for non-Kubernetes deployment, add file /src/datasources/db.datasource.(local|<env>).(json|js|ts) with MongoDB connection information such as following:
module.exports = {
- uri: 'mongodb://127.0.0.1:27017/notifyBC?replicaSet=rs0',
- user: process.env.MONGODB_USER,
- pass: process.env.MONGODB_PASSWORD,
-};
-Subscription
Configs in this section customize behavior of subscription and unsubscription workflow. They are all sub-properties of config object subscription. This object can be defined as service-agnostic static config as well as service-specific dynamic config, which overrides the static one on a service-by-service basis. Default static config is defined in file /src/config.ts. There is no default dynamic config.
To customize static config, create the config object subscription in file /src/config.local.js
module.exports = {
- "subscription": {
- ...
- }
-}
-curl -X POST http://localhost:3000/api/configurations \\
--H 'Content-Type: application/json' \\
--H 'Accept: application/json' -d @- << EOF
-{
- "name": "subscription",
- "serviceName": "myService",
- "value": {
- ...
- }
-}
-EOF
-Sub-properties denoted by ellipsis in the above two code blocks are documented below. A service can have at most one dynamic subscription config.
Confirmation Request Message
To prevent NotifyBC from being used as spam engine, when a subscription request is sent by user (as opposed to admin) without encryption, the content of confirmation request sent to user's notification channel has to come from a pre-configured template as opposed to be specified in subscription request.
The following default subscription sub-property confirmationRequest defines confirmation request message settings for different channels
{
- "subscription": {
- ...
- "confirmationRequest": {
- "sms": {
- "confirmationCodeRegex": "\\\\d{5}",
- "sendRequest": true,
- "textBody": "Enter {confirmation_code} on screen"
- },
- "email": {
- "confirmationCodeRegex": "\\\\d{5}",
- "sendRequest": true,
- "from": "no_reply@invlid.local",
- "subject": "Subscription confirmation",
- "textBody": "Enter {confirmation_code} on screen",
- "htmlBody": "Enter {confirmation_code} on screen"
- }
- }
- }
-}
-Confirmation Verification Acknowledgement Messages
You can customize NotifyBC's on-screen response message to confirmation code verification requests. The following is the default settings
{
- "subscription": {
- ...
- "confirmationAcknowledgements": {
- "successMessage": "You have been subscribed.",
- "failureMessage": "Error happened while confirming subscription."
- }
- }
-}
-In addition to customizing the message, you can define a redirect URL instead of displaying successMessage or failureMessage. For example, to redirect on-screen acknowledgement to a page in your app for service myService, create a dynamic config by calling REST config api
curl -X POST 'http://localhost:3000/api/configurations' \\
--H 'Content-Type: application/json' \\
--H 'Accept: application/json' -d @- << EOF
-{
- "name": "subscription",
- "serviceName": "myService",
- "value": {
- "confirmationAcknowledgements": {
- "redirectUrl": "https://myapp/subscription/acknowledgement"
- }
- }
-}
-EOF
-If error happened during subscription confirmation, query string ?err=<error> will be appended to redirectUrl.
Duplicated Subscription
NotifyBC by default allows a user subscribe to a service through same channel multiple times. If this is undesirable, you can set config subscription.detectDuplicatedSubscription to true. In such case instead of sending user a confirmation request, NotifyBC sends user a duplicated subscription notification message. Unlike a confirmation request, duplicated subscription notification message either shouldn't contain any information to allow user confirm the subscription, or it should contain a link that allows user to replace existing confirmed subscription with this one. You can customize duplicated subscription notification message by setting config subscription.duplicatedSubscriptionNotification in either config.local.js or using configuration api for service-specific dynamic config. Following is the default settings defined in config.json
{
- ...
- "subscription": {
- ...
- "detectDuplicatedSubscription": false,
- "duplicatedSubscriptionNotification": {
- "sms": {
- "textBody": "A duplicated subscription was submitted and rejected. you will continue receiving notifications. If the request was not created by you, pls ignore this msg."
- },
- "email": {
- "from": "no_reply@invalid.local",
- "subject": "Duplicated Subscription",
- "textBody": "A duplicated subscription was submitted and rejected. you will continue receiving notifications. If the request was not created by you, please ignore this message."
- }
- }
- }
-}
-To allow user to replace existing confirmed subscription, set the message to something like
{
- ...
- "subscription": {
- ...
- "detectDuplicatedSubscription": false,
- "duplicatedSubscriptionNotification": {
- "email": {
- "textBody": "A duplicated subscription was submitted. If the request is not submitted by you, please ignore this message. Otherwise if you want to replace existing subscription with this one, click {subscription_confirmation_url}&replace=true."
- }
- }
- }
-}
-The query parameter &replace=true following the token {subscription_confirmation_url} will cause existing subscription be replaced.
Anonymous Unsubscription
For anonymous subscription, NotifyBC supports one-click opt-out by allowing unsubscription URL provided in notifications. To thwart unauthorized unsubscription attempts, NotifyBC implemented and enabled by default two security measurements
`,20),b=s("li",null,"Anonymous unsubscription request requires unsubscription code, which is a random string generated at subscription time. Unsubscription code reduces brute force attack risk by increasing size of key space. Without it, an attacker only needs to successfully guess subscription id. Be aware, however, the unsubscription code has to be embedded in unsubscription link. If the user forwarded a notification to other people, he/she is still vulnerable to unauthorized unsubscription.",-1),k=s("em",null,"anonymousUnsubscription",-1),g={href:"https://github.com/bcgov/NotifyBC/blob/main/src/config.ts",target:"_blank",rel:"noopener noreferrer"},f=e(`module.exports = {
- subscription: {
- anonymousUnsubscription: {
- code: {
- required: true,
- regex: '\\\\d{5}',
- },
- acknowledgements: {
- onScreen: {
- successMessage: 'You have been un-subscribed.',
- failureMessage: 'Error happened while un-subscribing.',
- },
- notification: {
- email: {
- from: 'no_reply@invalid.local',
- subject: 'Un-subscription acknowledgement',
- textBody:
- 'This is to acknowledge you have been un-subscribed from receiving notification for {unsubscription_service_names}. If you did not authorize this change or if you changed your mind, open {unsubscription_reversion_url} to revert.',
- htmlBody:
- 'This is to acknowledge you have been un-subscribed from receiving notification for {unsubscription_service_names}. If you did not authorize this change or if you changed your mind, click <a href="{unsubscription_reversion_url}">here</a> to revert.',
- },
- },
- },
- },
- },
-};
-The settings control whether or not unsubscription code is required, its RegEx pattern, and acknowledgement message templates for both on-screen and push notifications. Customization should be made to file /src/config.local.js for static config or using configuration api for service-specific dynamic config.
To disable acknowledgement notification, set subscription.anonymousUnsubscription.acknowledgements.notification or a specific channel underneath to null
module.exports = {
- subscription: {
- anonymousUnsubscription: {
- acknowledgements: {
- notification: null,
- },
- },
- },
-};
-For on-screen acknowledgement, you can define a redirect URL instead of displaying successMessage or failureMessage. For example, to redirect on-screen acknowledgement to a page in your app for all services, create following config in file /src/config.local.js
module.exports = {
- subscription: {
- anonymousUnsubscription: {
- acknowledgements: {
- onScreen: {
- redirectUrl: 'https://myapp/unsubscription/acknowledgement',
- },
- },
- },
- },
-};
-If error happened during unsubscription, query string ?err=<error> will be appended to redirectUrl.
You can customize message displayed on-screen when user clicks revert unsubscription link in the acknowledgement notification. The default settings are
{
- "subscription": {
- "anonymousUndoUnsubscription": {
- "successMessage": "You have been re-subscribed.",
- "failureMessage": "Error happened while re-subscribing."
- }
- }
-}
-You can redirect the message page by defining anonymousUndoUnsubscription.redirectUrl.
`,10);function h(q,y){const t=o("RouterLink"),p=o("ExternalLinkIcon");return r(),l("div",null,[d,s("p",null,[m,n(" to create a service-specific dynamic subscription config, use REST "),a(t,{to:"/docs/api-config/"},{default:i(()=>[n("config api")]),_:1})]),v,s("ul",null,[b,s("li",null,[n("Acknowledgement notification - a (final) notification is sent to user acknowledging unsubscription, and offers a link to revert had the change been made unauthorized. A deleted subscription (unsubscription) may have a limited lifetime (30 days by default) according to retention policy defined in "),a(t,{to:"/docs/config-cronJobs/"},{default:i(()=>[n("cron jobs")]),_:1}),n(" so the reversion can only be performed within the lifetime.")])]),s("p",null,[n("You can customize anonymous unsubscription settings by changing the "),k,n(" configuration. Following is the default settings defined in "),s("a",g,[n("config.json"),a(p)])]),f])}const _=c(u,[["render",h],["__file","index.html.vue"]]);export{_ as default}; diff --git a/version/5.0/assets/index.html-dd35ba9b.js b/version/5.0/assets/index.html-dd35ba9b.js deleted file mode 100644 index 79ddb8c6f..000000000 --- a/version/5.0/assets/index.html-dd35ba9b.js +++ /dev/null @@ -1,4 +0,0 @@ -import{_ as l,r as s,o as r,c as d,a as t,b as e,d as o,w as a,e as i}from"./app-5138f739.js";const u={},h=t("h1",{id:"web-console",tabindex:"-1"},[t("a",{class:"header-anchor",href:"#web-console","aria-hidden":"true"},"#"),e(" Web Console")],-1),p=t("a",{href:"../installation"},"installing",-1),f=t("em",null,"NotifyBC",-1),m=t("em",null,"NotifyBC",-1),b={href:"http://localhost:3000",target:"_blank",rel:"noopener noreferrer"},_=t("p",null,"What you see in web console and what you get from API calls depend on how your requests are authenticated.",-1),y=t("h2",{id:"ip-whitelisting-authentication",tabindex:"-1"},[t("a",{class:"header-anchor",href:"#ip-whitelisting-authentication","aria-hidden":"true"},"#"),e(" Ip whitelisting authentication")],-1),g=t("span",{class:"material-icons"},"verified_user",-1),w=t("p",null,"To see the result of non super-admin requests, you can choose one of the following methods",-1),v=t("ul",null,[t("li",null,"customize admin ip list to omit localhost (127.0.0.1)"),t("li",null,"access web console from another ip not in the admin ip list")],-1),I=t("h2",{id:"client-certificate-authentication",tabindex:"-1"},[t("a",{class:"header-anchor",href:"#client-certificate-authentication","aria-hidden":"true"},"#"),e(" Client certificate authentication")],-1),x=t("em",null,"NotifyBC",-1),A=t("span",{class:"material-icons"},"verified",-1),k=i('Anonymous
If you access web console from a client that is not in the admin ip list, you are by default anonymous user. Anonymous authentication status is indicated by the LOGIN button on top right corner of web console. Click the button to login.
Access token authentication
',3),P=t("em",null,"Access Token",-1),C=t("em",null,"Access Token",-1),O=t("span",{class:"material-icons"},"login",-1),S=i('Tokens are not shared between API Explorer and web console
Despite API Explorer appears to be part of web console, it is a separate application. At this point neither the access token nor the OIDC access token are shared between the two applications. You have to use API Explorer's Authorize button to authenticate even if you have logged into web console.
OIDC authentication
If you have configured OIDC, then the login button will direct you to OIDC provider's login page. Once login successfully, you will be redirected back to NoitfyBC web console. OIDC authentication status is indicated by the LOGOUT button.
',3),q=i(`SiteMinder authentication
To get results of a SiteMinder authenticated user, do one of the following
- access the API via a SiteMinder proxy if you have configured SiteMinder properly
- use a tool such as curl that allows to specify custom headers, and supply SiteMinder header SM_USER:
curl -X GET --header "Accept: application/json" \\
- --header "SM_USER: foo" \\
- "http://localhost:3000/api/notifications"
-Model Schema
The API operates on following subscription data model fields:
| Name | Attributes | ||||||
|---|---|---|---|---|---|---|---|
serviceName name of the service. Avoid prefixing the name with underscore (_), or it may conflict with internal implementation. |
| ||||||
channel name of the delivery channel. Valid values: email and sms. Notice inApp is invalid as in-app notification doesn't need subscription. |
| ||||||
userChannelId user's delivery channel id, for example, email address |
| ||||||
id subscription id |
| ||||||
state state of subscription. Valid values: unconfirmed, confirmed, deleted |
| ||||||
userId user id. Auto-populated for authenticated user requests. |
| ||||||
created date and time of creation |
| ||||||
updated date and time of last update |
| ||||||
confirmationRequest an object containing these child fields
|
| ||||||
broadcastPushNotificationFilter a string conforming to jmespath filter expressions syntax after the question mark (?). The filter is matched against the data field of the subscription. Examples of filter
|
| ||||||
An object used by
|
| ||||||
unsubscriptionCode generated randomly according to RegEx config anonymousUnsubscription.code.regex during anonymous subscription if config anonymousUnsubscription.code.required is set to true |
| ||||||
unsubscribedAdditionalServices generated if parameter additionalServices is supplied in unsubscription request. Contains 2 sub-fields: ids and names, each being a list identifying the additional unsubscribed subscriptions. |
|
Get Subscriptions
GET /subscriptions
-a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
",3),R=e("li",null,"URL-encoded stringified JSON object (see example below); or",-1),S={href:"https://github.com/ljharb/qs",target:"_blank",rel:"noopener noreferrer"},E=e("code",null,'?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"',-1),P=n(`Regardless, the filter will have to be parsed into a JSON object conforming to
{
- "where": {...},
- "fields": ...,
- "order": ...,
- "skip": ...,
- "limit": ...,
-}
-All properties are optional. The syntax for each property is documented, respectively
`,3),D=e("em",null,"where",-1),U={href:"https://www.mongodb.com/docs/manual/tutorial/query-documents/",target:"_blank",rel:"noopener noreferrer"},G=e("em",null,"fields",-1),V={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.select()",target:"_blank",rel:"noopener noreferrer"},$=e("em",null,"order",-1),F={href:"https://mongoosejs.com/docs/api/query.html#Query.prototype.sort()",target:"_blank",rel:"noopener noreferrer"},L=e("em",null,"skip",-1),M={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.skip/",target:"_blank",rel:"noopener noreferrer"},O=e("em",null,"limit",-1),J={href:"https://www.mongodb.com/docs/manual/reference/method/cursor.limit/",target:"_blank",rel:"noopener noreferrer"},Q=n(`outcome
- for admin requests, returns unabridged array of subscription data matching the filter
- for authenticated user requests, in addition to filter, following constraints are imposed on the returned array
- only non-deleted subscriptions
- only subscriptions created by the user
- the confirmationRequest field is removed.
- forbidden for anonymous user requests
example
to retrieve subscriptions created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/subscriptions?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D'
-the value of the filter query parameter is URL-encoded stringified JSON object
{
- "where": {
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
- }
-}
-Get Subscription Count
GET /subscriptions/count
-outcome
Validations rules are the same as GET /subscriptions. If passed, the output is a count of subscriptions matching the query
{
- "count": <number>
-}
-example
to retrieve the count of subscriptions created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/subscriptions/count?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D'
-the value of the where query parameter is URL-encoded stringified JSON object
{
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
-}
-Create a Subscription
POST /subscriptions
-inputs are validated. If validation fails, error is returned.
for user requests, the state field is forced to unconfirmed
for authenticated user request, userId field is populated with authenticated userId
otherwise, unsubscriptionCode is generated if config subscription.anonymousUnsubscription.code.required is true, unless if the request is made by admin and the field is already populated
if confirmationRequest.confirmationCodeEncrypted is populated, a confirmation code is generated by decrypting this field using private RSA key, then put decrypted confirmation code to field confirmationRequest.confirmationCode
otherwise, for user requests and for admin requests missing message template, the message template is set to configured value. Then, if confirmationRequest.confirmationCodeRegex is populated, a confirmation code is generated conforming to regex and put to field confirmationRequest.confirmationCode
the subscription request is saved to database.
if confirmationRequest.sendRequest is true, then a message is sent to userChannelId. The message template is determined by
- if detectDuplicatedSubscription is true and there is already a confirmed subscription to the same serviceName, channel and userChannelId, then message is sent using duplicatedSubscriptionNotification as template;
- otherwise, a confirmation request is sent to using the template fields in confirmationRequest.
examples
- To subscribe a user to service education, copy and paste following json object to the data value box in API explorer, change email addresses as needed, and click Try it out! button:
{
- "serviceName": "education",
- "channel": "email",
- "userChannelId": "foo@bar.com"
-}
-As a result, foo@bar.com should receive an email confirmation request, and following json object is returned to caller upon sending the email successfully for admin request:
{
- "serviceName": "education",
- "channel": "email",
- "userChannelId": "foo@bar.com",
- "state": "unconfirmed",
- "confirmationRequest": {
- "confirmationCodeRegex": "\\\\d{5}",
- "sendRequest": true,
- "from": "no_reply@bar.com",
- "subject": "confirmation",
- "textBody": "Enter {confirmation_code} on screen",
- "confirmationCode": "45304"
- },
- "created": "2016-10-03T17:35:40.202Z",
- "updated": "2016-10-03T17:35:40.202Z",
- "id": "57f296ec7eead50554c61de7"
-}
-For non-admin request, the field confirmationRequest is removed from response, and field userId is populated if request is authenticated:
{
- "serviceName": "education",
- "channel": "email",
- "userChannelId": "foo@bar.com",
- "state": "unconfirmed",
- "userId": "<user_id>",
- "created": "2016-10-03T18:17:09.778Z",
- "updated": "2016-10-03T18:17:09.778Z",
- "id": "57f2a0a5b1aa0e2d5009eced"
-}
-To subscribe a user to service education with RSA public key encrypted confirmation code supplied, POST following request
{ - "serviceName": "education", - "channel": "email", - "userChannelId": "foo@bar.com", - "confirmationRequest": { - "confirmationCodeEncrypted": "<encrypted-confirmation-code>", - "sendRequest": true, - "from": "no_reply@bar.com", - "subject": "confirmation", - "textBody": "Enter {confirmation_code} on screen" - } -} -As a result, NotifyBC will decrypt the confirmation code using the private RSA key, replace placeholder {confirmation_code} in the email template with the confirmation code, and send confirmation request to foo@bar.com.
Verify a Subscription
GET /subscriptions/{id}/verify
-inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- confirmation code
- parameter name: confirmationCode
- required: true
- parameter type: query
- data type: string
- whether or not replacing existing subscriptions
- parameter name: replace
- required: false
- parameter type: query
- data type: boolean
- subscription id
outcome
NotifyBC performs following actions in sequence
- the subscription identified by id is retrieved
- for user request, the userId of the subscription is checked against current request user, if not match, error is returned; otherwise
- input parameter confirmationCode is checked against confirmationRequest.confirmationCode. If not match, error is returned; otherwise
- if input parameter replace is supplied and set to true, then existing confirmed subscriptions from the same serviceName, channel and userChannelId are deleted. No unsubscription acknowledgement notification is sent
- state is set to confirmed
- the subscription is saved back to database
- displays acknowledgement message according to configuration
example
to verify a subscription with id abc, confirmation code 12345, and delete existing confirmed subscriptions once verified, run
curl 'http://localhost:3000/api/subscriptions/abc/verify?confirmationCode=12345&replace=true' -
Update a Subscription
PATCH /subscriptions/{id}
-This API is used by authenticated user to change user channel id (such as email address) and resend confirmation code.
permissions required, one of
- super admin
- admin
- authenticated user
inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- an object containing fields to be updated.
- parameter name: data
- required: true
- parameter type: body
- data type: object
- subscription id
outcome
NotifyBC processes the request similarly as creating a subscription except during input validation it imposes following extra constraints to user request
- only fields userChannelId, state and confirmationRequest can be updated
- when changing userChannelId, confirmationRequest must also be supplied
- if userChannelId is different from the saved record, state is forced to unconfirmed.
Delete a Subscription (unsubscribing)
DELETE /subscriptions/{id}?unsubscriptionCode={unsubscriptionCode}&additionalServices[]={additionalServices}&userChannelId={userChannelId}
-or
-GET /subscriptions/{id}/unsubscribe?unsubscriptionCode={unsubscriptionCode}&additionalServices[]={additionalServices}&userChannelId={userChannelId}
-inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- unsubscription code for anonymous request
- parameter name: unsubscriptionCode
- required: false
- parameter type: query
- data type: string
- additional service names to unsubscribe
- parameter name: additionalServices
- required: false
- parameter type: query
- data type: array of strings. If there is only one item and its value is _all, then all services the user subscribed on this NotifyBC instance are included. Supply multiple items by repeating this query parameter.
- user channel id for extended validation
- parameter name: userChannelId
- required: false
- parameter type: query
- data type: string
outcome
NotifyBC performs following actions in sequence
- the subscription identified by id is retrieved
- for user request,
- if request is authenticated, the userId of the subscription is checked against current request user, if not match, request is rejected
- if request is anonymous, and server is configured to require unsubscription code, the input unsubscription code is matched against the unsubscriptionCode field. Request is rejected if not match. In addition, if input parameter userChannelId is populated but doesn't match, request is rejected
- if the subscription state is not confirmed, request is rejected
- if additionalServices is populated, database is queried to retrieve the serviceName and id fields of the additional subscriptions
- the field state is set to deleted for the subscription identified by id as well as additional subscriptions retrieved in previous step
- if additionalServices is not empty, the service names and ids of the additional subscriptions are added to field unsubscribedAdditionalServices of the subscription identified by id to allow bulk undo unsubscription later on
- for anonymous unsubscription, an acknowledgement notification is sent to user if configured so
- returns
- for anonymous request, either the message or redirect as configured in anonymousUnsubscription.acknowledgements.onScreen
- for authenticated user or admin requests, number of records affected or error message if occurred.
Un-deleting a Subscription
GET /subscriptions/{id}/unsubscribe/undo
-This API allows an anonymous subscriber to undo an unsubscription.
`,3),ge=n("inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- unsubscription code
- parameter name: unsubscriptionCode
- required: false
- parameter type: query
- data type: string
outcome
NotifyBC performs following actions in sequence
- the subscription identified by id is retrieved
- for user request,
- if request is anonymous, and server is configured to require unsubscription code, the input unsubscription code is matched against the unsubscriptionCode field. Request is rejected if not match
- if request is authenticated, request is rejected
- if the subscription state is not deleted, request is rejected
Get all services with confirmed subscribers
GET /subscriptions/services
-This API is designed to facilitate implementing autocomplete for admin web console.
- permissions required, one of
- super admin
- admin
- inputs - none
- outcome
- for admin requests, returns an array of unique service names with confirmed subscribers
- forbidden for non-admin requests
Replace a Subscription
PUT /subscriptions/{id}
-This API is intended to be only used by admin web console to modify a subscription without triggering any confirmation or acknowledgement notification.
- permissions required, one of
- super admin
- admin
- permissions required, one of
- super admin
- admin
- authenticated user
- inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- subscription data
- parameter name: data
- required: true
- parameter type: body
- data type: object
- subscription id
- outcome
- for admin requests, replace subscription identified by id with parameter data and save to database. No notification is sent.
- forbidden for non-admin requests
Internal HTTP Host
By default, HTTP requests submitted by NotifyBC back to itself will be sent to httpHost if defined or the host of the incoming HTTP request that spawns such internal requests. But if config internalHttpHost, which has no default value, is defined, for example in file /src/config.local.js
module.exports = {
- internalHttpHost: 'http://notifybc:3000',
-};
-Purge Data
This cron job purges old notifications, subscriptions and notification bounces. The default frequency of cron job and retention policy are defined by cron.purgeData config object in file /src/config.ts
module.exports = {
- cron: {
- purgeData: {
- // daily at 1am
- timeSpec: '0 0 1 * * *',
- pushNotificationRetentionDays: 30,
- expiredInAppNotificationRetentionDays: 30,
- nonConfirmedSubscriptionRetentionDays: 30,
- deletedBounceRetentionDays: 30,
- expiredAccessTokenRetentionDays: 30,
- defaultRetentionDays: 30,
- },
- },
-};
-where
- pushNotificationRetentionDays: the retention days of push notifications
- expiredInAppNotificationRetentionDays: the retention days of expired inApp notifications
- nonConfirmedSubscriptionRetentionDays: the retention days of non-confirmed subscriptions, i.e. all unconfirmed and deleted subscriptions
- deletedBounceRetentionDays: the retention days of deleted notification bounces
- expiredAccessTokenRetentionDays: the retention days of expired access tokens
- defaultRetentionDays: if any of the above retention day config item is omitted, default retention days is used as fall back.
To change a config item, set the config item in file /src/config.local.js. For example, to run cron jobs at 2am daily, add following object to /src/config.local.js
module.exports = {
- cron: {
- purgeData: {
- timeSpec: '0 0 2 * * *',
- },
- },
-};
-Dispatch Live Notifications
This cron job sends out future-dated notifications when the notification becomes current. The default config is defined by cron.dispatchLiveNotifications config object in file /src/config.ts
module.exports = {
- cron: {
- dispatchLiveNotifications: {
- // minutely
- timeSpec: '0 * * * * *',
- },
- },
-};
-Check Rss Config Updates
This cron job monitors RSS feed notification dynamic config items. If a config item is created, updated or deleted, the cron job starts, restarts, or stops the RSS-specific cron job. The default config is defined by cron.checkRssConfigUpdates config object in file /src/config.ts
module.exports = {
- cron: {
- checkRssConfigUpdates: {
- // minutely
- timeSpec: '0 * * * * *',
- },
- },
-};
-Note timeSpec doesn't control the RSS poll frequency (which is defined in dynamic configs and is service specific), instead it only controls the frequency to check for dynamic config changes.
Delete Notification Bounces
This cron job deletes notification bounces if the latest notification is deemed delivered successfully. The criteria of successful delivery are
- No bounce received since the latest notification started dispatching, and
- a configured time span has lapsed since the latest notification finished dispatching
The default config is defined by cron.deleteBounces config object in file /src/config.ts
module.exports = {
- cron: {
- deleteBounces: {
- // hourly
- timeSpec: '0 0 * * * *',
- minLapsedHoursSinceLatestNotificationEnded: 1,
- },
- },
-};
-where
- minLapsedHoursSinceLatestNotificationEnded is the time span
Re-dispatch Broadcast Push Notifications
`,22),g=c(`The default config is defined by cron.reDispatchBroadcastPushNotifications config object in file /src/config.ts
module.exports = {
- cron: {
- reDispatchBroadcastPushNotifications: {
- // minutely
- timeSpec: '0 * * * * *',
- },
- },
-};
-Clear Redis Datastore
This cron job clears Redis datastore used for SMS and email throttle. The job is enabled only if Redis is used. Datastore is cleared only when there is no broadcast push notifications in sending state. Without this cron job, updated throttle settings in config file will never take effect, and staled jobs in Redis datastore will not be cleaned up.
The default config is defined by cron.clearRedisDatastore config object in file /src/config.ts
module.exports = {
- cron: {
- clearRedisDatastore: {
- // hourly
- timeSpec: '0 0 * * * *',
- },
- },
-};
--
-
- simple
- province == 'BC' -
- - calling jmespath's built-in functions
- contains(province,'B') -
- - calling custom filter functions
- contains_ci(province,'b') -
- - compound
- (contains(province,'BC') || contains_ci(province,'b')) && city == 'Victoria' -
-
Acknowledgments
NotifyBC is built on a myriad of open source software. At runtime it also depends on a few services. Credit goes to their contributors. Notably
Administrator
The administrator API provides knowledge factor authentication to identify admin request by access token (aka API token in other literatures) associated with a registered administrator maintained in NotifyBC database. Because knowledge factor authentication is vulnerable to brute-force attack, administrator API based access token authentication is less favorable than admin ip list, client certificate, or OIDC authentication.
Avoid Administrator API
Administrator API was created to circumvent an OpenShift limitation - the source ip of a request initiated from an OpenShift pod cannot be exclusively allocated to the pod's project, rather it has to be shared by all OpenShift projects. Therefore it's difficult to impose granular access control based on source ip.
With the introduction client certificate in v2.4.0, most use cases, if not all, that need Administrator API including the OpenShift use case mentioned above can be addressed by client certificate. Therefore only use Administrator API sparingly as last resort.
To enable access token authentication,
a super-admin signs up an administrator
For example,
curl -X POST "http://localhost:3000/api/administrators" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"username\":\"Foo\",\"email\":\"user@example.com\",\"password\":\"secret\"}" -The step can also be completed in web console using button in Administrators panel.
Either super-admin or the user login to generate an access token
For example,
curl -X POST "http://localhost:3000/api/administrators/login" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"email\":\"user@example.com\",\"password\":\"secret\",\"tokenName\":\"myApp\"}" -The step can also be completed in web console GUI by an anonymous user using button at top right corner. Access token generated by GUI is valid for 12hrs.
Apply access token to either Authorization header or access_token query parameter to make authenticated requests. For example, to get a list of notifications
ACCESS_TOKEN=6Nb2ti5QEXIoDBS5FQGWIz4poRFiBCMMYJbYXSGHWuulOuy0GTEuGx2VCEVvbpBK - -# Authorization Header -curl -X GET -H "Authorization: $ACCESS_TOKEN" \ -http://localhost:3000/api/notifications - -# Query Parameter -curl -X GET http://localhost:3000/api/notifications?access_token=$ACCESS_TOKEN -In web console, once login as administrator, the access token is automatically applied.
Model Schemas
The Administrator API operates on three related sub-models - Administrator, UserCredential and AccessToken. An administrator has one and only one user credential and zero or more access tokens. Their relationship is diagramed as
Administrator
| Name | Attributes | ||||||
|---|---|---|---|---|---|---|---|
id |
| ||||||
| |||||||
username user name |
|
UserCredential
| Name | Attributes | ||||
|---|---|---|---|---|---|
id |
| ||||
password hashed password |
| ||||
userId foreign key to Administrator model |
|
AccessToken
| Name | Attributes | ||||
|---|---|---|---|---|---|
id 64-byte random alphanumeric characters |
| ||||
userId foreign key to Administrator model |
| ||||
ttl Time-to-live in seconds. If absent, access token never expires. |
| ||||
name Name of the access token. Can be used to identify applications that use the token. |
|
Sign Up
POST /administrators
-This API allows a super-admin to create an admin.
permissions required, one of
- super admin
inputs
user information
{ - "email": "string", - "password": "string", - "username": "string" -} -Password must meet following complexity rules:
- contains at least 10 characters
- contains at least one lower case character a-z
- contains at least one upper case character A-Z
- contains at least one numeric character 0-9
- contains at lease one special character in !_@#$&*
email must be unique. username is optional.
- required: true
- parameter type: request body
- data type: object
outcome
- for super-admin requests,
- an Administrator is generated, populated with email and username
- a UserCredential is generated, populated with hashed password
- Administrator is returned
- forbidden otherwise
- for super-admin requests,
Login
POST /administrators/login
-This API allows an admin to login and create an access token
inputs
user information
{ - "email": "user@example.com", - "password": "string", - "tokenName": "string", - "ttl": 0 -} -tokenName and ttl are optional. If ttl is absent, access token never expires.
- required: true
- parameter type: request body
- data type: object
outcome
- if login is successful
- a new AccessToken is generated with tokenName is saved to AccessToken.name and ttl is saved to AccessToken.ttl.
- the new access token is returned
{ - "token": "string" -} -
- forbidden otherwise
- if login is successful
Set Password
POST /administrators/{id}/user-credential
-This API allows a super-admin or admin to create or update password by id. An admin can only create/update own record.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
password
{ - "password": "string" -} -The password must meet complexity rules specified in Sign Up.
- required: true
- parameter type: request body
- data type: object
- outcome
- for super-admins or admin,
- hash the input password
- remove any existing UserCredential.password for the Administrator
- create a new UserCredential.password
- forbidden otherwise
- for super-admins or admin,
Get Administrators
GET /administrators
-This API allows a super-admin or admin to search for administrators. An admin can only search for own record
permissions required, one of
- super admin
- admin
inputs
a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"
Regardless, the filter will have to be parsed into a JSON object conforming to
{ - "where": {...}, - "fields": ..., - "order": ..., - "skip": ..., - "limit": ..., -} -All properties are optional. The syntax for each property is documented, respectively
- for where , see MongoDB Query Documents
- for fields , see Mongoose select
- for order, see Mongoose sort
- for skip, see MongoDB cursor.skip
- for limit, see MongoDB cursor.limit
outcome
- for super-admins, returns an array of Administrators matching the filter
- for admins, returns an array of one element - own record if the record matches the filter; empty array otherwise
- forbidden otherwise
example
to retrieve administrators created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/administrators?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D' -the value of the filter query parameter is URL-encoded stringified JSON object
{ - "where": { - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } - } -} -
Get Administrator Count
GET /administrators/count
-This API allows a super-admin or admin to count administrators by filter. An admin can only count own record therefore the number is at most 1.
permissions required, one of
- super admin
- admin
inputs
a where query parameter with value conforming to MongoDB Query Documents
- parameter name: where
- required: false
- parameter type: query
- data type: object
The value can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"
outcome
- for super-admins or admin, a count matching the filter
- forbidden otherwise
example
to retrieve the count of administrators created in year 2023 , run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/administrators/count?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D' -the value of the where query parameter is URL-encoded stringified JSON object
{ - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } -} -
Delete an Administrator
DELETE /administrators/{id}
-This API allows a super-admin or admin to delete administrator by id. An admin can only delete own record.
permissions required, one of
- super admin
- admin
inputs
- Administrator id
- required: true
- parameter type: path
- data type: string
- Administrator id
outcome
- for super-admins or admin
- all AccessToken of the Administrator are deleted
- the corresponding UserCredential is deleted
- the Administrator is deleted
- forbidden otherwise
- for super-admins or admin
Get an Administrator
GET /administrators/{id}
-This API allows a super-admin or admin to get administrator by id. An admin can only get own record.
permissions required, one of
- super admin
- admin
inputs
- Administrator id
- required: true
- parameter type: path
- data type: string
- Administrator id
outcome
- for super-admins or admin, returns the Administrator
- forbidden otherwise
Update an Administrator
PATCH /administrators/{id}
-This API allows a super-admin or admin to update administrator fields by id. An admin can only update own record.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
user information
{ - "username": "string", - "email": "string" -} -- required: true
- parameter type: request body
- data type: object
- outcome
- for super-admins or admin, updates the Administrator
- forbidden otherwise
Replace an Administrator
PUT /administrators/{id}
-This API allows a super-admin or admin to replace administrator records by id. An admin can only replace own record. This API is different from Update an Administrator in that update/patch needs only to contain fields that are changed, ie the delta, whereas replace/put needs to contain all fields to be saved.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
user information
{ - "username": "string", - "email": "string" -} -- required: true
- parameter type: request body
- data type: object
- outcome
- for super-admins or admin, updates the Administrator. If password is also supplied, the password is handled same way as Set Password API
- forbidden otherwise
Get an Administrator's AccessTokens
GET /administrators/{id}/access-tokens
-This API allows a super-admin or admin to get access tokens by Administrator id. An admin can only get own records.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"
Regardless, the filter will have to be parsed into a JSON object conforming to
{ - "where": {...}, - "fields": ..., - "order": ..., - "skip": ..., - "limit": ..., -} -All properties are optional. The syntax for each property is documented, respectively
- for where , see MongoDB Query Documents
- for fields , see Mongoose select
- for order, see Mongoose sort
- for skip, see MongoDB cursor.skip
- for limit, see MongoDB cursor.limit
outcome
- for super-admins or admin, a list of AccessTokens matching the filter
- forbidden otherwise
example
to retrieve access tokens created in year 2023 for administrator with id of 1, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/administrators/1/access-tokens?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D' -the value of the filter query parameter is URL-encoded stringified JSON object
{ - "where": { - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } - } -} -
Update an Administrator's AccessTokens
PATCH /administrators/{id}/access-tokens
-This API allows a super-admin or admin to update access tokens by Administrator id. An admin can only update own records.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
a where query parameter with value conforming to MongoDB Query Documents
- parameter name: where
- required: false
- parameter type: query
- data type: object
The value can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"
AccessToken information
{ - "ttl": 0, - "name": "string" -} -- required: true
- parameter type: request body
- data type: object
outcome
- for super-admins or admin, success count
- forbidden otherwise
example
to set ttl token to 0 for all access tokens created in year 2023 for administrator with id 1, run
curl -X PATCH --header 'Content-Type: application/json' 'http://localhost:3000/api/administrators/1/access-tokens?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D' -d '{"ttl":0}' -the value of the where query parameter is URL-encoded stringified JSON object
{ - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } -} -
Create an Administrator's AccessToken
POST /administrators/{id}/access-tokens
-This API allows a super-admin or admin to create an access token by Administrator id. An admin can only create own records.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
AccessToken information
{ - "ttl": 0, - "name": "string" -} -- required: true
- parameter type: request body
- data type: object
- outcome
- for super-admins or admin
- Create and save AccessToken
- return AccessToken created
- forbidden otherwise
- for super-admins or admin
Delete an Administrator's AccessTokens
DELETE /administrators/{id}/access-tokens
-This API allows a super-admin or admin to delete access tokens by Administrator id. An admin can only delete own records.
permissions required, one of
- super admin
- admin
inputs
Administrator id
- required: true
- parameter type: path
- data type: string
a where query parameter with value conforming to MongoDB Query Documents
- parameter name: where
- required: false
- parameter type: query
- data type: object
The value can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"
outcome
- for super-admins or admin
- delete all AccessToken under the Administrator matching the filter
- return success count
- forbidden otherwise
- for super-admins or admin
example
to delete all access tokens created in year 2023 for administrator with id 1, run
curl -X DELETE --header 'Accept: application/json' 'http://localhost:3000/api/administrators/1/access-tokens?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D' -the value of the where query parameter is URL-encoded stringified JSON object
{ - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } -} -
Bounce
Bounce handling involves recording bounce messages into bounce records, which are implemented using this bounce API and model. Administrator can view bounce records in web console or through API explorer. Bounce record is for internal use and should be read-only under normal circumstances.
Model Schema
The API operates on following data model fields:
| Name | Attributes | ||||
|---|---|---|---|---|---|
channel name of the delivery channel. Valid values: email, sms. |
| ||||
userChannelId user's delivery channel id, for example, email address. |
| ||||
hardBounceCount number of hard bounces recorded so far |
| ||||
state bounce record state. Valid values: active, deleted. |
| ||||
bounceMessages array of recorded bounce messages. Each element is an object containing the date bounce message was received and the message itself. |
| ||||
latestNotificationStarted latest notification started date. |
| ||||
latestNotificationEnded latest notification ended date. |
| ||||
created date and time bounce record was created |
| ||||
updated date and time of bounce record was last updated |
| ||||
id config id |
|
Configuration
The configuration API, accessible by only super-admin requests, is used to define dynamic configurations. Dynamic configuration is needed in situations like
- RSA key pair generated automatically at boot time if not present
- service-specific subscription confirmation request message template
Model Schema
The API operates on following configuration data model fields:
| Name | Attributes | ||||
|---|---|---|---|---|---|
id config id |
| ||||
name config name |
| ||||
value config value. |
| ||||
serviceName name of the service the config applicable to |
|
Get Configurations
GET /configurations
-permissions required, one of
- super admin
- admin
inputs
a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"
Regardless, the filter will have to be parsed into a JSON object conforming to
{ - "where": {...}, - "fields": ..., - "order": ..., - "skip": ..., - "limit": ..., -} -All properties are optional. The syntax for each property is documented, respectively
- for where , see MongoDB Query Documents
- for fields , see Mongoose select
- for order, see Mongoose sort
- for skip, see MongoDB cursor.skip
- for limit, see MongoDB cursor.limit
outcome
For admin request, a list of config items matching the filter; forbidden for user request
example
to retrieve configs created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/configurations?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D' -the value of the filter query parameter is URL-encoded stringified JSON object
{ - "where": { - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } - } -} -
Create a Configuration
POST /configurations
-permissions required, one of
- super admin
- admin
inputs
- an object containing configuration data model fields. At a minimum all required fields that don't have a default value must be supplied. Id field should be omitted since it's auto-generated. The API explorer only created an empty object for field value but you should populate the child fields.
- parameter name: data
- required: true
- parameter type: body
- data type: object
- an object containing configuration data model fields. At a minimum all required fields that don't have a default value must be supplied. Id field should be omitted since it's auto-generated. The API explorer only created an empty object for field value but you should populate the child fields.
outcome
NotifyBC performs following actions in sequence
- if it’s a user request, error is returned
- inputs are validated. For example, required fields without default values must be populated. If validation fails, error is returned
- if config item is notification with field value.rss populated, and if the field value.httpHost is missing, it is generated using this request's HTTP protocol , host name and port.
- item is saved to database
example
see the cURL command on how to create a dynamic subscription config
Update a Configuration
PATCH /configurations/{id}
-permissions required, one of
- super admin
- admin
inputs
- configuration id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- an object containing fields to be updated.
- parameter name: data
- required: true
- parameter type: body
- data type: object
- configuration id
outcome
Similar to POST except field update is always updated with current timestamp.
Update Configurations
PATCH /configurations
-permissions required, one of
- super admin
- admin
inputs
a where query parameter with value conforming to MongoDB Query Documents
- parameter name: where
- required: false
- parameter type: query
- data type: object
The value can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"
an object containing fields to be updated.
- required: true
- parameter type: body
- data type: object
outcome
Similar to POST except field update is always updated with current timestamp.
example
to set serviceName to myService for all configs created in year 2023 , run
curl -X PATCH --header 'Content-Type: application/json' 'http://localhost:3000/api/configurations?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D' -d @- << EOF -{ - "serviceName": "myService", -} -EOF -the value of the where query parameter is URL-encoded stringified JSON object
{ - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } -} -
Delete a Configuration
DELETE /configurations/{id}
-permissions required, one of
- super admin
- admin
inputs
- configuration id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- configuration id
outcome
For admin request, delete the config item requested; forbidden for user request
Replace a Configuration
PUT /configurations/{id}
-This API is intended to be only used by admin web console to modify a configuration.
permissions required, one of
- super admin
- admin
inputs
- configuration id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- configuration data
- parameter name: data
- required: true
- parameter type: body
- data type: object
- configuration id
outcome
For admin requests, replace configuration identified by id with parameter data and save to database.
Notification
The notification API encapsulates the backend workflow of staging and dispatching a message to targeted user after receiving the message from event source.
Depending on whether an API call comes from user browser as a user request or from an authorized server application as an admin request, NotifyBC applies different permissions. Admin request allows full CRUD operations. An authenticated user request, on the other hand, are only allowed to get a list of in-app pull notifications targeted to the current user and changing the state of the notifications. An unauthenticated user request can not access any API.
When a notification is created by the event source server application, the message is saved to database prior to responding to API caller. In addition, for push notification, the message is delivered immediately, i.e. the API call is synchronous. For in-app pull notification, the message, which by default is in state new, can be retrieved later on by browser user request. A user request can only get the list of in-app messages targeted to the current user. A user request can then change the message state to read or deleted depending on user action. A deleted message cannot be retrieved subsequently by user requests, but the state can be updated given the correct id.
Deleted message is still kept in database.
NotifyBC provides API for deleting a notification. For the purpose of auditing and recovery, this API only marks the state field as deleted rather than deleting the record from database.
undo in-app notification deletion within a session
Because "deleted" message is still kept in database, you can implement undo feature for in-app notification as long as the message id is retained prior to deletion within the current session. To undo, call update API to set desired state.
In-app pull notification also supports message expiration by setting a date in field validTill. An expired message cannot be retrieved by user requests.
A message, regardless of push or pull, can be unicast or broadcast. A unicast message is intended for an individual user whereas a broadcast message is intended for all confirmed subscribers of a service. A unicast message must have field userChannelId populated. The value of userChannelId is channel dependent. In the case of email for example, this would be user's email address. A broadcast message must set isBroadcast to true and leave userChannelId empty.
Why field isBroadcast?
Unicast and broadcast message can be distinguished by whether field userChannelId is empty or not alone. So why the extra field isBroadcast? This is in order to prevent inadvertent marking a unicast message broadcast by omitting userChannelId or populating it with empty value. The precaution is necessary because in-app notifications may contain personalized and confidential information.
NotifyBC ensures the state of an in-app broadcast message is isolated by user, so that for example, a message read by one user is still new to another user. To achieve this, NotifyBC maintains two internal fields of array type - readBy and deletedBy. When a user request updates the state field of an in-app broadcast message to read or deleted, instead of altering the state field, NotifyBC appends the current user to readBy or deletedBy list. When user request retrieving in-app messages, the state field of the broadcast message in HTTP response is updated based on whether the user exists in field deletedBy and readBy. If existing in both fields, deletedBy takes precedence (the message therefore is not returned). The record in database, meanwhile, is unchanged. Neither field deletedBy nor readBy is visible to user request.
Model Schema
The API operates on following notification data model fields:
| Name | Attributes | ||||||
|---|---|---|---|---|---|---|---|
id notification id |
| ||||||
serviceName name of the service |
| ||||||
channel name of the delivery channel. Valid values: inApp, email, sms. |
| ||||||
userChannelId user's delivery channel id, for example, email address. For unicast inApp notification, this is authenticated user id. When sending unicast push notification, either userChannelId or userId is required. |
| ||||||
userId authenticated user id. When sending unicast push notification, either userChannelId or userId is required. |
| ||||||
state state of notification. Valid values: new, read (inApp only), deleted (inApp only), sent (push only) or error. For inApp broadcast notification, if the user has read or deleted the message, the value of this field retrieved by admin request will still be new. The state for the user is tracked in fields readBy and deletedBy in such case. For user request, the value contains correct state. |
| ||||||
created date and time of creation |
| ||||||
updated date and time of last update |
| ||||||
isBroadcast whether it's a broadcast message. A broadcast message should omit userChannelId and userId, in addition to setting isBroadcast to true |
| ||||||
skipSubscriptionConfirmationCheck When sending unicast push notification, whether or not to verify if the recipient has a confirmed subscription. This field allows subscription information be kept elsewhere and NotifyBC be used as a unicast push notification gateway only. |
| ||||||
validTill expiration date-time of the message. Applicable to inApp notification only. |
| ||||||
invalidBefore date-time in the future after which the notification can be dispatched. |
| ||||||
an object whose child fields are channel dependent:
|
| ||||||
httpHost This field is used to replace token {http_host} in push notification message template during mail merge and overrides config httpHost. |
| ||||||
asyncBroadcastPushNotification this field determines if the API call to create an immediate (i.e. not future-dated) broadcast push notification is asynchronous or not. If omitted, the API call is synchronous, i.e. the API call blocks until notifications to all subscribers have been dispatched. If set, valid values and corresponding behaviors are
|
| ||||||
the event that triggers the notification, for example, a RSS feed item when the notification is generated automatically by RSS cron job. Field data serves two purposes
|
| ||||||
broadcastPushNotificationSubscriptionFilter a string conforming to jmespath filter expressions syntax after the question mark (?). The filter is matched against the data field of the subscription. Examples of filter
|
| ||||||
readBy this is an internal field to track the list of users who have read an inApp broadcast message. It's not visible to a user request. |
| ||||||
deletedBy this is an internal field to track the list of users who have marked an inApp broadcast message as deleted. It's not visible to a user request. |
| ||||||
dispatch this is an internal field to track the broadcast push notification dispatch outcome. It consists of up to four arrays
|
|
Get Notifications
GET /notifications
-permissions required, one of
- super admin
- admin
- authenticated user
inputs
a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"
Regardless, the filter will have to be parsed into a JSON object conforming to
{ - "where": {...}, - "fields": ..., - "order": ..., - "skip": ..., - "limit": ..., -} -All properties are optional. The syntax for each property is documented, respectively
- for where , see MongoDB Query Documents
- for fields , see Mongoose select
- for order, see Mongoose sort
- for skip, see MongoDB cursor.skip
- for limit, see MongoDB cursor.limit
outcome
- for admin requests, returns unabridged array of notification data matching the filter
- for authenticated user requests, in addition to filter, following constraints are imposed on the returned array
- only inApp notifications
- only non-deleted notifications. For broadcast notification, non-deleted means not marked by current user as deleted
- only non-expired notifications
- for unicast notifications, only the ones targeted to current user
- if current user is in readBy, then the state is changed to read
- the internal field readBy and deletedBy are removed
- forbidden to anonymous user requests
example
to retrieve notifications created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/notifications?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D' -the value of the filter query parameter is URL-encoded stringified JSON object
{ - "where": { - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } - } -} -
Get Notification Count
GET /notifications/count
-permissions required, one of
- super admin
- admin
- authenticated user
inputs
a where query parameter with value conforming to MongoDB Query Documents
- parameter name: where
- required: false
- parameter type: query
- data type: object
The value can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"
outcome
Validations rules are the same as GET /notifications. If passed, the output is a count of notifications matching the query
{ - "count": <number> -} -example
to retrieve the count of notifications created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/notifications/count?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D' -the value of the where query parameter is URL-encoded stringified JSON object
{ - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } -} -
Create/Send Notifications
POST /notifications
-permissions required, one of
- super admin
- admin
inputs
- an object containing notification data model fields. At a minimum all required fields that don't have a default value must be supplied. Id field should be omitted since it's auto-generated. The API explorer only created an empty object for field message but you should populate the child fields according to model schema
- parameter name: data
- required: true
- parameter type: body
- data type: object
- an object containing notification data model fields. At a minimum all required fields that don't have a default value must be supplied. Id field should be omitted since it's auto-generated. The API explorer only created an empty object for field message but you should populate the child fields according to model schema
outcome
NotifyBC performs following actions in sequence
if it's a user request, error is returned
inputs are validated. If validation fails, error is returned. In particular, for unicast push notification, the recipient as identified by either userChannelId or userId must have a confirmed subscription if field skipSubscriptionConfirmationCheck is not set to true. If skipSubscriptionConfirmationCheck is set to true, then the subscription check is skipped, but in such case the request must contain userChannelId, not userId as subscription data is not queried to obtain userChannelId from userId.
for push notification, if field httpHost is empty, it is populated based on request's http protocol and host.
the notification request is saved to database
if the notification is future-dated, then all subsequent request processing is skipped and response is sent back to user. Steps 7-11 below will be carried out later on by the cron job when the notification becomes current.
if it's an async broadcast push notification, then response is sent back to user but steps 7-12 below is processed separately
for unicast push notification, the message is dispatched to targeted user; for broadcast push notification, following actions are performed:
number of confirmed subscriptions is retrieved
the subscriptions are partitioned and processed concurrently as described in config section Broadcast Push Notification Task Concurrency
when processing an individual subscription,
- if the subscription has filter rule defined in field broadcastPushNotificationFilter and notification contains field data, then the data is matched against the filter rule. Notification message is only dispatched if there is a match.
- if the notification has filter rule defined in field broadcastPushNotificationSubscriptionFilter and subscription contains field data, then the data is matched against the filter rule. Notification message is only dispatched if there is a match.
If the subscription failed to pass any of the two filters, and if both guaranteedBroadcastPushDispatchProcessing and logSkippedBroadcastPushDispatches are true, the subscription id is logged to dispatch.skipped
Regardless of unicast or broadcast, mail merge is performed on messages before dispatching.
the state of push notification is updated to sent or error depending on sending status. For broadcast push notification, the dispatching could be failed only for a subset of users. In such case, the field dispatch.failed contains a list of objects of {userChannelId, subscriptionId, error} the message failed to deliver to, but the state will still be set to sent.
For broadcast push notifications, if guaranteedBroadcastPushDispatchProcessing is true, then field dispatch.successful is populated with a list of subscriptionId of the successful dispatches.
For push notifications, the bounce records of successful dispatches are updated
the updated notification is saved back to database
if it's an async broadcast push notification with a callback url, then the url is called with POST verb containing the notification with updated status as the request body
for synchronous notification, the saved record is returned unless there is an error saving to database, in which case error is returned
example
To send a unicast email push notification, copy and paste following json object to the data value box in API explorer, change email addresses as needed, and click Try it out! button:
{ - "serviceName": "education", - "userChannelId": "foo@bar.com", - "skipSubscriptionConfirmationCheck": true, - "message": { - "from": "no_reply@bar.com", - "subject": "test", - "textBody": "This is a test" - }, - "channel": "email" -} -As the result, foo@bar.com should receive an email notification even if the user is not a confirmed subscriber, and following json object is returned to caller upon sending the email successfully:
{ - "serviceName": "education", - "state": "sent", - "userChannelId": "foo@bar.com", - "skipSubscriptionConfirmationCheck": true, - "message": { - "from": "no_reply@bar.com", - "subject": "test", - "textBody": "This is a test" - }, - "created": "2016-09-30T20:37:06.011Z", - "updated": "2016-09-30T20:37:06.011Z", - "channel": "email", - "isBroadcast": false, - "id": "57eeccf23427b61a4820775e" -} -
Update a Notification
PATCH /notifications/{id}
-This API is mainly used for updating an inApp notification.
permissions required, one of
- super admin
- admin
- authenticated user
inputs
- notification id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- an object containing fields to be updated.
- parameter name: data
- required: true
- parameter type: body
- data type: object
- notification id
outcome
- for user requests, NotifyBC performs following actions in sequence
- for unicast notification, if the notification is not targeted to current user, error is returned
- all fields except for state are discarded from the input
- for broadcast notification, current user id in appended to array readBy or deletedBy, depending on whether state is read or deleted, unless the user id is already in the array. The state field itself is then discarded
- the notification identified by id is merged with the updates and saved to database
- HTTP response code 204 is returned, unless there is error.
- admin requests are allowed to update any field
- for user requests, NotifyBC performs following actions in sequence
Delete a Notification
This API is mainly used for marking an inApp notification deleted. It has the same effect as updating a notification with state set to deleted.
DELETE /notifications/{id}
-- permissions required, one of
- super admin
- admin
- authenticated user
- inputs
- notification id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- notification id
- outcome: same as the outcome of Update a Notification with state set to deleted.
Replace a Notification
PUT /notifications/{id}
-This API is intended to be only used by admin web console to modify a notification in new state. Notifications in such state are typically future-dated or of channel in-app.
permissions required, one of
- super admin
- admin
inputs
- notification id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- notification data
- parameter name: data
- required: true
- parameter type: body
- data type: object
- notification id
outcome
NotifyBC process the request same way as Create/Send Notifications except that notification data is saved with id supplied in the parameter, replacing existing one.
API Overview
NotifyBC's core function is implemented by two models - subscription and notification. Other models - configuration, administrator and bounces etc, are for administrative purposes. A model determines the underlying database schema and the API. The APIs displayed in the web console (by default http://localhost:3000) and API explorer are also grouped by models. Click on a model in API explorer, say notification, to explore the operations on that model. Model specific APIs are available here:
Subscription
The subscription API encapsulates the backend workflow of user subscription and un-subscription of push notification service. Depending on whether a API call comes from user browser as a user request or from an authorized server as an admin request, NotifyBC applies different validation rules. For user requests, the notification channel entered by user is unconfirmed. A confirmation code will be associated with this request. The confirmation code can be created in one of two ways:
- by NotifyBC based on channel dependent subscription.confirmationRequest.<channel>.confirmationCodeRegex config.
- by a trusted third party. This trusted third party encrypts the confirmation code using the public RSA key of the NotifyBC instance (see more about RSA Key Config) and pass the encrypted confirmation code to NotifyBC via user browser in the same subscription request. NotifyBC then decrypts to obtain the confirmation code. This method allows user subscribe to multiple notification services provided by NotifyBC instances in different trust domains (i.e. service providers) and only have to confirm the subscription channel once during one browser session. In such case only one NotifyBC instance should be chosen to deliver confirmation request to user.
Equipped with the confirmation code and a message template, NotifyBC can now send out confirmation request to unconfirmed subscription channel. At a minimum this confirmation request should contain the confirmation code. When user receives the message, he/she echos the confirmation code back to a NotifyBC provided API to verify against saved record. If match, the state of the subscription request is changed to confirmed.
For admin requests, NotifyBC can still perform the above confirmation process. But admin request has full CRUD privilege, including set the subscription state to confirmed, bypassing the confirmation process.
The workflow of user subscribing to notification services offered by a single service provider is illustrated by sequence diagram below. In this case, the confirmation code is generated by NotifyBC. 
In the case user subscribing to notifications offered by different service providers in separate trust domains, the confirmation code is generated by a third-party server app trusted by all NotifyBC instances. Following sequence diagram shows the workflow. The diagram indicates NotifyBC API Server 2 is chosen to send confirmation request.
Model Schema
The API operates on following subscription data model fields:
| Name | Attributes | ||||||
|---|---|---|---|---|---|---|---|
serviceName name of the service. Avoid prefixing the name with underscore (_), or it may conflict with internal implementation. |
| ||||||
channel name of the delivery channel. Valid values: email and sms. Notice inApp is invalid as in-app notification doesn't need subscription. |
| ||||||
userChannelId user's delivery channel id, for example, email address |
| ||||||
id subscription id |
| ||||||
state state of subscription. Valid values: unconfirmed, confirmed, deleted |
| ||||||
userId user id. Auto-populated for authenticated user requests. |
| ||||||
created date and time of creation |
| ||||||
updated date and time of last update |
| ||||||
confirmationRequest an object containing these child fields
|
| ||||||
broadcastPushNotificationFilter a string conforming to jmespath filter expressions syntax after the question mark (?). The filter is matched against the data field of the subscription. Examples of filter
|
| ||||||
An object used by
|
| ||||||
unsubscriptionCode generated randomly according to RegEx config anonymousUnsubscription.code.regex during anonymous subscription if config anonymousUnsubscription.code.required is set to true |
| ||||||
unsubscribedAdditionalServices generated if parameter additionalServices is supplied in unsubscription request. Contains 2 sub-fields: ids and names, each being a list identifying the additional unsubscribed subscriptions. |
|
Get Subscriptions
GET /subscriptions
-permissions required, one of
- super admin
- admin
- authenticated user
inputs
a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
- required: false
- parameter type: query
- data type: object
The filter can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"
Regardless, the filter will have to be parsed into a JSON object conforming to
{ - "where": {...}, - "fields": ..., - "order": ..., - "skip": ..., - "limit": ..., -} -All properties are optional. The syntax for each property is documented, respectively
- for where , see MongoDB Query Documents
- for fields , see Mongoose select
- for order, see Mongoose sort
- for skip, see MongoDB cursor.skip
- for limit, see MongoDB cursor.limit
outcome
- for admin requests, returns unabridged array of subscription data matching the filter
- for authenticated user requests, in addition to filter, following constraints are imposed on the returned array
- only non-deleted subscriptions
- only subscriptions created by the user
- the confirmationRequest field is removed.
- forbidden for anonymous user requests
example
to retrieve subscriptions created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/subscriptions?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D' -the value of the filter query parameter is URL-encoded stringified JSON object
{ - "where": { - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } - } -} -
Get Subscription Count
GET /subscriptions/count
-permissions required, one of
- super admin
- admin
- authenticated user
inputs
a where query parameter with value conforming to MongoDB Query Documents
- parameter name: where
- required: false
- parameter type: query
- data type: object
The value can be expressed as either
- URL-encoded stringified JSON object (see example below); or
- in the format supported by qs, for example
?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"
outcome
Validations rules are the same as GET /subscriptions. If passed, the output is a count of subscriptions matching the query
{ - "count": <number> -} -example
to retrieve the count of subscriptions created in year 2023, run
curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/subscriptions/count?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D' -the value of the where query parameter is URL-encoded stringified JSON object
{ - "created": { - "$gte": "2023-01-01", - "$lt": "2024-01-01" - } -} -
Create a Subscription
POST /subscriptions
-inputs
- an object containing subscription data model fields. At a minimum all required fields that don't have a default value must be supplied. Id field should be omitted since it's auto-generated.
- parameter name: data
- required: true
- parameter type: body
- data type: object
- an object containing subscription data model fields. At a minimum all required fields that don't have a default value must be supplied. Id field should be omitted since it's auto-generated.
outcome
NotifyBC performs following actions in sequence
inputs are validated. If validation fails, error is returned.
for user requests, the state field is forced to unconfirmed
for authenticated user request, userId field is populated with authenticated userId
otherwise, unsubscriptionCode is generated if config subscription.anonymousUnsubscription.code.required is true, unless if the request is made by admin and the field is already populated
if confirmationRequest.confirmationCodeEncrypted is populated, a confirmation code is generated by decrypting this field using private RSA key, then put decrypted confirmation code to field confirmationRequest.confirmationCode
otherwise, for user requests and for admin requests missing message template, the message template is set to configured value. Then, if confirmationRequest.confirmationCodeRegex is populated, a confirmation code is generated conforming to regex and put to field confirmationRequest.confirmationCode
the subscription request is saved to database.
if confirmationRequest.sendRequest is true, then a message is sent to userChannelId. The message template is determined by
- if detectDuplicatedSubscription is true and there is already a confirmed subscription to the same serviceName, channel and userChannelId, then message is sent using duplicatedSubscriptionNotification as template;
- otherwise, a confirmation request is sent to using the template fields in confirmationRequest.
Mail merge is performed on the template regardless.
The subscription data, including auto-generated id, is returned as response unless there is error when sending confirmation request or saving to database. For user request, some fields containing sensitive information such as confirmationRequest are removed prior to sending the response.
examples
- To subscribe a user to service education, copy and paste following json object to the data value box in API explorer, change email addresses as needed, and click Try it out! button:
{ - "serviceName": "education", - "channel": "email", - "userChannelId": "foo@bar.com" -} -As a result, foo@bar.com should receive an email confirmation request, and following json object is returned to caller upon sending the email successfully for admin request:
{ - "serviceName": "education", - "channel": "email", - "userChannelId": "foo@bar.com", - "state": "unconfirmed", - "confirmationRequest": { - "confirmationCodeRegex": "\\d{5}", - "sendRequest": true, - "from": "no_reply@bar.com", - "subject": "confirmation", - "textBody": "Enter {confirmation_code} on screen", - "confirmationCode": "45304" - }, - "created": "2016-10-03T17:35:40.202Z", - "updated": "2016-10-03T17:35:40.202Z", - "id": "57f296ec7eead50554c61de7" -} -For non-admin request, the field confirmationRequest is removed from response, and field userId is populated if request is authenticated:
{ - "serviceName": "education", - "channel": "email", - "userChannelId": "foo@bar.com", - "state": "unconfirmed", - "userId": "<user_id>", - "created": "2016-10-03T18:17:09.778Z", - "updated": "2016-10-03T18:17:09.778Z", - "id": "57f2a0a5b1aa0e2d5009eced" -} -To subscribe a user to service education with RSA public key encrypted confirmation code supplied, POST following request
{ - "serviceName": "education", - "channel": "email", - "userChannelId": "foo@bar.com", - "confirmationRequest": { - "confirmationCodeEncrypted": "<encrypted-confirmation-code>", - "sendRequest": true, - "from": "no_reply@bar.com", - "subject": "confirmation", - "textBody": "Enter {confirmation_code} on screen" - } -} -As a result, NotifyBC will decrypt the confirmation code using the private RSA key, replace placeholder {confirmation_code} in the email template with the confirmation code, and send confirmation request to foo@bar.com.
Verify a Subscription
GET /subscriptions/{id}/verify
-inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- confirmation code
- parameter name: confirmationCode
- required: true
- parameter type: query
- data type: string
- whether or not replacing existing subscriptions
- parameter name: replace
- required: false
- parameter type: query
- data type: boolean
- subscription id
outcome
NotifyBC performs following actions in sequence
- the subscription identified by id is retrieved
- for user request, the userId of the subscription is checked against current request user, if not match, error is returned; otherwise
- input parameter confirmationCode is checked against confirmationRequest.confirmationCode. If not match, error is returned; otherwise
- if input parameter replace is supplied and set to true, then existing confirmed subscriptions from the same serviceName, channel and userChannelId are deleted. No unsubscription acknowledgement notification is sent
- state is set to confirmed
- the subscription is saved back to database
- displays acknowledgement message according to configuration
example
to verify a subscription with id abc, confirmation code 12345, and delete existing confirmed subscriptions once verified, run
curl 'http://localhost:3000/api/subscriptions/abc/verify?confirmationCode=12345&replace=true' -
Update a Subscription
PATCH /subscriptions/{id}
-This API is used by authenticated user to change user channel id (such as email address) and resend confirmation code.
permissions required, one of
- super admin
- admin
- authenticated user
inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- an object containing fields to be updated.
- parameter name: data
- required: true
- parameter type: body
- data type: object
- subscription id
outcome
NotifyBC processes the request similarly as creating a subscription except during input validation it imposes following extra constraints to user request
- only fields userChannelId, state and confirmationRequest can be updated
- when changing userChannelId, confirmationRequest must also be supplied
- if userChannelId is different from the saved record, state is forced to unconfirmed.
Delete a Subscription (unsubscribing)
DELETE /subscriptions/{id}?unsubscriptionCode={unsubscriptionCode}&additionalServices[]={additionalServices}&userChannelId={userChannelId}
-or
-GET /subscriptions/{id}/unsubscribe?unsubscriptionCode={unsubscriptionCode}&additionalServices[]={additionalServices}&userChannelId={userChannelId}
-inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- unsubscription code for anonymous request
- parameter name: unsubscriptionCode
- required: false
- parameter type: query
- data type: string
- additional service names to unsubscribe
- parameter name: additionalServices
- required: false
- parameter type: query
- data type: array of strings. If there is only one item and its value is _all, then all services the user subscribed on this NotifyBC instance are included. Supply multiple items by repeating this query parameter.
- user channel id for extended validation
- parameter name: userChannelId
- required: false
- parameter type: query
- data type: string
- subscription id
outcome
NotifyBC performs following actions in sequence
- the subscription identified by id is retrieved
- for user request,
- if request is authenticated, the userId of the subscription is checked against current request user, if not match, request is rejected
- if request is anonymous, and server is configured to require unsubscription code, the input unsubscription code is matched against the unsubscriptionCode field. Request is rejected if not match. In addition, if input parameter userChannelId is populated but doesn't match, request is rejected
- if the subscription state is not confirmed, request is rejected
- if additionalServices is populated, database is queried to retrieve the serviceName and id fields of the additional subscriptions
- the field state is set to deleted for the subscription identified by id as well as additional subscriptions retrieved in previous step
- if additionalServices is not empty, the service names and ids of the additional subscriptions are added to field unsubscribedAdditionalServices of the subscription identified by id to allow bulk undo unsubscription later on
- for anonymous unsubscription, an acknowledgement notification is sent to user if configured so
- returns
- for anonymous request, either the message or redirect as configured in anonymousUnsubscription.acknowledgements.onScreen
- for authenticated user or admin requests, number of records affected or error message if occurred.
- To allow an anonymous subscriber to unsubscribe single subscription, provide url token {unsubscription_url} in notification messages. When sending notification, mail merge is performed on the token resolving to the GET API url and parameters.
- To allow an anonymous subscriber to unsubscribe all subscriptions, provide url token {unsubscription_all_url} in notification messages.
Un-deleting a Subscription
GET /subscriptions/{id}/unsubscribe/undo
-This API allows an anonymous subscriber to undo an unsubscription.
inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- unsubscription code
- parameter name: unsubscriptionCode
- required: false
- parameter type: query
- data type: string
- subscription id
outcome
NotifyBC performs following actions in sequence
- the subscription identified by id is retrieved
- for user request,
- if request is anonymous, and server is configured to require unsubscription code, the input unsubscription code is matched against the unsubscriptionCode field. Request is rejected if not match
- if request is authenticated, request is rejected
- if the subscription state is not deleted, request is rejected
- the field state is set to confirmed for the subscription identified by id as well as additional subscriptions identified in field unsubscribedAdditionalServices, if populated
- field unsubscribedAdditionalServices is removed if populated
- returns either the message or redirect as configured in anonymousUndoUnsubscription
example
To allow an anonymous subscriber to undo unsubscription, provide link token {unsubscription_reversion_url} in unsubscription acknowledgement notification, which is by default set. When sending notification, mail merge is performed on this token resolving to the API url and parameters.
Get all services with confirmed subscribers
GET /subscriptions/services
-This API is designed to facilitate implementing autocomplete for admin web console.
- permissions required, one of
- super admin
- admin
- inputs - none
- outcome
- for admin requests, returns an array of unique service names with confirmed subscribers
- forbidden for non-admin requests
Replace a Subscription
PUT /subscriptions/{id}
-This API is intended to be only used by admin web console to modify a subscription without triggering any confirmation or acknowledgement notification.
- permissions required, one of
- super admin
- admin
- permissions required, one of
- super admin
- admin
- authenticated user
- inputs
- subscription id
- parameter name: id
- required: true
- parameter type: path
- data type: string
- subscription data
- parameter name: data
- required: true
- parameter type: body
- data type: object
- subscription id
- outcome
- for admin requests, replace subscription identified by id with parameter data and save to database. No notification is sent.
- forbidden for non-admin requests
Benchmarks
tl;dr
A NotifyBC server node can deliver 1 million emails in as little as 1 hour to a SMTP server node. SMTP server node's disk I/O is the bottleneck in such case. Throughput can be improved through horizontal scaling.
When NotifyBC is used to deliver broadcast push notifications to a large number of subscribers, probably the most important benchmark is throughput. The benchmark is especially critical if a latency cap is desired. To facilitate capacity planning, load testing on the email channel has been conducted. The test environment, procedure, results and performance tuning advices are provided hereafter.
Environment
Hardware
Two computers, connected by 1Gbps LAN, are used to host
- NotifyBC
- Mac Mini Late 2012 model
- Intel core i7-3615QM
- 16GB RAM
- 2TB HDD
- SMTP and mail delivery
- Lenovo ThinkCentre M Series 2015 model
- Intel core i5-3470
- 8GB RAM
- 256GB SSD
Software Stack
The test was performed in August 2017. Unless otherwise specified, the versions of all other software were reasonably up-to-date at the time of testing.
NotifyBC
- MacOS Sierra Version 10.12.6
- Virtualbox VM with 8vCPU, 10GB RAM, created using miniShift v1.3.1+f4900b07
- OpenShift 1.5.1+7b451fc with metrics
- default NotifyBC OpenShift installation, which contains following relevant pods
- 1 mongodb pod with 1 core, 1GiB RAM limit
- a variable number of Node.js app pods each with 1 core, 1GiB RAM limit. The number varies by test runs as indicated in result.
SMTP and mail delivery
- Windows 7 host
- Virtualbox VM with 4 vCPU, 3.5GB RAM, running Windows Server 2012
- added SMTP Server feature
- in SMTP Server properties dialog box, uncheck all of following boxes in Messages tab
- Limit message size to (KB)
- Limit session size to (KB)
- Limit number of messages per connection to
- Limit number of recipients per message to
Procedure
update or create file /src/config.local.js through configMap. Add sections for SMTP server and a custom filter function
var _ = require('lodash'); -module.exports = { - smtp: { - host: '<smtp-vm-ip-or-hostname>', - secure: false, - port: 25, - pool: true, - direct: false, - maxMessages: Infinity, - maxConnections: 50, - }, - notification: { - broadcastCustomFilterFunctions: { - /*jshint camelcase: false */ - contains_ci: { - _func: function (resolvedArgs) { - if (!resolvedArgs[0] || !resolvedArgs[1]) { - return false; - } - return ( - _.toLower(resolvedArgs[0]).indexOf(_.toLower(resolvedArgs[1])) >= - 0 - ); - }, - _signature: [ - { - types: [2], - }, - { - types: [2], - }, - ], - }, - }, - }, -}; -create a number of subscriptions in bulk using script bulk-post-subs.js. To load test different email volumes, you can create bulk subscriptions in different services. For example, generate 10 subscriptions under service named load10; 1,000,000 subscriptions under service load1000000 etc. bulk-post-subs.js takes userChannelId and other optional parameters
$ node dist/utils/load-testing/bulk-post-subs.js -h -Usage: node bulk-post-subs.js [Options] <userChannelId> -[Options]: --a, --api-url-prefix=<string> api url prefix. default to http://localhost:3000/api --c, --channel=<string> channel. default to email --s, --service-name=<string> service name. default to load --n, --number-of-subscribers=<int> number of subscribers. positive integer. default to 1000 --f, --broadcast-push-notification-filter=<string> broadcast push notification filter. default to "contains_ci(title,'vancouver') || contains_ci(title,'victoria')" --h, --help display this help -The generated subscriptions contain a filter, hence all load testing results below included time spent on filtering.
launch load testing using script curl-ntf.sh, which takes following optional parameters
dist/utils/load-testing/curl-ntf.sh <apiUrlPrefix> <serviceName> <senderEmail> -The script will print start time and the time taken to dispatch the notification.
Results
| email count | time taken (min) | throughput (#/min) | app pod count | notes on bottleneck |
|---|---|---|---|---|
| 1,000,000 | 71.5 | 13,986 | 1 | app pod cpu capped |
| 100,000 | 5.8 | 17,241 | 2 | smtp vm disk queue length hits 1 frequently |
| 1,000,000 | 57 | 17,544 | 2 | smtp vm disk queue length hits 1 frequently |
| 1,000,000 | 57.8 | 17,301 | 3 | smtp vm disk queue length hits 1 frequently |
Test runs using other software or configurations described below have also been conducted. Because throughput is significantly lower, results are not shown
- using Linux sendmail SMTP. The throughput of a 4-vCPU Linux VM is about the same as a 1-vCPU Windows SMTP server. Bottleneck in such case is the CPU of SMTP server.
- Reducing NotifyBC app pod's resource limit to 100 millicore CPU and 512MiB RAM. Even when scaled up pod count to 15, throughput is still about 1/3 of a 1-core pod.
Here is a sample email saved onto the mail drop folder of SMTP server.
Comparison to Other Benchmarks
According to Baseline Performance for SMTP published on Microsoft Technet in 2005, Windows SMTP server has a max throughput of 142 emails/s. However this NotifyBC load test yields a max throughput of 292 emails/s. The discrepancy may be attributed to following factors
- Email size in Microsoft's load test is 50k, as opposed to 1k used in this test
- SSD storage is used in this test. It is unlikely the test conducted in 2005 used SSD.
Advices
- Avoid using default direct mode in production. Instead use SMTP server. Direct mode doesn't support connection pooling, resulting in port depletion quickly.
- Enable SMTP pooling.
- Set smtp config maxConnections to a number big enough as long as SMTP server can handle. Test found for Windows SMTP server 50 is a suitable number, beyond which performance gain is insignificant.
- Set smtp config maxMessages to maximum possible number allowed by your SMTP server, or Infinity if SMTP server imposes no such constraint
- Avoid setting CPU resource limit too low for NotifyBC app pods.
- If you have control over the SMTP server,
- use SSD for its storage
- create a load balanced cluster if possible, since SMTP server is more likely to be the bottleneck.
Bulk Import
To migrate subscriptions from other notification systems, you can use mongoimport. NotifyBC also provides a utility script to bulk import subscription data from a .csv file. To use the utility, you need
- Software installed
- Node.js
- Git
- Admin Access to a NotifyBC instance by adding your client ip to the Admin IP List
- a csv file with header row matching subscription model schema. A sample csv file is provided. Compound fields (of object type) should be dot-flattened as shown in the sample for field confirmationRequest.sendRequest
To run the utility
git clone https://github.com/bcgov/NotifyBC.git
-cd NotifyBC
-npm i -g yarn && yarn install && yarn build
-node dist/utils/bulk-import/subscription.js -a <api-url-prefix> -c <concurrency> <csv-file-path>
-Here <csv-file-path> is the path to csv file and <api-url-prefix> is the NotifyBC api url prefix , default to http://localhost:3000/api.
The script parses the csv file and generates a HTTP post request for each row. The concurrency of HTTP request is controlled by option -c which is default to 10 if omitted. A successful run should output the number of rows imported without any error message
success row count = ***
-Field Parsers
The utility script takes care of type conversion for built-in fields. If you need to import proprietary fields, by default the fields are imported as strings. To import non-string fields or manipulating json output, you need to define custom parsers in file src/utils/bulk-import/subscription.ts. For example, to parse myCustomIntegerField to integer, add in the colParser object
colParser: {
- ...
- , myCustomIntegerField: (item, head, resultRow, row, colIdx) => {
- return parseInt(item)
- }
- }
-Code of Conduct
As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality.
Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery
- Personal attacks
- Trolling or insulting/derogatory comments
- Public or private harassment
- Publishing other's private information, such as physical or electronic addresses, without explicit permission
- Other unethical or unprofessional conduct
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect of managing this project. Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team.
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community.
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting a project maintainer. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. Maintainers are obligated to maintain confidentiality with regard to the reporter of an incident.
This Code of Conduct is adapted from the Contributor Covenant, version 1.3.0, available at http://contributor-covenant.org/version/1/3/0/
Admin IP List
By design, NotifyBC classifies incoming requests into four types. For a request to be classified as super-admin, the request's source ip must be in admin ip list. By default, the list contains localhost only as defined by adminIps in /src/config.ts
module.exports = {
- adminIps: ['127.0.0.1'],
-};
-to modify, create config object adminIps with updated list in file /src/config.local.js instead. For example, to add ip range 192.168.0.0/24 to the list
module.exports = {
- adminIps: ['127.0.0.1', '192.168.0.0/24'],
-};
-It should be noted that NotifyBC may generate http requests sending to itself. These http requests are expected to be admin requests. If you have created an app cluster such as in Kubernetes, you should add the cluster ip range to adminIps. In Kubernetes, this ip range is a private ip range. For example, in BCGov's OpenShift cluster OCP4, the ip range starts with octet 10.
TLS Certificates
NotifyBC supports HTTPS TLS to achieve end-to-end encryption. In addition, both server and client can be authenticated using certificates.
To enable HTTPS for server authentication only, you need to create two files
- server/certs/key.pem - a PEM encoded private key
- server/certs/cert.pem - a PEM encoded X.509 certificate chain
Use ConfigMaps on Kubernetes
Create key.pem and cert.pem as items in ConfigMap notify-bc, then mount the items under /home/node/app/server/certs similar to how config.local.js and middleware.local.js are implemented.
For self-signed certificate, run
openssl req -x509 -newkey rsa:4096 -keyout server/certs/key.pem -out server/certs/cert.pem -nodes -days 365 -subj "/CN=NotifyBC"
-to generate both files in one shot.
Caution about self-signed cert
Self-signed cert is intended to be used in non-production environments only to authenticate server. In such environments to allow NotifyBC connecting to itself, environment variable NODE_TLS_REJECT_UNAUTHORIZED must be set to 0.
To create a CSR from the private key generated above, run
openssl req -new -key server/certs/key.pem -out server/certs/csr.pem
-Then bring your CSR to your CA to sign. Replace server/certs/cert.pem with the cert signed by CA. If your CA also supplied intermediate certificate in PEM encoded format, say in a file called intermediate.pem, append all of the content of intermediate.pem to file server/certs/cert.pem.
Make a copy of self-signed server/certs/cert.pem
If you want to enable client certificate authentication documented below, make sure to copy self-signed server/certs/cert.pem to server/certs/ca.pem before replacing the file with the cert signed by CA. You need the self-signed server/certs/cert.pem to sign client CSR.
In case you created server/certs/key.pem and server/certs/cert.pem but don't want to enable HTTPS, create following config in src/config.local.js
module.exports = {
- tls: {
- enabled: false,
- },
-};
-Update URL configs after enabling HTTPS
Make sure to update the protocol of following URL configs after enabling HTTPS
Client certificate authentication
After enabling HTTPS, you can further configure such that a client request can be authenticated using client certificate. To do so, copy self-signed server/certs/cert.pem to server/certs/ca.pem. You will use your server key to sign client certificate CSR, and advertise server/certs/ca.pem as acceptable CAs during TLS handshake.
Assuming a client's CSR file is named myClientApp_csr.pem, to sign the CSR
openssl x509 -req -in myClientApp_csr.pem -CA server/certs/ca.pem -CAkey server/certs/key.pem -out myClientApp_cert.pem -set_serial 01 -days 365
-Then give myClientApp_cert.pem to the client. How a client app supplies the client certificate when making a request to NotifyBC varies by client type. Usually the client first needs to bundle the signed client cert and client key into PKCS#12 format
openssl pkcs12 -export -clcerts -in myClientApp_cert.pem -inkey myClientApp_key.pem -out myClientApp.p12
-To use myClientApp.p12, for cURL,
curl --insecure --cert myClientApp.p12 --cert-type p12 https://localhost:3000/api/administrators/whoami
-For browsers, check browser's instruction how to import myClientApp.p12. When browser accessing NotifyBC API endpoints such as https://localhost:3000/api/administrators/whoami, the browser will prompt to choose from a list certificates that are signed by the server certificate.
In case you created server/certs/ca.pem but don't want to enable client certificate authentication, create following config in src/config.local.js
module.exports = {
- tls: {
- clientCertificateEnabled: false,
- },
-};
-TLS termination has to be passthrough
For client certification authentication to work, TLS termination of all reverse proxies has to be set to passthrough rather than offload and reload. This means, for example, when NotifyBC is hosted on OpenShift, router tls termination has to be changed from edge to passthrough.
NotifyBC internal request does not use client certificate
Requests sent by a NotifyBC node back to the app cluster use admin ip list authentication.
Cron Jobs
NotifyBC runs several cron jobs described below. These jobs are controlled by sub-properties defined in config object cron. To change config, create the object and properties in file /src/config.local.js.
By default cron jobs are enabled. In a multi-node deployment, cron jobs should only run on the master node to ensure single execution.
All cron jobs have a property named timeSpec with the value of a space separated fields conforming to unix crontab format with an optional left-most seconds field. See allowed ranges of each field.
Purge Data
This cron job purges old notifications, subscriptions and notification bounces. The default frequency of cron job and retention policy are defined by cron.purgeData config object in file /src/config.ts
module.exports = {
- cron: {
- purgeData: {
- // daily at 1am
- timeSpec: '0 0 1 * * *',
- pushNotificationRetentionDays: 30,
- expiredInAppNotificationRetentionDays: 30,
- nonConfirmedSubscriptionRetentionDays: 30,
- deletedBounceRetentionDays: 30,
- expiredAccessTokenRetentionDays: 30,
- defaultRetentionDays: 30,
- },
- },
-};
-where
- pushNotificationRetentionDays: the retention days of push notifications
- expiredInAppNotificationRetentionDays: the retention days of expired inApp notifications
- nonConfirmedSubscriptionRetentionDays: the retention days of non-confirmed subscriptions, i.e. all unconfirmed and deleted subscriptions
- deletedBounceRetentionDays: the retention days of deleted notification bounces
- expiredAccessTokenRetentionDays: the retention days of expired access tokens
- defaultRetentionDays: if any of the above retention day config item is omitted, default retention days is used as fall back.
To change a config item, set the config item in file /src/config.local.js. For example, to run cron jobs at 2am daily, add following object to /src/config.local.js
module.exports = {
- cron: {
- purgeData: {
- timeSpec: '0 0 2 * * *',
- },
- },
-};
-Dispatch Live Notifications
This cron job sends out future-dated notifications when the notification becomes current. The default config is defined by cron.dispatchLiveNotifications config object in file /src/config.ts
module.exports = {
- cron: {
- dispatchLiveNotifications: {
- // minutely
- timeSpec: '0 * * * * *',
- },
- },
-};
-Check Rss Config Updates
This cron job monitors RSS feed notification dynamic config items. If a config item is created, updated or deleted, the cron job starts, restarts, or stops the RSS-specific cron job. The default config is defined by cron.checkRssConfigUpdates config object in file /src/config.ts
module.exports = {
- cron: {
- checkRssConfigUpdates: {
- // minutely
- timeSpec: '0 * * * * *',
- },
- },
-};
-Note timeSpec doesn't control the RSS poll frequency (which is defined in dynamic configs and is service specific), instead it only controls the frequency to check for dynamic config changes.
Delete Notification Bounces
This cron job deletes notification bounces if the latest notification is deemed delivered successfully. The criteria of successful delivery are
- No bounce received since the latest notification started dispatching, and
- a configured time span has lapsed since the latest notification finished dispatching
The default config is defined by cron.deleteBounces config object in file /src/config.ts
module.exports = {
- cron: {
- deleteBounces: {
- // hourly
- timeSpec: '0 0 * * * *',
- minLapsedHoursSinceLatestNotificationEnded: 1,
- },
- },
-};
-where
- minLapsedHoursSinceLatestNotificationEnded is the time span
Re-dispatch Broadcast Push Notifications
This cron job re-dispatches a broadcast push notifications when original request failed. It is part of guaranteed broadcast push dispatch processing
The default config is defined by cron.reDispatchBroadcastPushNotifications config object in file /src/config.ts
module.exports = {
- cron: {
- reDispatchBroadcastPushNotifications: {
- // minutely
- timeSpec: '0 * * * * *',
- },
- },
-};
-Clear Redis Datastore
This cron job clears Redis datastore used for SMS and email throttle. The job is enabled only if Redis is used. Datastore is cleared only when there is no broadcast push notifications in sending state. Without this cron job, updated throttle settings in config file will never take effect, and staled jobs in Redis datastore will not be cleaned up.
The default config is defined by cron.clearRedisDatastore config object in file /src/config.ts
module.exports = {
- cron: {
- clearRedisDatastore: {
- // hourly
- timeSpec: '0 0 * * * *',
- },
- },
-};
-Database
By default NotifyBC uses in-memory database backed up by folder /server/database/ for local and docker deployment and MongoDB for Kubernetes deployment. To use MongoDB for non-Kubernetes deployment, add file /src/datasources/db.datasource.(local|<env>).(json|js|ts) with MongoDB connection information such as following:
module.exports = {
- uri: 'mongodb://127.0.0.1:27017/notifyBC?replicaSet=rs0',
- user: process.env.MONGODB_USER,
- pass: process.env.MONGODB_PASSWORD,
-};
-See Mongoose connection options for more configurable properties.
SMTP
By default NotifyBC acts as the SMTP server itself and connects directly to recipient's SMTP server. To setup SMTP relay to a host, say smtp.foo.com, add following smtp config object to /src/config.local.js
module.exports = {
- email: {
- smtp: {
- host: 'smtp.foo.com',
- port: 25,
- pool: true,
- tls: {
- rejectUnauthorized: false,
- },
- },
- },
-};
-Check out Nodemailer for other config options that you can define in smtp object. Using SMTP relay and fine-tuning some options are critical for performance. See benchmark advices.
Throttle
NotifyBC can throttle email requests if SMTP server imposes rate limit. To enable throttle and set rate limit, create following config in file /src/config.local.js
module.exports = {
- email: {
- throttle: {
- enabled: true,
- // minimum request interval in ms
- minTime: 250,
- },
- },
-};
-where
- enabled - whether to enable throttle or not. Default to false.
- minTime - minimum request interval in ms. Example value 250 throttles request rate to 4/sec.
When NotifyBC is deployed from source code, by default the rate limit applies to one Node.js process only. If there are multiple processes, i.e. a cluster, the aggregated rate limit is multiplied by the number of processes. To enforce the rate limit across entire cluster, install Redis and add Redis config to email.throttle
module.exports = {
- email: {
- throttle: {
- enabled: true,
- // minimum request interval in ms
- minTime: 250,
- /* Redis clustering options */
- datastore: 'ioredis',
- clientOptions: {
- host: '127.0.0.1',
- port: 6379,
- },
- },
- },
-};
-If you installed Redis Sentinel,
module.exports = {
- email: {
- throttle: {
- enabled: true,
- // minimum request interval in ms
- minTime: 250,
- /* Redis clustering options */
- datastore: 'ioredis',
- clientOptions: {
- name: 'mymaster',
- sentinels: [{ host: '127.0.0.1', port: 26379 }],
- },
- },
- },
-};
-Throttle is implemented using Bottleneck and ioredis. See their documentations for more configurations. The only deviation made by NotifyBC is using jobExpiration to denote Bottleneck expiration job option with a default value of 2min as defined in config.ts.
When NotifyBC is deployed to Kubernetes using Helm, by default throttle, if enabled, uses Redis Sentinel therefore rate limit applies to whole cluster.
Inbound SMTP Server
NotifyBC implemented an inbound SMTP server to handle
In order for the emails from internet to reach the SMTP server, a host where one of the following servers should be listening on port 25 open to internet
- NotifyBC, if it can be installed on such internet-facing host directly; otherwise,
- a tcp proxy server, such as nginx with stream proxy module that can proxy tcp port 25 traffic to backend NotifyBC instances.
Regardless which above option is chosen, you need to config NotifyBC inbound SMTP server by adding following static config email.inboundSmtpServer to file /src/config.local.js
module.exports = {
- email: {
- inboundSmtpServer: {
- enabled: true,
- domain: 'host.foo.com',
- listeningSmtpPort: 25,
- options: {
- // ...
- },
- },
- },
-};
-where
- enabled enables/disables the inbound SMTP server with default to true.
- domain is the internet-facing host domain. It has no default so must be set.
- listeningSmtpPort should be set to 25 if option 1 above is chosen. For options 2, listeningSmtpPort can be set to any opening port. On Unix, NotifyBC has to be run under root account to bind to port 25. If missing, NotifyBC will randomly select an available port upon launch which is usually undesirable so it should be set.
- optional options object defines the behavior of Nodemailer SMTP Server.
Inbound SMTP Server on OpenShift
OpenShift deployment template deploys an inbound SMTP server. Due to the limitation that OpenShift can only expose port 80 and 443 to external, to use the SMTP server, you have to setup a TCP proxy server (i.e. option 2). The inbound SMTP server is exposed as ${INBOUND_SMTP_DOMAIN}:443 , where ${INBOUND_SMTP_DOMAIN} is a template parameter which in absence, a default domain will be created. Configure your TCP proxy server to route traffic to ${INBOUND_SMTP_DOMAIN}:443 over TLS.
TCP Proxy Server
If NotifyBC is not able to bind to port 25 that opens to internet, perhaps due to firewall restriction, you can setup a TCP Proxy Server such as Nginx with ngx_stream_proxy_module. For example, the following nginx config will proxy SMTP traffic from port 25 to a NotifyBC inbound SMTP server running on OpenShift
stream {
- server {
- listen 25;
- proxy_pass ${INBOUND_SMTP_DOMAIN}:443;
- proxy_ssl on;
- proxy_ssl_verify off;
- proxy_ssl_server_name on;
- proxy_connect_timeout 10s;
- }
-}
-Replace ${INBOUND_SMTP_DOMAIN} with the inbound SMTP server route domain.
Bounce
Bounces, or Non-Delivery Reports (NDRs), are system-generated emails informing sender of failed delivery. NotifyBC can be configured to receive bounces, record bounces, and automatically unsubscribe all subscriptions of a recipient if the number of recorded hard bounces against the recipient exceeds threshold. A deemed successful notification delivery deletes the bounce record.
Although NotifyBC records all bounce emails, not all of them should count towards unsubscription threshold, but rather only the hard bounces - those which indicate permanent unrecoverable errors such as destination address no longer exists. In principle this can be distinguished using smtp response code. In practice, however, there are some challenges to make the distinction
- the smtp response code is not fully standardized and may vary by recipient's smtp server so it's unreliable
- there is no standard smtp header in bounce email to contain smtp response code. Often the response code is embedded in bounce email body.
- the bounce email template varies by sender's smtp server
To mitigate, NotifyBC defines several customizable string pattern filters in terms of regular expression. Only bounce emails matched the filters count towards unsubscription threshold. It's a matter of trial-and-error to get the correct filter suitable to your smtp server.
to improve hard bounce recognition
Send non-existing emails to several external email systems. Inspect the bounce messages for common string patterns. After gone live, review bounce records in web console from time to time to identify new bounce types and decide whether the bounce types qualify as hard bounce. To avoid false positives resulting in premature unsubscription, it is advisable to start with a high unsubscription threshold.
Bounce handling involves four actions
- during notification dispatching, envelop address is set to a VERP in the form bn-{subscriptionId}-{unsubscriptionCode}@{inboundSmtpServerDomain} routed to NotifyBC inbound smtp server.
- when a notification finished dispatching, the dispatching start and end time is recorded to all bounce records matching affects recipient addresses
- when inbound smtp server receives a bounce message, it updates the bounce record by saving the message and incrementing the hard bounce count when the message matches the filter criteria. The filter criteria are regular expressions matched against bounce email subject and body, as well as regular expression to extract recipient's email address from bounce email body. It also unsubscribes the user from all subscriptions when the hard bounce count exceeds a predefined threshold.
- A cron job runs periodically to delete bounce records if the latest notification is deemed delivered successfully.
To setup bounce handling
set up inbound smtp server
verify config email.bounce.enabled is set to true or absent in /src/config.local.js
verify and adjust unsubscription threshold and bounce filter criteria if needed. Following is the default config in file /src/config.ts compatible with rfc 3464
module.exports = { - email: { - bounce: { - enabled: true, - unsubThreshold: 5, - subjectRegex: '', - smtpStatusCodeRegex: '5\\.\\d{1,3}\\.\\d{1,3}', - failedRecipientRegex: - '(?:[a-z0-9!#$%&\'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&\'*+/=?^_`{|}~-]+)*|"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])', - }, - }, -}; -where
unsubThreshold is the threshold of hard bounce count above which the user is unsubscribed from all subscriptions
subjectRegex is the regular expression that bounce message subject must match in order to count towards the threshold. If subjectRegex is set to empty string or undefined, then this filter is disabled.
smtpStatusCodeRegex is the regular expression that smtp status code embedded in the message body must match in order to count towards the threshold. The default value matches all rfc3463 class 5 status codes. For a multi-part bounce message, the body limits to the one of the following parts by content type in descending order
- message/delivery-status
- html
- plain text
failedRecipientRegex is the regular expression used to extract recipient's email address from bounce message body. This extracted recipient's email address is compared against the subscription record as a means of validation. If failedRecipientRegex is set to empty string or undefined, then this validation method is skipped. The default RegEx is taken from a stackoverflow answer. For a multi-part bounce message, the body limits to the one of the following parts by content type in descending order
- message/delivery-status
- html
- plain text
Change config of cron job Delete Notification Bounces if needed
List-unsubscribe by Email
Some email clients provide a consistent UI to unsubscribe if an unsubscription email address is supplied. For example, newer iOS built-in email app will display following banner
To support this unsubscription method, NotifyBC implements a custom inbound SMTP server to transform received emails sent to address un-{subscriptionId}-{unsubscriptionCode}@{inboundSmtpServerDomain} to NotifyBC unsubscribing API calls. This unsubscription email address is generated by NotifyBC and set in header List-Unsubscribe of all notification emails.
To enable list-unsubscribe by email
- set up inbound smtp server
- verify config email.listUnsubscribeByEmail.enabled is set to true or absent in /src/config.local.js
To disable list-unsubscribe by email, set email.listUnsubscribeByEmail.enabled to false in /src/config.local.js
module.exports = {
- email: {
- listUnsubscribeByEmail: { enabled: false },
- },
-};
-HTTP Host
httpHost config sets the fallback http host used by
- mail merge token substitution
- internal HTTP requests spawned by NotifyBC
httpHost can be overridden by other configs or data. For example
- internalHttpHost config
- httpHost field in a notification
There are contexts where there is no alternatives to httpHost. Therefore this config should be defined.
Define the config, which has no default value, in /src/config.local.js
module.exports = {
- httpHost: 'http://foo.com',
-};
-Internal HTTP Host
By default, HTTP requests submitted by NotifyBC back to itself will be sent to httpHost if defined or the host of the incoming HTTP request that spawns such internal requests. But if config internalHttpHost, which has no default value, is defined, for example in file /src/config.local.js
module.exports = {
- internalHttpHost: 'http://notifybc:3000',
-};
-then the HTTP request will be sent to the configured host. An internal request can be generated, for example, as a sub-request of broadcast push notification. internalHttpHost shouldn't be accessible from internet.
All internal requests are supposed to be admin requests. The purpose of internalHttpHost is to facilitate identifying the internal server ip as admin ip.
Kubernetes Use Case
The Kubernetes deployment script sets internalHttpHost to notify-bc-app service url in config map. The source ip in such case would be in a private Kubernetes ip range. You should add this private ip range to admin ip list. The private ip range varies from Kubernetes installation. In BCGov's OCP4 cluster, it starts with octet 10.
Middleware
NotifyBC pre-installed following Express middleware as defined in /src/middleware.ts
- compression
- helmet
- morgan (disabled by default)
/src/middleware.ts contains following default middleware settings
import path from 'path';
-module.exports = {
- all: {
- compression: {},
- },
- apiOnly: {
- helmet: {},
- morgan: {
- params: [
- ':remote-addr - :remote-user [:date[clf]] ":method :url HTTP/:http-version" :status ":req[X-Forwarded-For]"',
- ],
- enabled: false,
- },
- },
-};
-/src/middleware.ts has following structure
module.exports = {
- all: {
- '<middlewareName>': {params: [], enabled: <boolean>},
- },
- apiOnly: {
- '<middlewareName>': {params: [], enabled: <boolean>},
- },
-};
-Middleware defined under all applies to both API and web console requests, as opposed to apiOnly, which applies to API requests only. params are passed to middleware function as arguments. enabled toggles the middleware on or off.
To change default settings defined in /src/middleware.ts, create file /src/middleware.local.ts or /src/middleware.<env>.ts to override. For example, to enable access log,
module.exports = {
- apiOnly: {
- morgan: {
- enabled: true,
- },
- },
-};
-Node Roles
In a multi-node deployment, some tasks should only be run by one node. That node is designated as master. The distinction is made using environment variable NOTIFYBC_NODE_ROLE. Setting to anything other than slave, including not set, will be regarded as master.
Notification
Configs in this section customize the handling of notification request or generating notifications from RSS feeds. They are all sub-properties of config object notification. Service-agnostic configs are static and service-dependent configs are dynamic.
RSS Feeds
NotifyBC can generate broadcast push notifications automatically by polling RSS feeds periodically and detect changes by comparing with an internally maintained history list. The polling frequency, RSS url, RSS item change detection criteria, and message template can be defined in dynamic configs.
Only first page is retrieved for paginated RSS feeds
If a RSS feed is paginated, NotifyBC only retrieves the first page rather than auto-fetch subsequent pages. Hence paginated RSS feeds should be sorted descendingly by last modified timestamp. Refresh interval should be adjusted small enough such that all new or updated items are contained in first page.
For example, to notify subscribers of myService on updates to feed http://my-serivce/rss, create following config item using POST configuration API
{
- "name": "notification",
- "serviceName": "myService",
- "value": {
- "rss": {
- "url": "http://my-serivce/rss",
- "timeSpec": "* * * * *",
- "itemKeyField": "guid",
- "outdatedItemRetentionGenerations": 1,
- "includeUpdatedItems": true,
- "fieldsToCheckForUpdate": ["title", "pubDate", "description"]
- },
- "httpHost": "http://localhost:3000",
- "messageTemplates": {
- "email": {
- "from": "no_reply@invlid.local",
- "subject": "{title}",
- "textBody": "{description}",
- "htmlBody": "{description}"
- },
- "sms": {
- "textBody": "{description}"
- }
- }
- }
-}
-The config items in the value field are
- rss
- url: RSS url
- timeSpec: RSS poll frequency, a space separated fields conformed to unix crontab format with an optional left-most seconds field. See allowed ranges of each field
- itemKeyField: rss item's unique key field to identify new items. By default guid
- outdatedItemRetentionGenerations: number of last consecutive polls from which results an item has to be absent before the item can be removed from the history list. This config is designed to prevent multiple notifications triggered by the same item because RSS poll returns inconsistent results, usually due to a combination of pagination and lack of sorting. By default 1, meaning the history list only keeps the last poll result
- includeUpdatedItems: whether to notify also updated items or just new items. By default false
- fieldsToCheckForUpdate: list of fields to check for updates if includeUpdatedItems is true. By default ["pubDate"]
- httpHost: the http protocol, host and port used by mail merge. If missing, the value is auto-populated based on the REST request that creates this config item.
- messageTemplates: channel-specific message templates with channel name as the key. NotifyBC generates a notification for each channel specified in the message templates. Message template fields are the same as those in notification api. Message template fields support dynamic token.
Broadcast Push Notification Task Concurrency
To achieve horizontal scaling, when a broadcast push notification request, hereby known as original request, is received, NotifyBC divides subscribers into chunks and generates a HTTP sub-request for each chunk. The original request supervises the execution of sub-requests. The chunk size is defined by config broadcastSubscriberChunkSize. All subscribers in a sub-request chunk are processed concurrently when the sub-requests are submitted.
The original request submits sub-requests back to (preferably load-balanced) NotifyBC server cluster for processing. Sub-request submission is throttled by config broadcastSubRequestBatchSize. broadcastSubRequestBatchSize defines the upper limit of the number of Sub-requests that can be processed at any given time.
As an example, assuming the total number of subscribers for a notification is 1,000,000, broadcastSubscriberChunkSize is 1,000 and broadcastSubRequestBatchSize is 10, NotifyBC will divide the 1M subscribers into 1,000 chunks and generates 1,000 sub-requests, one for each chunk. The 1,000 sub-requests will be submitted back to NotifyBC cluster to be processed. The original request will ensure at most 10 sub-requests are submitted and being processed at any given time. In fact, the only time concurrency is less than 10 is near the end of the task when remaining sub-requests is less than 10. When a sub-request is received by NotifyBC cluster, all 1,000 subscribers are processed concurrently. Suppose each sub-request (i.e. 1,000 subscribers) takes 1 minute to process on average, then the total time to dispatch notifications to 1M subscribers takes 1,000/10 = 100min, or 1hr40min.
The default value for broadcastSubscriberChunkSize and broadcastSubRequestBatchSize are defined in /src/config.ts
module.exports = {
- notification: {
- broadcastSubscriberChunkSize: 1000,
- broadcastSubRequestBatchSize: 10,
- },
-};
-To customize, create the config with updated value in file /src/config.local.js.
If total number of subscribers is less than broadcastSubscriberChunkSize, then no sub-requests are spawned. Instead, the main request dispatches all notifications.
How to determine the optimal value for broadcastSubscriberChunkSize and broadcastSubRequestBatchSize?
broadcastSubscriberChunkSize is determined by the concurrency capability of the downstream message handlers such as SMTP server or SMS service provider. broadcastSubRequestBatchSize is determined by the size of NotifyBC cluster. As a rule of thumb, set broadcastSubRequestBatchSize equal to the number of non-master nodes in NotifyBC cluster.
Broadcast Push Notification Custom Filter Functions
Advanced Topic
Defining custom function requires knowledge of JavaScript and understanding how external libraries are added and referenced in Node.js. Setting a development environment to test the custom function is also recommended.
To support rule-based notification event filtering, NotifyBC uses a modified version of jmespath to implement json query. The modified version allows defining custom functions that can be used in broadcastPushNotificationFilter field of subscription API and broadcastPushNotificationSubscriptionFilter field of subscription API. The functions must be implemented using JavaScript in config notification.broadcastCustomFilterFunctions. The functions can even be async. For example, the case-insensitive string matching function contains_ci shown in the example of that field can be created in file /src/config.local.js
const _ = require('lodash')
-module.exports = {
- ...
- notification: {
- broadcastCustomFilterFunctions: {
- contains_ci: {
- _func: async function(resolvedArgs) {
- if (!resolvedArgs[0] || !resolvedArgs[1]) {
- return false
- }
- return _.toLower(resolvedArgs[0]).indexOf(_.toLower(resolvedArgs[1])) >= 0
- },
- _signature: [
- {
- types: [2]
- },
- {
- types: [2]
- }
- ]
- }
- }
- }
-}
-Consult jmespath.js source code on the functionTable syntax and type constants used by above code. Note the function can use any Node.js modules (lodash in this case).
install additional Node.js modules
The recommended way to install additional Node.js modules is by running command npm install <your_module> from the directory one level above NotifyBC root. For example, if NotifyBC is installed on /data/notifyBC, then run the command from directory /data. The command will then install the module to /data/node_modules/<your_module>.
Guaranteed Broadcast Push Dispatch Processing
As a major enhancement in v3, by default NotifyBC guarantees all subscribers of a broadcast push notification will be processed in spite of NotifyBC node failures during dispatching. Node failure is a concern because the time takes to dispatch broadcast push notification is proportional to number of subscribers, which is potentially large.
The guarantee is achieved by
- logging the dispatch result to database individually right after each dispatch
- when subscribers are divided into chunks and a chunk sub-request fails, the original request re-submits the sub-request
- the original request periodically updates the notification updated timestamp field as heartbeat during dispatching
- if original request fails,
- a cron job detects the failure from the stale timestamp, and re-submits the original request
- all chunk sub-requests detects the the failure from the socket error, and stop processing
Guaranteed processing doesn't mean notification will be dispatched to every intended subscriber, however. Dispatch can still be rejected by smtp/sms server. Furthermore, even if dispatch is successful, it only means the sending is successful. It doesn't guarantee the recipient receives the notification. Bounce may occur for a successful dispatch, for instance; or the recipient may not read the message.
The guarantee comes at a performance penalty because result of each dispatch is written to database one by one, taking a toll on the database. It should be noted that the benchmarks were conducted without the guarantee.
If performance is a higher priority to you, disable both the guarantee and bounce handling by setting config notification.guaranteedBroadcastPushDispatchProcessing and email.bounce.enabled to false in file /src/config.local.js
module.exports = {
- notification: {
- guaranteedBroadcastPushDispatchProcessing: false,
- },
- email: {
- bounce: {enabled: false},
- },
-};
-In such case only failed dispatches are written to dispatch.failed field of the notification.
Also log skipped dispatches for broadcast push notifications
When guaranteedBroadcastPushDispatchProcessing is true, by default only successful and failed dispatches are logged, along with dispatch candidates. Dispatches that are skipped by filters defined at subscription (broadcastPushNotificationFilter) or notification (broadcastPushNotificationSubscriptionFilter) are not logged for performance reason. If you also want skipped dispatches to be logged to dispatch.skipped field of the notification, set logSkippedBroadcastPushDispatches to true in file /src/config.local.js
module.exports = {
- ...
- notification: {
- ...
- logSkippedBroadcastPushDispatches: true,
- }
-}
-Setting logSkippedBroadcastPushDispatches to true only has effect when guaranteedBroadcastPushDispatchProcessing is true.
OIDC
NotifyBC currently can only authenticate RSA signed OIDC access token if the token is a JWT. OIDC providers such as Keycloak meet the requirement.
To enable OIDC authentication strategy, add oidc configuration object to /src/config.local.js. The object supports following properties
- discoveryUrl - OIDC discovery url
- clientId - OIDC client id
- isAdmin - a predicate function to determine if authenticated user is NotifyBC administrator. The function takes the decoded OIDC access token JWT payload as input user object and should return either a boolean or a promise of boolean, i.e. the function can be both sync or async.
- isAuthorizedUser - an optional predicate function to determine if authenticated user is an authorized NotifyBC user. If omitted, any authenticated user is authorized NotifyBC user. This function has same signature as isAdmin
A example of complete OIDC configuration looks like
module.exports = {
- ...
- oidc: {
- discoveryUrl:
- 'https://op.example.com/auth/realms/foo/.well-known/openid-configuration',
- clientId: 'NotifyBC',
- isAdmin(user) {
- const roles = user.resource_access.NotifyBC.roles;
- if (!(roles instanceof Array) || roles.length === 0) return false;
- return roles.indexOf('admin') > -1;
- },
- isAuthorizedUser(user) {
- return user.realm_access.roles.indexOf('offline_access') > -1;
- },
- },
-};
-In NotifyBC web console and only in the web console, OIDC authentication takes precedence over built-in admin user, meaning if OIDC is configured, the login button goes to OIDC provider rather than the login form.
There is no default OIDC configuration in /src/config.ts.
Configuration Overview
Helm Chart Configurations
The document pages in this section cover NoitfyBC app level configurations only. If your NotifyBC is deployed to Kubernetes using Helm, you can also customize infrastructure level configurations.
There are two types of configurations - static and dynamic. Static configurations are defined in files or environment variables, requiring restarting NotifyBC to take effect, whereas dynamic configurations are defined in databases and updates take effect immediately.
Static Configurations
Most static configurations are specified in file /src/config.ts. If you need to change, instead of updating /src/config.ts file, create local file /src/config.local.js or environment specific file /src/config.<env>.js, which is only included when environment variable NODE_ENV equals <env>. Besides js, ts and json file extensions are also supported. The rest of the documentation assumes the file extension is js. Content in these files are deeply merged in following ascending precedence
- default file /src/config.ts
- environment specific file /src/config.<env>.js
- local file /src/config.local.js
Run build script whenever changing file in /src
Every time a file under /src, including config files, is updated, run yarn build before restarting NotifyBC to take effect.
Following configs should be customized per installation
In addition, if installing from source code
Customizing other configs only if needed.
Dynamic Configurations
Dynamic configs are managed using REST configuration api.
Why Dynamic Configs?
Dynamic configs are needed in cases such as
- to allow define service-specific configs such as message templates
- in a multi-node deployment, configs can be generated by one node (typically master) and shared with other nodes
Reverse Proxy IP Lists
SiteMinder, being a gateway approached SSO solution, expects the backend HTTP access point of the web sites it protests to be firewall restricted, otherwise the SiteMinder injected HTTP headers can be easily spoofed. However, the restriction cannot be easily implemented on PAAS such as OpenShift. To mitigate, two configuration objects are introduced to create an application-level firewall, both are arrays of ip addresses in the format of dot-decimal or CIDR notation
- siteMinderReverseProxyIps contains a list of ips or ranges of SiteMinder Web Agents. If set, then the SiteMinder HTTP headers are trusted only if the request is routed from the listed nodes.
- trustedReverseProxyIps contains a list of ips or ranges of trusted reverse proxies. If NotifyBC is placed behind SiteMinder Web Agents, then trusted reverse proxies should include only those between SiteMinder Web Agents and NotifyBC application. When running on OpenShift, this is usually the OpenShift router. Express.js trust proxy is set to this config object.
By default trustedReverseProxyIps is empty and siteMinderReverseProxyIps contains only localhost as defined in /src/config.ts
module.exports = {
- siteMinderReverseProxyIps: ['127.0.0.1'],
-};
-To modify, add following objects to file /src/config.local.js
module.exports = {
- siteMinderReverseProxyIps: ['130.32.12.0'],
- trustedReverseProxyIps: ['172.17.0.0/16'],
-};
-The rule to determine if the incoming request is authenticated by SiteMinder is
- obtain the real client ip address by filtering out trusted proxy ips according to Express behind proxies
- if the real client ip is contained in siteMinderReverseProxyIps, then the request is from SiteMinder, and its SiteMinder headers are trusted; otherwise, the request is considered as directly from internet, and its SiteMinder headers are ignored.
RSA Keys
When NotifyBC starts up, it checks if an RSA key pair exists in database as dynamic config. If not it will generate the dynamic config and save it to database. This RSA key pair is used to exchange confidential information with third party server applications through user's browser. For an example of use case, see Subscription API. To make it work, send the public key to the third party and have their server app encrypt the data using the public key. To obtain public key, call the REST Configuration API from an admin ip, for example, by running cURL command
curl -X GET 'http://localhost:3000/api/configurations?filter=%7B%22where%22%3A%20%7B%22name%22%3A%20%22rsa%22%7D%7D'
-or you can open API explorer, expand GET /configurations and set filter to
{"where": {"name": "rsa"}}
-The response should be something like
[
- {
- "name": "rsa",
- "value": {
- "private": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpgIBAAKCAQEA8Hl+/cF3AOxKVRHtZpeSDM+LLGc2hkDkKxRXe72maUAzDUoO\noNd6wd02Cf6iO7kj0RSDHXUyINxCgvXy2Q7gME4zRN5WG4ItWZ7FITeNgJJW1r+J\nshDjTwKVpMvcKHy0vyUl25ah7hnwGK6PbJvFWMmtIBw6Rs5DaERAlmilgkuUgdri\naA4YhhS4pCJLvO2p9wZd+dLWUT+tpsOZGeecC8If3fyShgrocMbd8pYYDzf65oCt\nVaLaNdERaIJSDcmbHxFpeBdEQEzxw2qRPbUCnSgQb8cVFLJ2eOEn5LylWhU96A1S\n3w1IlRm5N2zG0En58Vruo26gEtl5KFu0zivlawIDAQABAoIBAQCAawFsFcKtVYIk\nh9xVax/tg2/5GG0/qKuwbb6CMDcMAeLBeAjzz96YZL+U+sw8RJRh9ShHtOw+LCHA\nugMj8xO5+Cjc4DbvnccGEwmGwZnpTTzelY687tPUv7aWON+rJ12GrhnXeEulUWis\nZZvmDhGHZrvzZ9+fLEtHBRvQtrWcLCN0G5l1Z1KEWUj23vn1HZpfNvqigIbC05Pq\nWUewRZShHUklhzky6DwLklWUKv2951ypd5CHhYfXn0eXjeyqcoYeZzoCSGqtvZar\nVVOCPBKPn3cLZVKzYd02WO4CV07SpHCBtYPWf4OvXbOY6wV1Vc92S0K+ijASDDc0\nB7Vjgb8RAoGBAPg4dSbn9GWNHydveidi2Zt4kftEW18C9xHbJ3t+QkhpLjq2kwcY\no0iOWkEd4d1l0lKAVanBQazrazKiSyq1PJSJDyL3osHItA7Twq+gCXOfXw/0LbJh\napK5DH3S2ZTM42wOdZLYIHvSqRuYUmnzhy9+Ads87b/ICCctUMCLz1afAoGBAPgC\n4/zE/Au/A3wb48AywfmJ5kqPO0V7lqLrn/aBwdF1H/DHQ95cSuKrTEIysZxz52bh\n7mAHjnWnY4zFNaUvcruHw78NOxUJvje8cDIUsrTefh+qmctiGR119z7iso9FlsxR\np/o5BVT/K8q76xtkpOln2A0rc8sBNwtCoeeUzfm1AoGBAInH/O99raF49iQTswCN\n1DCCerW4uedBZBebSI06BlzfVXPtyCsWN/ycV+jxR2B3lomJBwPVbDkp7DUM9SBd\nvaTNd4N3ZfafC6N3VAfck6KEgmX+qibsABY1dYOaOIBqQorGc+jw4wcYZhoVMRny\nvcVU8n7ZkTb1N+FXPA3FDXANAoGBAOuSg0/TI71cgEjgjOJA1DLco1vq1NfY3mp9\n+QFCmwEDiYVBINwTOiY3o0W1tTLwfLoinDOmudBTYKGTqLLwcMBj4rCUNqxzBrUW\nTlOjiWN3esFFYLPoyAZNyL14wzaHWQdWAIISq1fi0IvPFzB71pDFTFimD2SiENCn\nR/YaR9OJAoGBAM21MRvTEMHF/EvqZ/X6t2zm9dtA22L2LeVy68aEdo82F/1RFvCM\nGBWjGS7G7fXk/tV/YHbjibhgktvLu3Rss1wlHfGEjtDAIdp9dqH0cNxMgy/eTfoy\nFfzV3l7pNSdILn1bNqoMz9CaYK7CGIYpBWCbRJlRSYw2FHJwl5tzgmkk\n-----END RSA PRIVATE KEY-----",
- "public": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8Hl+/cF3AOxKVRHtZpeS\nDM+LLGc2hkDkKxRXe72maUAzDUoOoNd6wd02Cf6iO7kj0RSDHXUyINxCgvXy2Q7g\nME4zRN5WG4ItWZ7FITeNgJJW1r+JshDjTwKVpMvcKHy0vyUl25ah7hnwGK6PbJvF\nWMmtIBw6Rs5DaERAlmilgkuUgdriaA4YhhS4pCJLvO2p9wZd+dLWUT+tpsOZGeec\nC8If3fyShgrocMbd8pYYDzf65oCtVaLaNdERaIJSDcmbHxFpeBdEQEzxw2qRPbUC\nnSgQb8cVFLJ2eOEn5LylWhU96A1S3w1IlRm5N2zG0En58Vruo26gEtl5KFu0zivl\nawIDAQAB\n-----END PUBLIC KEY-----"
- },
- "id": "591cda5d6c7adec42a1874bc",
- "updated": "2017-05-17T23:18:53.385Z"
- }
-]
-The public key is the string -----BEGIN PUBLIC KEY-----...-----END PUBLIC KEY-----
In a multi-node deployment, when the cluster is first started up, database is empty and rsa key pair doesn't exist. To prevent multiple rsa keys being generated by different nodes, only the master node can generate the rsa key pair. other nodes will wait for the key pair available in database before proceeding with rest bootstrap.
Expose RSA public key to only trusted party
Despite of the adjective public, NotifyBC's public key should only be distributed to trusted third party. The trusted third party should only use the public key at server backend. Using the public key in client-side JavaScript poses a security loophole.
SMS
Provider
NotifyBC depends on underlying SMS service providers to deliver SMS messages. The supported service providers are
Only one service provider can be chosen per installation. To change service provider, add following config to file /src/config.local.js
module.exports = {
- sms: {
- provider: 'swift',
- },
-};
-Provider Settings
Provider specific settings are defined in config sms.providerSettings. You should have an account with the chosen service provider before proceeding.
Twilio
Add sms.providerSettings.twilio config object to file /src/config.local.js
module.exports = {
- sms: {
- providerSettings: {
- twilio: {
- accountSid: '<AccountSid>',
- authToken: '<AuthToken>',
- fromNumber: '<FromNumber>',
- },
- },
- },
-};
-Obtain <AccountSid>, <AuthToken> and <FromNumber> from your Twilio account.
Swift
Add sms.providerSettings.swift config object to file /src/config.local.js
module.exports = {
- sms: {
- providerSettings: {
- swift: {
- accountKey: '<accountKey>',
- },
- },
- },
-};
-Obtain <accountKey> from your Swift account.
Unsubscription by replying a keyword
With Swift short code, sms user can unsubscribe by replying to a sms message with a keyword. The keyword must be pre-registered with Swift.
To enable this feature,
Generate a random string, hereafter referred to as <randomly-generated-string>
Add it to sms.providerSettings.swift.notifyBCSwiftKey in file /src/config.local.js
module.exports = { - sms: { - providerSettings: { - swift: { - notifyBCSwiftKey: '<randomly-generated-string>', - }, - }, - }, -}; -Go to Swift web admin console, click Number Management tab
Click Launch button next to Manage Short Code Keywords
Click Features button next to the registered keyword(s). A keyword may have multiple entries. In such case do this for each entry.
Click Redirect To Webpage tab in the popup window
Enter following information in the tab
- set URL to <NotifyBCHttpHost>/api/subscriptions/swift, where <NotifyBCHttpHost> is NotifyBC HTTP host name and should be the same as HTTP Host config
- set Method to POST
- set Custom Parameter 1 Name to notifyBCSwiftKey
- set Custom Parameter 1 Value to <randomly-generated-string>
Click Save Changes button and then Done
Throttle
All supported SMS service providers impose request rate limit. NotifyBC by default throttles request rate to 4/sec. To adjust the rate, create following config in file /src/config.local.js
module.exports = {
- sms: {
- throttle: {
- // minimum request interval in ms
- minTime: 250,
- },
- },
-};
-When NotifyBC is deployed from source code, by default the rate limit applies to one Node.js process only. If there are multiple processes, i.e. a cluster, the aggregated rate limit is multiplied by the number of processes. To enforce the rate limit across entire cluster, install Redis and add Redis config to sms.throttle
module.exports = {
- sms: {
- throttle: {
- /* Redis clustering options */
- datastore: 'ioredis',
- clientOptions: {
- host: '127.0.0.1',
- port: 6379,
- },
- },
- },
-};
-If you installed Redis Sentinel,
module.exports = {
- sms: {
- throttle: {
- /* Redis clustering options */
- datastore: 'ioredis',
- clientOptions: {
- name: 'mymaster',
- sentinels: [{ host: '127.0.0.1', port: 26379 }],
- },
- },
- },
-};
-Throttle is implemented using Bottleneck and ioredis. See their documentations for more configurations. The only deviation made by NotifyBC is using jobExpiration to denote Bottleneck expiration job option with a default value of 2min as defined in config.ts.
When NotifyBC is deployed to Kubernetes using Helm, by default throttle, if enabled, uses Redis Sentinel therefore rate limit applies to whole cluster.
To disable throttle, set sms.throttle.enabled to false in file /src/config.local.js
module.exports = {
- sms: {
- throttle: {
- enabled: false,
- },
- },
-};
-Subscription
Configs in this section customize behavior of subscription and unsubscription workflow. They are all sub-properties of config object subscription. This object can be defined as service-agnostic static config as well as service-specific dynamic config, which overrides the static one on a service-by-service basis. Default static config is defined in file /src/config.ts. There is no default dynamic config.
To customize static config, create the config object subscription in file /src/config.local.js
module.exports = {
- "subscription": {
- ...
- }
-}
-to create a service-specific dynamic subscription config, use REST config api
curl -X POST http://localhost:3000/api/configurations \
--H 'Content-Type: application/json' \
--H 'Accept: application/json' -d @- << EOF
-{
- "name": "subscription",
- "serviceName": "myService",
- "value": {
- ...
- }
-}
-EOF
-Sub-properties denoted by ellipsis in the above two code blocks are documented below. A service can have at most one dynamic subscription config.
Confirmation Request Message
To prevent NotifyBC from being used as spam engine, when a subscription request is sent by user (as opposed to admin) without encryption, the content of confirmation request sent to user's notification channel has to come from a pre-configured template as opposed to be specified in subscription request.
The following default subscription sub-property confirmationRequest defines confirmation request message settings for different channels
{
- "subscription": {
- ...
- "confirmationRequest": {
- "sms": {
- "confirmationCodeRegex": "\\d{5}",
- "sendRequest": true,
- "textBody": "Enter {confirmation_code} on screen"
- },
- "email": {
- "confirmationCodeRegex": "\\d{5}",
- "sendRequest": true,
- "from": "no_reply@invlid.local",
- "subject": "Subscription confirmation",
- "textBody": "Enter {confirmation_code} on screen",
- "htmlBody": "Enter {confirmation_code} on screen"
- }
- }
- }
-}
-Confirmation Verification Acknowledgement Messages
You can customize NotifyBC's on-screen response message to confirmation code verification requests. The following is the default settings
{
- "subscription": {
- ...
- "confirmationAcknowledgements": {
- "successMessage": "You have been subscribed.",
- "failureMessage": "Error happened while confirming subscription."
- }
- }
-}
-In addition to customizing the message, you can define a redirect URL instead of displaying successMessage or failureMessage. For example, to redirect on-screen acknowledgement to a page in your app for service myService, create a dynamic config by calling REST config api
curl -X POST 'http://localhost:3000/api/configurations' \
--H 'Content-Type: application/json' \
--H 'Accept: application/json' -d @- << EOF
-{
- "name": "subscription",
- "serviceName": "myService",
- "value": {
- "confirmationAcknowledgements": {
- "redirectUrl": "https://myapp/subscription/acknowledgement"
- }
- }
-}
-EOF
-If error happened during subscription confirmation, query string ?err=<error> will be appended to redirectUrl.
Duplicated Subscription
NotifyBC by default allows a user subscribe to a service through same channel multiple times. If this is undesirable, you can set config subscription.detectDuplicatedSubscription to true. In such case instead of sending user a confirmation request, NotifyBC sends user a duplicated subscription notification message. Unlike a confirmation request, duplicated subscription notification message either shouldn't contain any information to allow user confirm the subscription, or it should contain a link that allows user to replace existing confirmed subscription with this one. You can customize duplicated subscription notification message by setting config subscription.duplicatedSubscriptionNotification in either config.local.js or using configuration api for service-specific dynamic config. Following is the default settings defined in config.json
{
- ...
- "subscription": {
- ...
- "detectDuplicatedSubscription": false,
- "duplicatedSubscriptionNotification": {
- "sms": {
- "textBody": "A duplicated subscription was submitted and rejected. you will continue receiving notifications. If the request was not created by you, pls ignore this msg."
- },
- "email": {
- "from": "no_reply@invalid.local",
- "subject": "Duplicated Subscription",
- "textBody": "A duplicated subscription was submitted and rejected. you will continue receiving notifications. If the request was not created by you, please ignore this message."
- }
- }
- }
-}
-To allow user to replace existing confirmed subscription, set the message to something like
{
- ...
- "subscription": {
- ...
- "detectDuplicatedSubscription": false,
- "duplicatedSubscriptionNotification": {
- "email": {
- "textBody": "A duplicated subscription was submitted. If the request is not submitted by you, please ignore this message. Otherwise if you want to replace existing subscription with this one, click {subscription_confirmation_url}&replace=true."
- }
- }
- }
-}
-The query parameter &replace=true following the token {subscription_confirmation_url} will cause existing subscription be replaced.
Anonymous Unsubscription
For anonymous subscription, NotifyBC supports one-click opt-out by allowing unsubscription URL provided in notifications. To thwart unauthorized unsubscription attempts, NotifyBC implemented and enabled by default two security measurements
- Anonymous unsubscription request requires unsubscription code, which is a random string generated at subscription time. Unsubscription code reduces brute force attack risk by increasing size of key space. Without it, an attacker only needs to successfully guess subscription id. Be aware, however, the unsubscription code has to be embedded in unsubscription link. If the user forwarded a notification to other people, he/she is still vulnerable to unauthorized unsubscription.
- Acknowledgement notification - a (final) notification is sent to user acknowledging unsubscription, and offers a link to revert had the change been made unauthorized. A deleted subscription (unsubscription) may have a limited lifetime (30 days by default) according to retention policy defined in cron jobs so the reversion can only be performed within the lifetime.
You can customize anonymous unsubscription settings by changing the anonymousUnsubscription configuration. Following is the default settings defined in config.json
module.exports = {
- subscription: {
- anonymousUnsubscription: {
- code: {
- required: true,
- regex: '\\d{5}',
- },
- acknowledgements: {
- onScreen: {
- successMessage: 'You have been un-subscribed.',
- failureMessage: 'Error happened while un-subscribing.',
- },
- notification: {
- email: {
- from: 'no_reply@invalid.local',
- subject: 'Un-subscription acknowledgement',
- textBody:
- 'This is to acknowledge you have been un-subscribed from receiving notification for {unsubscription_service_names}. If you did not authorize this change or if you changed your mind, open {unsubscription_reversion_url} to revert.',
- htmlBody:
- 'This is to acknowledge you have been un-subscribed from receiving notification for {unsubscription_service_names}. If you did not authorize this change or if you changed your mind, click <a href="{unsubscription_reversion_url}">here</a> to revert.',
- },
- },
- },
- },
- },
-};
-The settings control whether or not unsubscription code is required, its RegEx pattern, and acknowledgement message templates for both on-screen and push notifications. Customization should be made to file /src/config.local.js for static config or using configuration api for service-specific dynamic config.
To disable acknowledgement notification, set subscription.anonymousUnsubscription.acknowledgements.notification or a specific channel underneath to null
module.exports = {
- subscription: {
- anonymousUnsubscription: {
- acknowledgements: {
- notification: null,
- },
- },
- },
-};
-For on-screen acknowledgement, you can define a redirect URL instead of displaying successMessage or failureMessage. For example, to redirect on-screen acknowledgement to a page in your app for all services, create following config in file /src/config.local.js
module.exports = {
- subscription: {
- anonymousUnsubscription: {
- acknowledgements: {
- onScreen: {
- redirectUrl: 'https://myapp/unsubscription/acknowledgement',
- },
- },
- },
- },
-};
-If error happened during unsubscription, query string ?err=<error> will be appended to redirectUrl.
You can customize message displayed on-screen when user clicks revert unsubscription link in the acknowledgement notification. The default settings are
{
- "subscription": {
- "anonymousUndoUnsubscription": {
- "successMessage": "You have been re-subscribed.",
- "failureMessage": "Error happened while re-subscribing."
- }
- }
-}
-You can redirect the message page by defining anonymousUndoUnsubscription.redirectUrl.
Worker Process Count
When NotifyBC runs on a host with multiple CPUs, by default it creates a cluster of worker processes of which the count matches CPU count. You can override the number with the environment variable NOTIFYBC_WORKER_PROCESS_COUNT.
A note about worker process count on OpenShift
It has been observed that on OpenShift Node.js returns incorrect CPU count. The template therefore sets NOTIFYBC_WORKER_PROCESS_COUNT to 1. After all, on OpenShift NotifyBC is expected to be horizontally scaled by pods rather by CPUs.
Developer Notes
Setup development environment
Install Visual Studio Code and following extensions:
- Prettier
- ESLint
- Vetur
- Code Spell Checker
- Debugger for Chrome
Multiple run configs have been created to facilitate debugging server, client, test and docs.
Client certificate authentication doesn't work in client debugger
Because Vue cli webpack dev server cannot proxy passthrough HTTPS connections, client certificate authentication doesn't work in client debugger. If testing client certificate authentication in web console is needed, run yarn build to generate prod client distribution and launch server debugger on https://localhost:3000
Automated Testing
NotifyBC uses Jest test framework bundled in NestJS. To launch test, run yarn test:e2e. A Test launch config is provided to debug in VS Code.
Github Actions runs tests as part of the build. All test scripts should be able to run unattended, headless, quickly and depend only on local resources.
Writing Test Specs
Thanks to supertest and MongoDB In-Memory Server, test specs can be written to cover nearly end-to-end request processing workflow (only sendMail and sendSMS need to be mocked). This allows test specs to anchor onto business requirements rather than program units such as functions or files, resulting in regression tests that are more resilient to code refactoring. Whenever possible, a test spec should be written to
- start at a processing phase as early as possible. For example, to test a REST end point, start with the HTTP user request.
- assert outcome of a processing phase as late and down below as possible - the HTTP response body/code, the database record created, for example.
- avoid asserting middleware function input/output to facilitate code refactoring.
- mock email/sms sending function (implemented by default). Inspect the input of the function, or at least assert the function has been called.
Install Docs Website
If you want to contribute to NotifyBC docs beyond simple fix ups, run
yarn --cwd docs install
-yarn --cwd docs dev
-If everything goes well, the last line of the output will be
> VuePress dev server listening at http://localhost:8080/NotifyBC/
-You can now browse to the local docs site http://localhost:8080/NotifyBC
Publish Version Checklist
Before adding a release,
- update version in package.json
- update version appVersion in helm/Chart.yaml (major/minor only)
- update What's new (major/minor only)
Welcome
This site aims to be a comprehensive guide to NotifyBC. We’ll cover topics such as getting your instance up and running, interacting with browser or other server components, deployment, and give you some advice on participating in the future development of NotifyBC itself.
Helpful Hints
Throughout this guide there are a number of small-but-handy pieces of information that can make using NotifyBC easier, more interesting, and less hazardous. Here’s what to look out for.
General information
These are tips and tricks that will help you become a NotifyBC wizard!
Important information
These are tidbits you might want to keep in mind.
Warnings
Be aware of these messages if you wish to avoid disaster.
If you come across anything along the way that we haven’t covered, or if you know of a tip you think others would find handy, please file an issue and we’ll see about including it in this guide.
Installation
NotifyBC can be installed in 3 ways:
For the purpose of evaluation, both source code and docker container will do. For production, the recommendation is one of
- deploying to Kubernetes
- setting up a load balanced app cluster from source code build, backed by MongoDB.
To setup a development environment in order to contribute to NotifyBC, installing from source code is preferred.
Deploy locally from Source Code
System Requirements
- Software
- Git
- Node.js@>=18
- openssl (if enable HTTPS)
- Services
- MongoDB with replica set, required for production
- A standard SMTP server to deliver outgoing email, required for production if email is enabled.
- A tcp proxy server such as nginx stream proxy if list-unsubscribe by email is needed and NotifyBC server cannot expose port 25 to internet
- A SMS service provider if needs to enable SMS channel. The supported service providers are
- Twilio (default)
- Swift
- Redis v6, required if email or sms throttling is enabled
- SiteMinder, if needs SiteMinder authentication
- An OIDC provider, if needs OIDC authentication
- Network and Permissions
- Minimum runtime firewall requirements:
- outbound to your ISP DNS server
- outbound to any on port 80 and 443 in order to run build scripts and send SMS messages
- outbound to any on SMTP port 25 if using direct mail; for SMTP relay, outbound to your configured SMTP server and port only
- inbound to listening port (3000 by default) from other authorized server ips
- if NotifyBC instance will handle anonymous subscription from client browser, the listening port should be open to internet either directly or indirectly through a reverse proxy; If NotifyBC instance will only handle SiteMinder authenticated webapp requests, the listening port should NOT be open to internet. Instead, it should only open to SiteMinder web agent reverse proxy.
- If list-unsubscribe by email is needed, then one of the following must be met
- NotifyBC can bind to port 25 opening to internet
- a tcp proxy server of which port 25 is open to internet. This proxy server can reach NotifyBC on a tcp port.
- Minimum runtime firewall requirements:
Installation
Run following commands
git clone https://github.com/bcgov/NotifyBC.git
-cd NotifyBC
-npm i -g yarn && yarn install && yarn build
-yarn start
-If successful, you will see following output
...
-Server is running at http://0.0.0.0:3000
-Now open http://localhost:3000. The page displays NotifyBC Web Console.
The above commands installs the main version, i.e. main branch tip of NotifyBC GitHub repository. To install a specific version, say v2.1.0, run
git checkout tags/v2.1.0 -b v2.1.0
-after cd NotifyBC. A list of versions can be found here.
install from behind firewall
If you want to install on a server behind firewall which restricts internet connection, you can work around the firewall as long as you have access to a http(s) forward proxy server. Assuming the proxy server is http://my_proxy:8080 which proxies both http and https requests, to use it:
For Linux
export http_proxy=http://my_proxy:8080 -export https_proxy=http://my_proxy:8080 -git config --global url."https://".insteadOf git:// -For Windows
git config --global http.proxy http://my_proxy:8080 -git config --global url."https://".insteadOf git:// -npm config set proxy http://my_proxy:8080 -npm i -g yarn -yarn config set proxy http://my_proxy:8080 -
Install Windows Service
After get the app running interactively, if your server is Windows and you want to install the app as a Windows service, run
npm install -g node-windows
-npm link node-windows
-node windows-service.js
-This will create and start service notifyBC. To change service name, modify file windows-service.js before running it. See node-windows for other operations such as uninstalling the service.
Deploy to Kubernetes
NotifyBC provides a container package in GitHub Container Registry and a Helm chart to facilitate Deploying to Kubernetes. Azure AKS and OpenShift are the two tested platforms. Other Kubernetes platforms are likely to work subject to customizations. Before deploying to AKS, create an ingress controller .
The deployment can be initiated from localhost or automated by CI service such as Jenkins. Regardless, at the initiator's side following software needs to be installed:
- git
- Platform-specific CLI such as Azure CLI or OpenShift CLI
- Helm CLI
To install,
Follow your platform's instruction to login to the platform. For AKS, run
az loginandaz aks get-credentials; for OpenShift, runoc loginRun
git clone https://github.com/bcgov/NotifyBC.git -cd NotifyBC -helm install -gf helm/platform-specific/<platform>.yaml helm -replace <platform> with openshift or aks depending on your platform.
The above commands create following artifacts:
- 1 stateful set of 3 pods running a MongoDB replicaset
- 1 stateful set of 3 pods running a Redis sentinel
- 2 deployments - notify-bc-app and notify-bc-cron
- 1 HPA - notify-bc-cron
- 5 services - notify-bc, notify-bc-smtp, mongodb-headless, redis and redis-headless
- 3 PVCs each for one MongoDB pod
- 3 service accounts - notify-bc, mongodb and redis
- a few config maps, most importantly notify-bc
- a few secrets, most importantly mongodb and redis, containing credentials for Mongodb and Redis respectively
- On AKS,
- a notify-bc ingress
- On OpenShift,
- 2 routes - notify-bc-web and notify-bc-smtp
To upgrade,
helm upgrade <release-name> -f helm/platform-specific/<platform>.yaml --set mongodb.auth.rootPassword=<mongodb-root-password> --set mongodb.auth.replicaSetKey=<mongodb-replica-set-key> --set mongodb.auth.password=<mongodb-password> helm
-replace <release-name> with installed helm release name and <platform> with openshift or aks depending on your platform. MongoDB credentials <mongodb-root-password>, <mongodb-replica-set-key> and <mongodb-password> can be found in secret <release-name>-mongodb. It is recommended to specify mongodb credentials in a file rather than command line. See Customizations below.
To uninstall,
helm uninstall <release-name>
-replace <release-name> with installed helm release name.
Customizations
Various customizations can be made to chart. Some are platform dependent. To customize, first create a file with extension .local.yaml. The rest of the document assumes the file is helm/values.local.yaml. Then add customized parameters to the file. See helm/values.yaml and Bitnami MongoDB chart readme for customizable parameters. Parameters in helm/values.local.yaml overrides corresponding ones in helm/values.yaml. In particular, parameters under mongodb of helm/values.local.yaml overrides Bitnami MongoDB chart parameters.
To apply customizations, add -f helm/values.local.yaml to the helm command after -f helm/platform-specific/<platform>.yaml. For example, to install chart with customization on OpenShift,
helm install -gf helm/platform-specific/openshift.yaml -f helm/values.local.yaml helm
-to upgrade an existing release with customization on OpenShift,
helm upgrade <release-name> -f helm/platform-specific/openshift.yaml -f helm/values.local.yaml helm
-Backup helm/values.local.yaml
Backup helm/values.local.yaml to a private secured SCM is highly recommended, especially for production environment.
Following are some common customizations
Update config.local.js in ConfigMap, for example to define httpHost
# in file helm/values.local.yaml -configMap: - config.local.js: |- - module.exports = { - httpHost: 'https://myNotifyBC.myOrg.com', - } -Set hostname on AKS,
# in file helm/values.local.yaml -ingress: - hosts: - - host: myNotifyBC.myOrg.com - paths: - - path: / -Use Let's Encrypt on AKS. After following the instructions in the link, add following ingress customizations to file helm/values.local.yaml
# in file helm/values.local.yaml -ingress: - annotations: - cert-manager.io/cluster-issuer: letsencrypt - tls: - - secretName: tls-secret - hosts: - - notify-bc.local -Route host names on Openshift are by default auto-generated. To set to fixed values
# in file helm/values.local.yaml -route: - web: - host: 'myNotifyBC.myOrg.com' - smtp: - host: 'smtp.myNotifyBC.myOrg.com' -Add certificates to OpenShift web route
# in file helm/values.local.yaml -route: - web: - tls: - caCertificate: |- - -----BEGIN CERTIFICATE----- - ... - -----END CERTIFICATE----- - certificate: |- - -----BEGIN CERTIFICATE----- - ... - -----END CERTIFICATE----- - insecureEdgeTerminationPolicy: Redirect - key: |- - -----BEGIN PRIVATE KEY----- - ... - -----END PRIVATE KEY----- -MongoDb
NotifyBC chart depends on Bitnami MongoDB chart for MongoDB database provisioning. All documented parameters are customizable under mongodb. For example, to change architecture to standalone
# in file helm/values.local.yaml -mongodb: - architecture: standalone -To set credentials,
# in file helm/values.local.yaml -mongodb: - auth: - rootPassword: <secret> - replicaSetKey: <secret> - passwords: - - <secret> -To install a Helm chart, the above credentials can be randomly defined. To upgrade an existing release, they must match what's defined in secret <release-name>-mongodb.
Redis
NotifyBC chart depends on Bitnami Redis chart for Redis provisioning. All documented parameters are customizable under redis. For example, to set credential
# in file helm/values.local.yaml -redis: - auth: - password: <secret> -To install a Helm chart, the above credential can be randomly defined. To upgrade an existing release, It must match what's defined in secret <release-name>-redis.
Both Bitnami MongoDB and Redis use Docker Hub for docker registry. Rate limit imposed by Docker Hub can cause runtime problems. If your organization has JFrog artifactory, you can change the registry
# in file helm/values.local.yaml
-global:
- imageRegistry: <artifactory.myOrg.com>
- imagePullSecrets:
- - <docker-pull-secret>
-The above settings assume you have setup secret <docker-pull-secret> to access <artifactory.myOrg.com>. The secret can be created using kubectl.
Enable scheduled MongoDB backup CronJob
# in file helm/values.local.yaml -cronJob: - enabled: true - schedule: '1 0 * * *' - retentionDays: 7 - timeZone: UTC - persistence: - size: 5Gi -where
- enabled: whether to enable the MongoDB backup CronJob or not; default to
false - schedule: the Unix crontab schedule; default to
'1 0 * * *'which runs daily at 12:01AM - retentionDays: how many days the backup is retained; default to
7 - timeZone: the Unix TZ environment variable; default to
UTC - persistence size: size of PVC; default to
5Gi
The CronJob backs up MongoDB to a PVC named after the chart with suffix -cronjob-mongodb-backup and purges backups that are older than retentionDays.
To facilitate restoration, mount the PVC to MongoDB pod
# in file helm/values.local.yaml -mongodb: - extraVolumes: - - name: export - persistentVolumeClaim: - claimName: <PVC_NAME> - extraVolumeMounts: - - name: export - mountPath: /export - readOnly: true -Restoration can then be achieved by running in MongoDB pod
mongorestore -u "$MONGODB_EXTRA_USERNAMES" -p"$MONGODB_EXTRA_PASSWORDS" \ ---uri="mongodb://$K8S_SERVICE_NAME" --db $MONGODB_EXTRA_DATABASES --gzip --drop \ ---archive=/export/<mongodb-backup-YYMMDD-hhmmss.gz> -- enabled: whether to enable the MongoDB backup CronJob or not; default to
NotifyBC image tag defaults to appVersion in file helm/Chart.yaml. To change to latest, i.e. tip of the main branch,
# in file helm/values.local.yaml -image: - tag: latest -Enable autoscaling for app pod
# in file helm/values.local.yaml -autoscaling: - enabled: true -
Deploy Docker Container
If you have git and Docker installed, you can run following command to deploy NotifyBC Docker container:
docker run --platform linux/amd64 --rm -dp 3000:3000 ghcr.io/bcgov/notify-bc
-# open http://localhost:3000
-If successful, similar output is displayed as in source code installation.
Overview
NotifyBC is a general purpose API Server to manage subscriptions and dispatch notifications. It aims to implement some common backend processes of a notification service without imposing any constraints on the UI frontend, nor impeding other server components' functionality. This is achieved by interacting with user browser and other server components through RESTful API and other standard protocols in a loosely coupled way.
Features
NotifyBC facilitates both anonymous and authentication-enabled secure webapps implementing notification feature. A NotifyBC server instance supports multiple notification services. A service is a topic of interest that user wants to receive updates. It is used as the partition of notification messages and user subscriptions. A user may subscribe to a service in multiple push delivery channels allowed. A user may subscribe to multiple services. In-app pull notification doesn't require subscription as it's not intrusive to user.
notification
- both in-app pull notifications (a.k.a. messages or alerts) and push notifications
- multiple push notifications delivery channels
- sms
- unicast and broadcast message types
- future-dated notifications
- for in-app pull notifications
- support read and deleted message states
- message expiration
- deleted messages are not deleted immediately for auditing and recovery purposes
- for broadcast push notifications
- allow both sync and async POST API calls. For async API call, an optional callback url is supported
- can be auto-generated from RSS feeds
- allow user to specify filter rules evaluated against data contained in the notification
- allow sender to specify filter rules evaluated against data contained in the subscription
- allow application developer to create custom filter functions used by the filter rules mentioned above
subscription and un-subscription
- verify the ownership of push notification subscription channel:
- generates confirmation code based on a regex input
- send confirmation request to unconfirmed subscription channel
- verify confirmation code
- generate random un-subscription code
- send acknowledgement message after un-subscription for anonymous subscribers
- bulk unsubscription
- list-unsubscribe by email
- track bounces and unsubscribe the recipient from all subscriptions when hard bounce count exceeds threshold
- sms user can unsubscribe by replying a shortcode keyword with Swift sms provider
mail merge
Strings in notification or subscription message that are enclosed between curly braces { } are called tokens, also known as placeholders. Tokens are replaced based on the context of notification or subscription when dispatching the message. To avoid treating a string between curly braces as a token, escape the curly braces with backslash \. For example \{i_am_not_a_token\} is not a token. It will be rendered as {i_am_not_a_token}.
Tokens whose names are predetermined by NotifyBC are called static tokens; otherwise they are called dynamic tokens.
static tokens
NotifyBC recognizes following case-insensitive static tokens. Most of the names are self-explanatory.
- {subscription_confirmation_url}
- {subscription_confirmation_code}
- {service_name}
- {http_host} - http host in the form http(s): //<host_name>:<port>. The value is obtained from the http request that triggers the message
- {rest_api_root} - REST API URL path prefix
- {subscription_id}
- anonymous unsubscription related tokens
- {unsubscription_url}
- {unsubscription_all_url} - url to unsubscribe all services the user has subscribed on this NotifyBC instance
- {unsubscription_code}
- {unsubscription_reversion_url}
- {unsubscription_service_names} - includes {service_name} and additional services user has unsubscribed, prefixed with conditionally pluralized word service.
dynamic tokens
Dynamic tokens are replaced with correspondingly named sub-field of data field in the notification or subscription if exist. Qualify token name with notification:: or subscription:: to indicate the source of substitution. If token name is not qualified, then both notification and subscription are checked, with notification taking precedence. Nested and indexed sub-fields are supported.
Examples
- {notification::description} is replaced with field data.description of the notification if exist
- {subscription::gender} is replaced with field data.gender of the subscription if exist
- {addresses[0].city} is replaced with field data.addresses[0].city of the notification if exist; otherwise is replaced with field data.addresses[0].city of the subscription if exist
- {nonexistingDataField} is unreplaced if neither notification nor subscription contains data.nonexistingDataField
As exception, in order to prevent spamming by unconfirmed subscribers, dynamic tokens in subscription confirmation request message and duplicated subscription message are not replaced with subscription data, for example {subscription::...} tokens are left unchanged.
Notification by RSS feeds relies on dynamic token
A notification created by RSS feeds relies on dynamic token to supply the context to message template. In this case the data field contains the RSS item.
Architecture
Request Types
NotifyBC, designed to be a microservice, doesn't use full-blown ACL to secure API calls. Instead, it classifies incoming requests into admin and user types. The key difference is while both admin and user can subscribe to notifications, only admin can post notifications.
Each type has two subtypes based on following criteria
super-admin, if the request meets both of the following two requirements
The request carries one of the following two attributes
- the source ip is in the admin ip list
- has a client certificate that is signed using NotifyBC server certificate. See Client certificate authentication on how to sign.
The request doesn't contain any of following case insensitive HTTP headers, with the first three being SiteMinder headers
- sm_universalid
- sm_user
- smgov_userdisplayname
- is_anonymous
admin, if the request is not super-admin and meets one of the following criteria
- has a valid access token associated with an builtin admin user created and logged in using the administrator api, and the request doesn't contain any HTTP headers listed above
- has a valid OIDC access token containing customizable admin profile attributes
access token disambiguation
Here the term access token has been used to refer two different things
- the token associated with a builtin admin user
- the token generated by OIDC provider.
To reduce confusion, throughout the documentation the former is called access token and the latter is called OIDC access token.
authenticated user, if the request is neither super-admin nor admin, and meets one fo the following criteria
- contains any of the 3 SiteMinder headers listed above, and comes from either trusted SiteMinder proxy or admin ip list
- contains a valid OIDC access token
anonymous user, if the request doesn't meet any of the above criteria
The only extra privileges that a super-admin has over admin are that super-admin can perform CRUD operations on configuration, bounce and administrator entities through REST API. In the remaining docs, when no further distinction is necessary, an admin request refers to both super-admin and admin request; a user request refers to both authenticated and anonymous request.
An admin request carries full authorization whereas user request has limited access. For example, a user request is not allowed to
- send notification
- bypass the delivery channel confirmation process when subscribing to a service
- retrieve push notifications through API (can only receive notification from push notification channel such as email)
- retrieve in-app notifications that is not targeted to the current user
The result of an API call to the same end point may differ depending on the request type. For example, the call GET /notifications without a filter will return all notifications to all users for an admin request, but only non-deleted, non-expired in-app notifications for authenticated user request, and forbidden for anonymous user request. Sometimes it is desirable for a request from admin ip list, which would normally be admin request, to be voluntarily downgraded to user request in order to take advantage of predefined filters such as the ones described above. This can be achieved by adding one of the HTTP headers listed above to the request. This is also why admin request is not determined by ip or token alone.
The way NotifyBC interacts with other components is diagrammed below. 
Authentication Strategies
API requests to NotifyBC can be either anonymous or authenticated. As alluded in Request Types above, NotifyBC supports following authentication strategies
- ip whitelisting
- client certificate
- access token associated with an builtin admin user
- OpenID Connect (OIDC)
- CA SiteMinder
Authentication is performed in above order. Once a request passed an authentication strategy, the rest strategies are skipped. A request that failed all authentication strategies is anonymous.
The mapping between authentication strategy and request type is
| Admin | User | |||
| Super-admin | admin | authenticated | anonymous | |
| ip whitelisting | ✔ | |||
| client certifcate | ✔ | |||
| access token | ✔ | |||
| OIDC | ✔ | ✔ | ||
| SiteMinder | ✔ | |||
Which authentication strategy to use?
Because ip whitelist doesn't expire and client certificate usually has a relatively long expiration period (say one year), they are suitable for long-running unattended server processes such as server-side code of web apps, cron jobs, IOT sensors etc. The server processes have to be trusted because once authenticated, they have full privilege to NotifyBC. Usually the server processes and NotifyBC instance are in the same administrative domain, i.e. managed by the same admin group of an organization.
By contrast, OIDC and SiteMinder use short-lived tokens or session cookies. Therefore they are only suitable for interactive user sessions.
Access token associated with an builtin admin user should be avoided whenever possible.
Here are some common scenarios and recommendations
For server-side code of web apps
- use OIDC if the web app is OIDC enabled and user requests can be proxied to NotifyBC by web app; otherwise
- use ip whitelisting if obtaining ip is feasible; otherwise
- use client certificate (requires a little more config than ip whitelisting)
For front-end browser-based web apps such as SPAs
- use OIDC
For server apps that send requests spontaneously such as IOT sensors, cron jobs
- use ip whitelisting if obtaining ip is feasible; otherwise
- client certificate
If NotifyBC is ued by a SiteMinder protected web apps and NotifyBC is also protected by SiteMinder
- use SiteMinder
Application Framework
NotifyBC is created on NestJS. Contributors to source code of NotifyBC should be familiar with NestJS. NestJS Docs serves a good complement to this documentation.
Quick Start
For the impatient, here's how to get a boilerplate NotifyBC instance up and running if you have git and node.js installed:
git clone https://github.com/bcgov/NotifyBC.git
-cd NotifyBC
-npm i -g yarn && yarn install && yarn build
-yarn start
-# => Now browse to http://localhost:3000
-If you're running into problems, check out full installation guide.
a filter containing properties where, fields, order, skip, and limit
- parameter name: filter
-- required: false
-- parameter type: query
-- data type: object
-
-The filter can be expressed as either
-
- 1. URL-encoded stringified JSON object (see example below); or
- 2. in the format supported by [qs](https://github.com/ljharb/qs), for example `?filter[where][created][$gte]="2023-01-01"&filter[where][created][$lt]="2024-01-01"`
-
-Regardless, the filter will have to be parsed into a JSON object conforming to
-
-```json
-{
- "where": {...},
- "fields": ...,
- "order": ...,
- "skip": ...,
- "limit": ...,
-}
-```
-
-All properties are optional. The syntax for each property is documented, respectively
-- for *where* , see MongoDB [Query Documents](https://www.mongodb.com/docs/manual/tutorial/query-documents/)
-- for *fields* , see Mongoose [select](https://mongoosejs.com/docs/api/query.html#Query.prototype.select())
-- for *order*, see Mongoose [sort](https://mongoosejs.com/docs/api/query.html#Query.prototype.sort())
-- for *skip*, see MongoDB [cursor.skip](https://www.mongodb.com/docs/manual/reference/method/cursor.skip/)
-- for *limit*, see MongoDB [cursor.limit](https://www.mongodb.com/docs/manual/reference/method/cursor.limit/)
-?filter=%7B%22where%22%3A%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D%7D
the value of the filter query parameter is URL-encoded stringified JSON object
```json
-{
- "where": {
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
- }
-}
-```
- <div class="description">a string conforming to jmespath <a href="http://jmespath.org/specification.html#filter-expressions">filter expressions syntax</a> after the question mark (?). The filter is matched against the <i><a href="../api-subscription#data">data</a></i> field of the subscription. Examples of filter
- <ul>
- <li>simple <br/>
- <i>province == 'BC'</i>
- </li>
- <li>calling jmespath's <a href="http://jmespath.org/specification.html#built-in-functions">built-in functions</a> <br/>
- <i>contains(province,'B')</i>
- </li>
- <li>calling <a href="../config-notification/#broadcast-push-notification-custom-filter-functions">custom filter functions</a><br/>
- <i>contains_ci(province,'b')</i>
- </li>
- <li>compound <br/>
- <i>(contains(province,'BC') || contains_ci(province,'b')) && city == 'Victoria' </i>
- </li>
- </ul>
- All of above filters will match data object <i>{"province": "BC", "city": "Victoria"}</i>
- </div>
-Throttle is implemented using Bottleneck and ioredis. See their documentations for more configurations. The only deviation made by NotifyBC is using jobExpiration to denote Bottleneck expiration job option with a default value of 2min as defined in config.ts.
When NotifyBC is deployed to Kubernetes using Helm, by default throttle, if enabled, uses Redis Sentinel therefore rate limit applies to whole cluster.
a where query parameter with value conforming to MongoDB Query Documents
- parameter name: where
-- required: false
-- parameter type: query
-- data type: object
-
-The value can be expressed as either
-
- 1. URL-encoded stringified JSON object (see example below); or
- 2. in the format supported by [qs](https://github.com/ljharb/qs), for example `?where[created][$gte]="2023-01-01"&where[created][$lt]="2024-01-01"`
-?where=%7B%22created%22%3A%7B%22%24gte%22%3A%222023-01-01%22%2C%22%24lt%22%3A%222024-01-01%22%7D%7D
the value of the where query parameter is URL-encoded stringified JSON object
```json
-{
- "created": {
- "$gte": "2023-01-01",
- "$lt": "2024-01-01"
- }
-}
-```
-Upgrade Guide
Major version can only be upgraded incrementally from immediate previous major version, i.e. from N to N+1.
v1 to v2
Upgrading NotifyBC from v1 to v2 involves two steps
Update your client code
NotifyBC v2 introduced backward incompatible API changes documented in the rest of this section. If your client code will be impacted by the changes, update your code to address the incompatibility first.
Query parameter array syntax
In v1 array can be specified in query parameter using two formats
- by enclosing array elements in square brackets such as
&additionalServices=["s1","s2]in one query parameter - by repeating the query parameters, for example
&additionalServices=s1&additionalServices=s2
In v2 only the latter format is supported.
Date-Time fields
In v1 date-time fields can be specified in date-only string such as 2020-01-01. In v2 the field must be specified in ISO 8601 extended format such as 2020-01-01T00:00:00Z.
Return status codes
HTTP response code of success calls to following APIs are changed from 200 to 204
- most PATCH by id requests except for Update a Subscription
- most PUT by id requests except for Replace a Subscription
- most DELETE by id requests except for Delete a Subscription (unsubscribing)
Administrator API
- Password is saved to Administrator in v1 and UserCredential in v2. Password is not migrated. New password has to be created by following Create/Update an Administrator's UserCredential .
- Complexity rules have been applied to passwords.
- login API is open to non-admin
Upgrade NotifyBC server
The procedure to upgrade from v1 to v2 depends on how v1 was installed.
Source-code Installation
- Stop NotifyBC
- Backup app root and database!
- Make sure current branch is tracking correct remote branch
git remote set-url origin https://github.com/bcgov/NotifyBC.git -git branch -u origin/main - - Make a note of any extra packages added to package.json
- Run
git pull && git checkout tags/v2.x.x -b <branch_name>from app root, replace v2.x.x with a v2 release, preferably latest, found in GitHub such as v2.9.0. - Make sure version property in package.json is 2.x.x
- Add back extra packages noted in step 4
- Move server/config.(local|dev|production).(js|json) to src/ if exists
- Move server/datasources.(local|dev|production).(js|json) to src/datasources/db.datasource.(local|dev|production).(js|json) if exists. Notice the file name has changed.
- Move server/middleware.*.(js|json) to src/ if exists. Reorganize top level properties to all or apiOnly, where all applies to all requests including web console and apiOnly applies to API requests only. For example, given
module.exports = {
- initial: {
- compression: {},
- },
- 'routes:before': {
- morgan: {
- enabled: false,
- },
- },
-};
-if compression middleware will be applied to all requests and morgan will be applied to API requests only, then change the file to
module.exports = {
- all: {
- compression: {},
- },
- apiOnly: {
- morgan: {
- enabled: false,
- },
- },
-};
-- Run
yarn install && yarn build
-- Start server by running
yarn startor Windows Service
OpenShift Installation
Run
git clone https://github.com/bcgov/NotifyBC.git -cd NotifyBC -Run
oc delete bc/notify-bc -oc process -f .openshift-templates/notify-bc-build.yml | oc create -f- -ignore AlreadyExists errors
Follow OpenShift Build
For each environment,
run
oc project <yourprojectname-<env>> -oc delete dc/notify-bc-app dc/notify-bc-cron -oc process -f .openshift-templates/notify-bc.yml | oc create -f- -ignore AlreadyExists errors
copy value of environment variable MONGODB_USER from mongodb deployment config to the same environment variable of deployment config notify-bc-app and notify-bc-cron, replacing existing value
remove middleware.local.json from configMap notify-bc
add middleware.local.js to configMap notify-bc with following content
module.exports = { - apiOnly: { - morgan: { - enabled: false, - }, - }, -}; -Follow OpenShift Deploy or Change Propagation to tag image
OpenShift template to Helm
Upgrading NotifyBC on OpenShift created from OpenShift template to Helm involves 2 steps
Customize and install Helm chart
Follow customizations to create file helm/values.local.yaml containing customized configs such as
- notify-bc configMap
- web route host name and certificates
Then run helm install with documented arguments to install a release.
Migrate MongoDB data
backup data from source
oc exec -i <mongodb-pod> -- bash -c 'mongodump -u "$MONGODB_USER" \ --p "$MONGODB_PASSWORD" -d $MONGODB_DATABASE --gzip --archive' > notify-bc.gz -replace <mongodb-pod> with the mongodb pod name.
restore backup to target
cat notify-bc.gz | oc exec -i <mongodb-pod-0> -- \ -bash -c 'mongorestore -u "$MONGODB_USERNAME" -p"$MONGODB_PASSWORD" \ ---uri="mongodb://$K8S_SERVICE_NAME" --db $MONGODB_DATABASE --gzip --drop --archive' -replace <mongodb-pod-0> with the first pod name in the mongodb stateful set.
If both source and target are in the same OpenShift cluster, the two operations can be combined into one
oc exec -i <mongodb-pod> -- bash -c 'mongodump -u "$MONGODB_USER" \
--p "$MONGODB_PASSWORD" -d $MONGODB_DATABASE --gzip --archive' | \
-oc exec -i <mongodb-pod-0> -- bash -c \
-'mongorestore -u "$MONGODB_USERNAME" -p"$MONGODB_PASSWORD" \
---uri="mongodb://$K8S_SERVICE_NAME" --db $MONGODB_DATABASE --gzip --drop --archive'
-v2 to v3
v3 introduced following backward incompatible changes
- Changed output-only fields failedDispatches and successDispatches to dispatch.failed and dispatch.successful respectively in Notification api. If your client app depends on the fields, change accordingly.
- Changed config notification.logSuccessfulBroadcastDispatches to notification.guaranteedBroadcastPushDispatchProcessing and reversed default value from false to true. If you don't want NotifyBC guarantees processing all subscriptions to a broadcast push notification in a node failure resilient way, perhaps for performance reason, set the value to false in file /src/config.local.js.
After above changes are addressed, upgrading to v3 is as simple as
git pull
-git checkout tags/v3.x.x -b <branch_name>
-yarn install && yarn build
-or, if NotifyBC is deployed to Kubernetes using Helm.
git pull
-git checkout tags/v3.x.x -b <branch_name>
-helm upgrade <release-name> -f helm/platform-specific/<platform>.yaml -f helm/values.local.yaml helm
-Replace v3.x.x with a v3 release, preferably latest, found in GitHub such as v3.1.2.
v3 to v4
v4 introduced following backward incompatible changes that need to be addressed in this order
The precedence of config, middleware and datasource files has been changed. Local file takes higher precedence than environment specific file. For example, for config file, the new precedence in ascending order is
- default file /src/config.ts
- environment specific file /src/config.<env>.js, where <env> is determined by environment variable NODE_ENV
- local file /src/config.local.js
To upgrade, if you have environment specific file, merge its content into the local file, then delete it.
Config smtp is changed to email.smtp. See SMTP for example.
Config inboundSmtpServer is changed to email.inboundSmtpServer. See Inbound SMTP Server for example.
Config email.inboundSmtpServer.bounce is changed to email.bounce. See Bounce for example.
Config notification.handleBounce is changed to email.bounce.enabled.
Config notification.handleListUnsubscribeByEmail is changed to email.listUnsubscribeByEmail.enabled. See List-unsubscribe by Email for example.
Config smsServiceProvider is changed to sms.provider. See Provider for example.
SMS service provider specific settings defined in config sms are changed to sms.providerSettings. See Provider Settings for example. The config object sms now encapsulates all SMS configs - provider, providerSettings and throttle.
Legacy config subscription.unsubscriptionEmailDomain is removed. If you have it defined in your file /src/config.local.js, replace with email.inboundSmtpServer.domain.
Helm chart added Redis that requires authentication by default. Create a new password in helm/values.local.yaml to facilitate upgrading
# in file helm/values.local.yaml
-redis:
- auth:
- password: '<secret>'
-After above changes are addressed, upgrading to v4 is as simple as
git pull
-git checkout tags/v4.x.x -b <branch_name>
-yarn install && yarn build
-or, if NotifyBC is deployed to Kubernetes using Helm.
git pull
-git checkout tags/v4.x.x -b <branch_name>
-helm upgrade <release-name> -f helm/platform-specific/<platform>.yaml -f helm/values.local.yaml helm
-Replace v4.x.x with a v4 release, preferably latest, found in GitHub such as v4.0.0.
v4 to v5
v5 introduced following backward incompatible changes
Replica set is required for MongoDB. If you deployed NotifyBC using Helm, replica set is already enabled by default.
If you use default in-memory database, data in server/database/data.json will not be migrated automatically. Manually migrate if necessary.
Update file src/datasources/db.datasource.local.[json|js|ts]
- rename url property to uri
- for other properties, instead of following LoopBack MongoDB data source, follow Mongoose connection options. In particular, host, port and database properties are no longer supported. Use uri instead.
For example, change
{ - "name": "db", - "connector": "mongodb", - "url": "mongodb://127.0.0.1:27017/notifyBC" -} -to
{ - "uri": "mongodb://127.0.0.1:27017/notifyBC" -} -If you deployed NotifyBC using Helm, this is taken care of.
API querying operators have changed. Replace following Loopback operators with corresponding MongoDB operators at your client-side API call.
Loopback operators MongoDB operators eq $eq and $and or $or gt, gte $gt, $gte lt, lte $lt, $lte between (no equivalent, replace with $gt, $and and $lt) inq, nin $in, $nin near $near neq $ne like, nlike (replace with $regexp) like, nlike, options: i (replace with $regexp) regexp $regex API order filter syntax has changed. Replace syntax from Loopback to Mongoose at client-side API call. For example, if your client-side code generates following API call
GET http://localhost:3000/api/configurations?filter={"order":["serviceName asc"]} -change to either
GET http://localhost:3000/api/configurations?filter={"order":[["serviceName","asc"]]} -or
GET http://localhost:3000/api/configurations?filter={"order":"serviceName"} -In MongoDB administrator collection, email has changed from case-sensitively unique to case-insensitively unique. Make sure administrator emails differ not just by case.
When a subscription is created by anonymous user, the data field is preserved. In earlier versions this field is deleted.
Dynamic tokens in subscription confirmation request message and duplicated subscription message are not replaced with subscription data, for example {subscription::...} tokens are left unchanged. Update the template of the two messages if dynamic tokens in them depends on subscription data.
Inbound SMTP Server no longer accepts command line arguments or environment variables as inputs. All inputs have to be defined in config files shown in the link.
If you deployed NotifyBC using Helm, change MongoDB password format in your local values yaml file from
# in file helm/values.local.yaml -mongodb: - auth: - rootPassword: <secret> - replicaSetKey: <secret> - password: <secret> -to
# in file helm/values.local.yaml -mongodb: - auth: - rootPassword: <secret> - replicaSetKey: <secret> - passwords: - - <secret> -
After above changes are addressed, to upgrade NotifyBC to v5,
if NotifyBC is deployed from source code, run
git pull -git checkout tags/v5.x.x -b <branch_name> -yarn install && yarn build -if NotifyBC is deployed to Kubernetes using Helm,
- backup MongoDB database
- runReplace <release-name> with installed helm release name
helm uninstall <release-name> - - delete PVCs used by MongoDB stateful set
- runReplace
git pull -git checkout tags/v5.x.x -b <branch_name> -helm install <release-name> -f helm/platform-specific/<platform>.yaml -f helm/values.local.yaml helm -- v5.x.x with a v5 release, preferably latest, found in GitHub such as v5.0.0.
- <release-name> with installed helm release name
- <platform> with openshift or aks depending on your platform
- restore MongoDB database
Web Console
After installing NotifyBC, you can start exploring NotifyBC resources by opening web console, a curated GUI, at http://localhost:3000. You can further explore full-blown APIs by clicking the API explorer Swagger UI embedded in web console.
Consult the API docs for valid inputs and expected outcome while you are exploring the APIs. Once you are familiar with the APIs, you can start writing code to call the APIs from either user browser or from a server application.
What you see in web console and what you get from API calls depend on how your requests are authenticated.
Ip whitelisting authentication
The API calls you made with API explorer as well as API calls made by web console from localhost are by default authenticated as super-admin requests because localhost is in admin ip list by default. Ip whitelisting authentication status is indicated by the icon on top right corner of web console.
To see the result of non super-admin requests, you can choose one of the following methods
- customize admin ip list to omit localhost (127.0.0.1)
- access web console from another ip not in the admin ip list
Client certificate authentication
If your ip is not in the admin ip list but you have setup a client certificate issued by NotifyBC server in browser, the API calls you made with API explorer as well as API calls made by web console are also authenticated as super-admin requests. Client certificate authentication status is indicated by the icon on top right corner of web console.
Anonymous
If you access web console from a client that is not in the admin ip list, you are by default anonymous user. Anonymous authentication status is indicated by the LOGIN button on top right corner of web console. Click the button to login.
Access token authentication
If you have not configured OIDC, the login button opens a login form. After successful login, the login button is replaced with the Access Token text field on top right corner of web console. You can edit the text field. If the new access token you entered is invalid, you are essentially logging yourself out. In such case Access Token text field is replaced by the LOGIN button.
The procedure to create an admin login account is documented in Administrator API
Tokens are not shared between API Explorer and web console
Despite API Explorer appears to be part of web console, it is a separate application. At this point neither the access token nor the OIDC access token are shared between the two applications. You have to use API Explorer's Authorize button to authenticate even if you have logged into web console.
OIDC authentication
If you have configured OIDC, then the login button will direct you to OIDC provider's login page. Once login successfully, you will be redirected back to NoitfyBC web console. OIDC authentication status is indicated by the LOGOUT button.
If you passed isAdmin validation, you are admin; otherwise you are authenticated user.
SiteMinder authentication
To get results of a SiteMinder authenticated user, do one of the following
- access the API via a SiteMinder proxy if you have configured SiteMinder properly
- use a tool such as curl that allows to specify custom headers, and supply SiteMinder header SM_USER:
curl -X GET --header "Accept: application/json" \
- --header "SM_USER: foo" \
- "http://localhost:3000/api/notifications"
-What's New
NotifyBC uses semantic versioning.
v5.0.0
See Upgrade Guide for more information.
- Runs on NestJS
- Bitnami MongoDB Helm chart is updated from version 10.7.1 to 14.3.2, with corresponding MongoDB from 4.4 to 7.0.4
- Bitnami Redis Helm chart is updated from version 14.7.2 to 16.13.2, with corresponding Redis from 6.2.4 to 6.2.7
Why v5?
NotifyBC was built on LoopBack since the beginning. While Loopback is an awesome framework at the time, it is evident by 2022 Loopback is no longer actively maintained
- features such as GraphQL have been in experimental state for years
- recent commits are mostly chores rather than enhancements
- core developers have ceased to contribute
To pave the way for future growth, switching platform becomes necessary. NestJS was chosen because
- both NestJS and Loopback are server-side Node.js frameworks
- NestJS has the closest feature set as Loopback. To a large extent NestJS is a superset of Loopback
- NestJS incorporates more technologies
v4.1.0
- Issue #50: Email message throttle
- applied sms throttle to all sms messages rather than just broadcast push notification.
- docs updates
v4.0.0
See v3 to v4 upgrade guide for more information.
- Issue #48: SMS message throttle
- Re-ordered config file precedence
- Re-organized Email and SMS configs
- docs updates
v3.1.0
- Issue #45: Reliability - Log skipped dispatches for broadcast push notifications
- docs updates
v3.0.0
See v2 to v3 upgrade guide for more information.
v2.9.0
- Issue #34: Helm - add k8s cronJob to backup MongoDB
- docs updates
v2.8.0
- Issue #28: Allow subscription data be used by mail merge dynamic tokens
- Issue #32: Allow escape mail merge delimiter
- docs updates
v2.7.0
- Issue #26: Allow filter specified in a notification
- docs updates
v2.6.0
- Helm chart updates
- docs updates
v2.5.0
- added helm chart. See OpenShift template to Helm upgrade guide
- docs updates
v2.4.0
- Issue #16: Support client certificate authentication
- misc web console adjustments
- docs updates
v2.3.0
- Issue #15: Support OIDC authentication for both admin and non-admin user
- misc web console adjustments
- docs updates
v2.2.0
- Issue #14: Support Administrator login, changing password, obtain access token in web console
- misc web console adjustments
- docs updates
v2.1.0
- Issue #13: Upgraded Vuetify from v0.16.9 to v2.4.3
- misc web console adjustments
- docs updates
v2.0.0
See Upgrade Guide for more information.
- Runs on LoopBack v4
- All code is converted to TypeScript
- Upgraded OAS from v2 to v3
- Docs is converted from Jekyll to VuePress
Why v2?
NotifyBC has been built on Node.js LoopBack framework since 2016. LoopBack v4, which was released in 2019, is backward incompatible. To keep software stack up-to-date, unless rewriting from scratch, it is necessary to port NotifyBC to LoopBack v4. Great care has been taken to minimize upgrade effort.
Getting Help
Need help with NotifyBC? Try these resources.
Documentation
Our guide to NotifyBC covering installation, writing, customization, deployment, and more.
View source
Use the source, Luke.
Add NotifyBC to almost any query, and you'll find just what you need.
Outstanding issues and requests
Search through the issues on the main NotifyBC development. Think you've found a bug? File a new issue.
A versatile notification API server
Versatile
-
-
- Anonymous or authenticated subscriptions -
- Push and in-app pull notifications -
- Email and SMS push notification channels -
- Unicast and broadcast message types -
- Broadcast push notification filter rules specifiable by both sender and subscriber -
- Notification auto-gen from RSS -
Non-intrusive
-
-
- Handles common backend business logic only, allowing site developer implement frontend UI using widgets native to the site - -
- Loose coupling - interacts with user browser or other server components through RESTful API - -
Secure & Reliable
-
-
- Support end-to-end encryption - -
- Multiple authentication strategies including client certificate for server-server and OIDC for user-server -
- Resilient to node failures -