Universal Runtime Support #704

trieloff · 2021-01-26T14:28:05Z

Adds support for the universal runtime.

Request types:

@tripodsan for static_url static purge query embed we rely on the OpenWhisk linking feature that we have not replicated yet in helix-dispatch. Should we just hardcode a hostname (a.k.a. namespace) such as helix-shared.anywhere.run? We'd need to make sure the corresponding actions are deployed to this service.

The `dispatch` request type handles the majority of all runtime-bound requests

…d the same

github-actions · 2021-01-26T14:36:17Z

This PR will trigger a minor release when merged.

codecov · 2021-01-26T15:15:07Z

Codecov Report

Merging #704 (d1628cb) into main (d6be062) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              main      #704   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           15        15           
  Lines          348       348           
=========================================
  Hits           348       348

Impacted Files	Coverage Δ
src/fastly/backends.js	`100.00% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d5be38...d1628cb. Read the comment docs.

tripodsan · 2021-01-27T00:33:54Z

for static_url static purge query embed we rely on the OpenWhisk linking feature that we have not replicated yet in helix-dispatch. Should we just hardcode a hostname (a.k.a. namespace) such as helix-shared.anywhere.run? We'd need to make sure the corresponding actions are deployed to this service.

I don't quite understand the problem. the action invokes stay in the same environment.

eg: a request to:

https://helix-pages.anywhere.run/helix-services/dispatch@v4

will invoke the dispatch on the helix-pages namespace, which will then use the __OW_API_HOST and __OW_NAMESPACE to invoke the subsequent actions. so they don't use the anywhere gateway anymore.

see: https://github.com/adobe/helix-deploy/blob/7e21dc39969dadbd224d661b43240c948ded28c0/src/template/resolver.js#L94-L95

trieloff · 2021-01-27T09:24:50Z

Let's take dispatch and purge as examples.

dispatch works just as you described, Fastly requests https://helix-pages.anywhere.run/helix-services/dispatch@v4 so everything is fine.

Now, purge is different:

purge could be https://helix-pages.anywhere.run/helix-services/purge@v1, but this assumes that

either that the helix-pages project deployed all helix-related services to it's own namespace (on OW) and API gateway (on AWS)
or that the helix-services package is linked on OpenWhisk and the helix-pages API gateway is somehow mounting the lambdas from the helix AWS account into the helix-pages API gateway

or:
2. purge could be https://helix-services.anywhere.run/helix-services/purge@v1, but this comes with the drawback of:

no neat service sharing and accounting of invocations
all helix-services invocations count against the same rate limits

tripodsan · 2021-01-27T10:17:09Z

we don't have namespaces in AWS, and all helix-services are mounted in the helix-pages namespace in OW.

purge could be https://helix-services.anywhere.run/helix-services/purge@v1, ...

no. that would be wrong.

I still don't understand the problem :-)

trieloff · 2021-01-27T11:03:24Z

We deploy helix-pages (html action, etc) and helix-services (e.g. purge) differently in OW. I'd assume that we would have the same separation in AWS, i.e. different subdomains for anywhere.run. The question is how to reconcile this here.

tripodsan · 2021-01-27T11:41:45Z

We deploy helix-pages (html action, etc) and helix-services (e.g. purge) differently in OW. I'd assume that we would have the same separation in AWS, i.e. different subdomains for anywhere.run. The question is how to reconcile this here.

ah, ok. in AWS we just ignore the namespaces, and hope there is no collision in the packages :-)
at a later stage, we could choose different API Gateways for different namespaces, and then use different AWS resource groups to execute the functions, for separation of limits/costs (I guess)

trieloff · 2021-01-28T14:38:02Z

Ok, for now I will assume the specified subdomain has all the actions and later we can build adobe/helix-deploy#91

depends on adobe/helix-publish#704 and is probably broken. The `universal` strain can only be selected via header, so there is little risk of breaking things until you ask for it

tripodsan

hmm..there is a lot of duplicated code that calculates the universal/namespace vars. can't this be done in a function?

trieloff · 2021-01-30T15:05:57Z

VCL does not have function, it only has subroutines, which makes this difficult. Ideally, we'd have functions that return a struct with var.namespace and var.universal that we could simply call. I thought about building something that would inline a subroutine, but decided to go with the more obvious, but also more verbose way.

stefan-guggisberg

seems ok (but to be honest: I don't really understand it)

trieloff

@stefan-guggisberg I've added some elaboration.

trieloff · 2021-02-01T14:41:48Z

layouts/fastly/helix.vcl

+  # We need the action root for the next bit
+  call hlx_action_root;
+
+  if (req.http.X-Action-Root ~ "(^|^https://)([^/:\.]+)(/|\.([^/]+)/)([^/]+)") {


This pattern supports both formats of the package parameter: namespace/package and https://namespace.anywhere.run/package

trieloff · 2021-02-01T14:43:02Z

layouts/fastly/helix.vcl

+      set var.universal = true;
+      set var.hostname = var.namespace + "." + re.group.4;
+      set req.backend = F_UniversalRuntime;
+      set req.http.X-Backend-Host = var.hostname;


We are using the universal runtime, so we need to store the backend hostname, e.g. namespace

trieloff · 2021-02-01T14:44:17Z

layouts/fastly/helix.vcl

@@ -555,8 +581,8 @@ sub hlx_type_static_url {
    set var.static_version = {"const:static_version"};
  }

-  set req.http.X-Action-Root = "/api/v1/web/" + table.lookup(secrets, "OPENWHISK_NAMESPACE") + "/helix-services/static@" + var.static_version;
-  set req.http.X-Backend-URL = req.http.X-Action-Root
+  set req.http.X-Backend-URL = if(var.universal, "/", "/api/v1/web" + "/" + var.namespace)


Add the /api/v1/web/${namespace} prefix for OpenWhisk – Universal Runtime does not need it.

trieloff · 2021-02-01T14:47:23Z

layouts/fastly/helix.vcl

@@ -1761,6 +1910,19 @@ sub hlx_bereq {
      } else {
        set bereq.http.hlx-forwarded-host = req.http.X-Orig-Host;
      }
+    } elsif (req.backend == F_UniversalRuntime) {
+      // Set the hostame for the universal Runtime service
+      set bereq.http.Host = req.http.X-Backend-Host;


Use the backend hostname set here https://github.com/adobe/helix-publish/pull/704/files#diff-9e59debf5608a9bdb1f1a87b891b2f995823ea79254a122b8d1fcb2bb1f69785R567 (for instance)

As *.anywhere.run will always go to Fastly, setting the subdomain name here allows Fastly to pick the correct gateway service.

adobe-bot · 2021-02-02T09:04:15Z

🎉 This PR is included in version 8.2.0 🎉

The release is available on:

v8.2.0
GitHub release

Your semantic-release bot 📦🚀

trieloff added 4 commits January 26, 2021 11:41

feat(backends): add universal runtime backend

cbcff71

test(backends): adjust reset fixtures

ea4e719

test(backends): adjust test conditions

53f5264

feat(vcl): allow using universal runtime in dispatch request type

87025f1

The `dispatch` request type handles the majority of all runtime-bound requests

trieloff marked this pull request as draft January 26, 2021 14:34

fix(vcl): make sure adobe io runtime and universal runtime are treate…

fe505af

…d the same

trieloff added 4 commits January 26, 2021 14:37

test(integration): log errors when available

88dcdec

fix(vcl): use x-action-root to determine namespace and package

9a3b818

fix(vcl): fix syntax error

a9218f3

test(integration): run all tests again

5042695

trieloff added 6 commits January 29, 2021 10:48

feat(static): support universal runtime for static requests

add85df

feat(purge): add universal runtime support

e8bd33b

feat(cgi): add support for universal runtime

d297a7b

feat(embed): support universal runtime

97237a1

feat(query): add universal runtime support

4b84092

feat(content): add support for universal runtime

d1628cb

trieloff marked this pull request as ready for review January 29, 2021 12:26

trieloff requested review from tripodsan and stefan-guggisberg January 29, 2021 12:26

trieloff mentioned this pull request Jan 29, 2021

feat(helix): try using the universal runtime adobe/helix-pages#655

Merged

tripodsan approved these changes Jan 30, 2021

View reviewed changes

stefan-guggisberg approved these changes Feb 1, 2021

View reviewed changes

trieloff commented Feb 1, 2021

View reviewed changes

trieloff merged commit 988aca5 into main Feb 2, 2021

trieloff deleted the universal-runtime branch February 2, 2021 09:02

adobe-bot added the released label Feb 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Universal Runtime Support #704

Universal Runtime Support #704

trieloff commented Jan 26, 2021 •

edited

Loading

github-actions bot commented Jan 26, 2021

codecov bot commented Jan 26, 2021 •

edited

Loading

tripodsan commented Jan 27, 2021 •

edited

Loading

trieloff commented Jan 27, 2021

tripodsan commented Jan 27, 2021 •

edited

Loading

trieloff commented Jan 27, 2021

tripodsan commented Jan 27, 2021

trieloff commented Jan 28, 2021

tripodsan left a comment

trieloff commented Jan 30, 2021

stefan-guggisberg left a comment

trieloff left a comment

trieloff Feb 1, 2021

trieloff Feb 1, 2021

trieloff Feb 1, 2021

trieloff Feb 1, 2021

adobe-bot commented Feb 2, 2021

Universal Runtime Support #704

Universal Runtime Support #704

Conversation

trieloff commented Jan 26, 2021 • edited Loading

github-actions bot commented Jan 26, 2021

codecov bot commented Jan 26, 2021 • edited Loading

Codecov Report

tripodsan commented Jan 27, 2021 • edited Loading

trieloff commented Jan 27, 2021

tripodsan commented Jan 27, 2021 • edited Loading

trieloff commented Jan 27, 2021

tripodsan commented Jan 27, 2021

trieloff commented Jan 28, 2021

tripodsan left a comment

Choose a reason for hiding this comment

trieloff commented Jan 30, 2021

stefan-guggisberg left a comment

Choose a reason for hiding this comment

trieloff left a comment

Choose a reason for hiding this comment

trieloff Feb 1, 2021

Choose a reason for hiding this comment

trieloff Feb 1, 2021

Choose a reason for hiding this comment

trieloff Feb 1, 2021

Choose a reason for hiding this comment

trieloff Feb 1, 2021

Choose a reason for hiding this comment

adobe-bot commented Feb 2, 2021

trieloff commented Jan 26, 2021 •

edited

Loading

codecov bot commented Jan 26, 2021 •

edited

Loading

tripodsan commented Jan 27, 2021 •

edited

Loading

tripodsan commented Jan 27, 2021 •

edited

Loading