WIP adding Envoy to the image

[supaPatrick]

this is what I think the initial commit for envoy into the image would look like. Envoy would be turned off and not in the same slice as other services but this commit would allow me to test easier moving forward.

[darora]

Generally looks good, left minor comments

[steve-chavez]

@darora @supaPatrick Can you share the rationale for introducing envoy?

[supaPatrick]

@darora @supaPatrick Can you share the rationale for introducing envoy?

Hey Steve, in our testing it is using significantly less footprint in memory which was the desired outcome of replacing nginx/kong.

[steve-chavez]

@supaPatrick Oh, I see. Are the tests public? Or is there an internal Notion link?

[supaPatrick]

@supaPatrick Oh, I see. Are the tests public? Or is there an internal Notion link?

Here are some of the difference when running envoy other then kong

# ps eo user,pid,vsz,rss $(pgrep -f 'nginx')     
USER       PID    VSZ   RSS
root     61413 658300 24784
kong     61414 721244 92532
kong     61415 721244 92516
kong     61416 721244 92372
kong     61417 721308 93044
kong     61418 721244 92548
kong     61419 721244 92536
# kong stop
Kong stopped
# supervisorctl start envoy                      
envoy: started
# ps eo user,pid,vsz,rss $(pgrep -f 'envoy') 
USER       PID    VSZ   RSS
envoy    62854   6692  2972
envoy    62855 2320208 47844


CONTAINER ID   NAME                           CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
5bae4572c390   infrastructure-gotrue-1        0.00%     4.746MiB / 7.667GiB   0.06%     32.1kB / 26kB     32.6MB / 7.04MB   11
f273cdcfbfee   infrastructure-stripe-sync-1   0.00%     4.77MiB / 7.667GiB    0.06%     5.71kB / 1.88kB   107MB / 48.1MB    22
82324f7d55c4   infrastructure-envoy-1         0.71%     11.3MiB / 7.667GiB    0.14%     590kB / 619kB     2.86GB / 10.7MB   21
579351ef40e9   infrastructure-monitoring-1    0.01%     42.67MiB / 7.667GiB   0.54%     1.45MB / 1.96MB   23.4GB / 1.62GB   6
d2596899c88b   infrastructure-kong-1          7.24%     346.8MiB / 7.667GiB   4.42%     2.9MB / 3.26MB    1.4GB / 175MB     7
21812133eee9   infrastructure-rest-1          0.30%     30.82MiB / 7.667GiB   0.39%     6.59MB / 5.04MB   1.85GB / 11MB     39
6c4ed216a190   infrastructure-db-1            0.02%     100.2MiB / 7.667GiB   1.28%     32.7MB / 27.8MB   2.38GB / 29.5MB   22
1d3f1e652bef   infrastructure-pg-api-1        0.61%     61.87MiB / 7.667GiB   0.79%     1.88kB / 0B       2.11GB / 40.5MB   23

[darora]

Looks good once we resolve #743 (comment)