Weight changes for backends should not trigger a reload

[brianehlert]

When only the weight is changed for upstreams, such as performing a blue/green or canary - NGINX Plus should not be reloaded.
This is possible through the N+ API just like dynamic upstream configuration.
This idea is described here:
https://www.nginx.com/blog/dynamic-a-b-testing-with-nginx-plus/#Using-the-Key-Value-Store-with-Split-Clients
NGINX Ingress Controller already uses Split Clients to implement the weight feature.

• Support weights of 0 - 100 (1 - 99 is constraining)
• make weight changes without reloading
• Plus exclusive feature (changes without reload)
• NGINX OSS will still reload on weight change

This work should primarily be about using the NGINX Plus API to apply the weight changes when NGINX Plus is present.
If the deployment is using NGINX OSS then a soft reload should still be required.

This work can be split when delivered

[danielnginx]

Could we have Support weights of 0 - 100 (1 - 99 is constraining) done first as reduced piece of work to have it as base to the other requirements? That will also facilitating testing as we can make sure that the new weights of 0 - 100 works as expected without the new reload capabilities.

[shaun-nx]

Linking a few docs here that will be helpful when working on this:
NGINX docs on the split_clients module: https://nginx.org/en/docs/http/ngx_http_split_clients_module.html
Current VS/VSR docs on how we do weight configuration today: https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#split

[brianehlert]

#4509

[brianehlert]

Released with 3.5 when NGINX Plus is present.