Add retry and background run to node taint removal

[ConnorJC3]

Is this a bug fix or adding new feature?

Bug fix

What is this PR about? / Why do we need it?

Sometimes, the node taint removal can fail (for example, due to a temporary networking outage, or if another client tries to remove a taint at the same time as us and gets to the API server first). In that case we should retry the taint removal.

This PR does two things:

• Moves the taint removal to run in a goroutine in the background so it doesn't block node startup
• Adds retries with an exponential backoff to the taint removal

What testing is done?

Manually tested on a local cluster (but please at least one reviewer double check my work), as well as CI

[github-actions]

Code Coverage Diff

File	Old Coverage	New Coverage	Delta
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/node.go	79.8%	78.7%	-1.1

[torredil]

make verify needs fixing:

pkg/driver/node.go:859:30: `removeTaintInBackground` - `k8sClient` is unused (unparam)
func removeTaintInBackground(k8sClient cloud.KubernetesAPIClient) {

code changes lgtm, manually testing. Will apply label after manual validation + above is resolved.

[torredil]

/retest

[AndrewSirenko]

Solves active customer problem.

Offline discussion was had about configuration values for taintRemovalBackoff. Decided that currently this is a two-way door solution, and we may alleviate cx painpoints without introducing additional configuration option. Currently any retry/timeout is better than no retry and an infinite timeout.

[torredil]

/retest

[torredil]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment