Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KEP-4951: Configurable tolerance for HPA #4954

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
b01f5d8
Add KEP on Configurable tolerance for HPA.
jm-franc Nov 6, 2024
32bc885
Add details about upgrade/downgrades, feature gate.
jm-franc Nov 6, 2024
8f9396f
Set title.
jm-franc Nov 6, 2024
da1052d
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/kep.yaml
jm-franc Nov 13, 2024
49c8b2f
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/kep.yaml
jm-franc Nov 13, 2024
71ed0a6
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/kep.yaml
jm-franc Nov 13, 2024
2fa9137
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/README.md
jm-franc Nov 13, 2024
d3b8b5c
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/README.md
jm-franc Nov 13, 2024
3d48ef6
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/README.md
jm-franc Nov 13, 2024
d99d1cf
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/README.md
jm-franc Nov 13, 2024
ecc3d81
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/README.md
jm-franc Nov 13, 2024
dc19885
update the KEP with cluster access mechanism
ryanzhang-oss Nov 4, 2024
99aa891
DRA: update unit test coverage for 1.32
pohly Nov 5, 2024
c42815e
KEP-4412: update resource usage question for ServiceAccountNodeAudien…
aramase Nov 6, 2024
f22cc00
Update README.md
HarshalNeelkamal Oct 30, 2024
2c8614b
Mark KEP-2681 as implemented
carlory Nov 8, 2024
2b0bb31
Update to address pr00se and raywainman comments/suggestions.
jm-franc Nov 13, 2024
9fc2d07
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/README.md
jm-franc Nov 13, 2024
4058fff
Improve description of the 'tolerance' field.
jm-franc Nov 13, 2024
b8d7381
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/README.md
jm-franc Nov 13, 2024
566e097
Update keps/sig-autoscaling/4951-configurable-hpa-tolerance/README.md
jm-franc Nov 13, 2024
23c4ff1
Improve the section dedicated to risks.
jm-franc Nov 13, 2024
df5b1b9
Apply suggestions from code review.
jm-franc Nov 13, 2024
0f848b4
Fix wording.
jm-franc Nov 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1082,7 +1082,12 @@ This through this both in small and large cases, again with respect to the
[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
-->

No.
As part of the `ServiceAccountNodeAudienceRestriction` feature, KAS will need to watch PersistentVolumeClaims, PersistentVolumes and CSIDrivers
to determine the audiences that the kubelet is allowed to generate service account tokens for. These new informers (which are feature gated) will
result in additional resource usage in the KAS.
- Node authorizer is already watching persistent volumes via informers today.
- CSIDriver objects are expected to be ~few and ~slow-moving, so the impact is expected to be minimal.
- PersistentVolumeClaims are expected to be more numerous and more dynamic, so there could be more impact here.

###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

Expand Down
6 changes: 3 additions & 3 deletions keps/sig-auth/740-service-account-external-signing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,12 +245,12 @@ message Key {
message MetadataRequest {}

message MetadataResponse {
// used by kube-apiserver for defaulting/validation of JWT lifetime while accounting for configuration flag values:
// used by kube-apiserver as the max token lifetime and for validation against configuration flag values:
// 1. `--service-account-max-token-expiration`
// 2. `--service-account-extend-token-expiration`
//
// * If `--service-account-max-token-expiration` is greater than `max_token_expiration_seconds`, kube-apiserver treats that as misconfiguration and exits.
// * If `--service-account-max-token-expiration` is not explicitly set, kube-apiserver defaults to `max_token_expiration_seconds`.
// * If `--service-account-max-token-expiration` is set while external-jwt-signer is configured, kube-apiserver treats that as misconfiguration and exits.
// * If `--service-account-max-token-expiration` is not set, kube-apiserver uses `max_token_expiration_seconds` as max token lifetime.
// * If `--service-account-extend-token-expiration` is true, the extended expiration is `min(1 year, max_token_expiration_seconds)`.
//
// `max_token_expiration_seconds` must be at least 600s.
Expand Down
840 changes: 840 additions & 0 deletions keps/sig-autoscaling/4951-configurable-hpa-tolerance/README.md

Large diffs are not rendered by default.

44 changes: 44 additions & 0 deletions keps/sig-autoscaling/4951-configurable-hpa-tolerance/kep.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
title: Configurable tolerance for HPA
kep-number: 4951
authors:
- "@pr00se"
- "@jm-franc"
owning-sig: sig-autoscaling
status: provisional
creation-date: 2024-11-05
reviewers:
jm-franc marked this conversation as resolved.
Show resolved Hide resolved
- "@gjtempleton"
- "@raywainman"
approvers:
- TBD

see-also:
- "/keps/sig-autoscaling/853-configurable-hpa-scale-velocity"
replaces:

# The target maturity stage in the current dev cycle for this KEP.
stage: alpha

# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
latest-milestone: TBD

# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
alpha: TBD
beta: TBD
stable: TBD

# The following PRR answers are required at alpha release
# List the feature gate name and the components for which it must be enabled
feature-gates:
- name: HPAConfigurableTolerance
components:
- kube-apiserver
jm-franc marked this conversation as resolved.
Show resolved Hide resolved
- kube-controller-manager
disable-supported: true

# The following PRR answers are required at beta release
#metrics:
# - my_feature_metric
76 changes: 58 additions & 18 deletions keps/sig-multicluster/4322-cluster-inventory/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,11 @@ tags, and then generate with `hack/update-toc.sh`.
- [Version](#version)
- [Properties](#properties)
- [Conditions](#conditions)
- [Cluster Access](#cluster-access)
- [Pull Model with Work API](#pull-model-with-work-api)
- [Push Model with Identity Federation (Recommended)](#push-model-with-identity-federation-recommended)
- [Push Model via Credentials in Secret (Not Recommended)](#push-model-via-credentials-in-secret-not-recommended)
- [Secret format](#secret-format)
- [API Example](#api-example)
- [Scalability implication](#scalability-implication)
- [Test Plan](#test-plan)
Expand Down Expand Up @@ -164,7 +169,7 @@ Items marked with (R) are required *prior to targeting to a milestone / release*
- [ ] (R) Design details are appropriately documented
- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
- [ ] e2e Tests for all Beta API Operations (endpoints)
- [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
- [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
- [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
- [ ] (R) Graduation criteria is in place
- [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
Expand Down Expand Up @@ -249,7 +254,7 @@ management. Examples of consumers includes:
this area.
* GitOps tools (ArgoCD, flux etc) are having the requirement to deploy
workload to multiple clusters. They either need to build the cluster
concept by themselves or understand cluster API from each different
concept by themselves or understand APIs representing a cluster from each
cluster management project. A common ClusterProfile API can provide
a thin compatible layer for different projects.
* Operation tools or customized external consumers: this API gives a
Expand Down Expand Up @@ -288,6 +293,7 @@ and make progress.
* Provide a standard reference implementation.
* Define specific implementation details beyond general API behavior.
* Offering functionalities related to multi-cluster orchestration.
* Define the Consumer registration API


## Proposal
Expand Down Expand Up @@ -319,14 +325,14 @@ the API proposed by this KEP aims to
- **Member Cluster**: A kubernetes cluster that is part of a cluster inventory.

- **Cluster Manager**: An entity that creates the ClusterProfile API object per member cluster,
and keeps their status up-to-date. Each cluster manager MUST be identified with a unique name.
Each ClusterProfile resource SHOULD be managed by only one cluster manager. A cluster manager SHOULD
and keeps their status up-to-date. Each cluster manager MUST be identified with a unique name.
Each ClusterProfile resource SHOULD be managed by only one cluster manager. A cluster manager SHOULD
have sufficient permission to access the member cluster to fetch the information so it can update the status
of the ClusterProfile API resource.

- **ClusterProfile API Consumer**: the person running the cluster managers
or the person developing extensions for cluster managers for the purpose of
workload distribution, operation management etc.
- **ClusterInventory Consumer**: Controllers or tools that leverage the ClusterProfile for the purpose of
workload distribution, operation management etc. Their name must be unique for a single inventory. They
might need to register themselves with the Cluster Manager which is not defined in this KEP.

### User Stories (Optional)

Expand Down Expand Up @@ -401,19 +407,19 @@ if all its member clusters adhere to the [namespace sameness](https://github.com
Note that a cluster can only be in one ClusterSet while there is not such restriction for a cluster inventory.

#### How should the API be consumed?
We recommend that all ClusterProfile objects within the same cluster inventory reside on
We recommend that all ClusterProfile objects within the same cluster inventory reside on
a dedicated Kubernetes cluster (aka. the hub cluster). This approach allows consumers to have a single integration
point to access all the information within a cluster inventory. Additionally, a multi-cluster aware
controller can be run on the dedicated cluster to offer high-level functionalities over this inventory of clusters.

#### How should we organize ClusterProfile objects on a hub cluster?
While there are no strict requirements, we recommend making the ClusterProfile API a namespace-scoped object.
While there are no strict requirements, we recommend making the ClusterProfile API a namespace-scoped object.
This approach allows users to leverage Kubernetes' native namespace-based RBAC if they wish to restrict access to
certain clusters within the inventory.
certain clusters within the inventory.

However, if a cluster inventory represents a ClusterSet, all its ClusterProfile objects MUST be part of the same clusterSet
and namespace must be used as the grouping mechanism. In addition, the namespace must have a label with the key "clusterset.multicluster.x-k8s.io"
and the value as the name of the clusterSet.
and the value as the name of the clusterSet.

#### Uniqueness of the ClusterProfile object
While there are no strict requirements, we recommend that there is only one ClusterProfile object representing any member cluster
Expand Down Expand Up @@ -451,21 +457,22 @@ represent a cluster and support the above use case at the minimum scope.

The target consumers of the API are users who manage the clusters and
other tools to understand the clusters concept from the API for
multicluster scheduling, workload distribution and cluster management.
multi-cluster scheduling, workload distribution and cluster management.
The API aims to provide the information for the consumers to answer the
below questions:

* Is the cluster under management?
* How can I select a cluster?
* Is the cluster healthy?
* How to access the cluster?
* Does the cluster have certain capabilities or properties?
* Does the cluster have sufficient resources?

### Cluster Name

It is required that cluster name is unique for each cluster, and it
should also be unique among different providers (cluster manager). It
is cluster managers's responsibility to ensure the name uniqueness.
is cluster manager's responsibility to ensure the name uniqueness.

It's the responsibility of the cluster manager platform administrator
to ensure cluster name uniqueness.
Expand Down Expand Up @@ -552,6 +559,39 @@ Predefined condition types:
The status of the cluster SHOULD be updated by the cluster manager under
this condition.

### Cluster Access
There are multiple methods for a ClusterInventory Consumer to gain access to the cluster represented by a ClusterProfile API.
This KEP does not define the exact mechanism for each approach, but it is recommended that ClusterInventory Consumers avoid
using a secret if possible. Here are the three approaches:

#### Pull Model with Work API
The ClusterInventory Consumer can leverage the [Work API](https://multicluster.sigs.k8s.io/concepts/work-api/) so that
the work API agent on the Member Cluster can *pull* the necessary objects from the management cluster where the ClusterInventory
Consumer operates to the Member Cluster. This approach allows the Cluster Manager to manage the RBAC for each Consumer on the
management cluster, ensuring that their access is restricted to the corresponding namespaces of the ClusterProfile objects.

#### Push Model with Identity Federation (Recommended)
The ClusterInventory consumer can utilize identity federation mechanisms, such as [Azure Workload Identity Federation](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation)
or [GCP Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation). This allows the
ClusterInventory Consumer to use an identity that already has access to the clusters in the Cluster Inventory.
While the Cluster Manager can assist in setting up the federation, it is not a mandatory requirement.

#### Push Model via Credentials in Secret (Not Recommended)
The ClusterInventory Consumer can obtain credentials to access the cluster represented by a ClusterProfile object by reading
from a secret. In this approach, the Cluster Manager generates secrets containing the necessary credentials within the namespace
accessible to the ClusterInventory Consumer. For this to function correctly, Cluster Managers must be aware of the following details
about the consumer: their name, whether credentials are required, and the preferred unique namespace for reading credentials as secrets.
Those information can be obtained during the "registering" process but this is out of the scope of this KEP.

##### Secret format
- The secret *MUST* reside in the namespace with the label `x-k8s.io/cluster-inventory-consumer` with the value being the name of the ClusterInventory Consumer.
- The secret *MUST* contain the label `x-k8s.io/cluster-profile` with the value being the name of the ClusterProfile object that the secret is associated with.
- The access information in the secret must contain the following fields
- **Config**: This field contains cluster access information compatible with the
[kubeconfig format](https://github.com/kubernetes/kubernetes/blob/v1.31.2/staging/src/k8s.io/client-go/tools/clientcmd/api/types.go#L31).
- Since a single [Kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) supports access to multiple clusters, the Cluster manager *MUST* ensure that each secret contains access information for only a single consumer.


## API Example

```yaml
Expand All @@ -562,18 +602,18 @@ metadata:
labels:
x-k8s.io/cluster-manager: some-cluster-manager
spec:
displayName: some-cluster
displayName: cluster-us-east
clusterManager:
name: some-cluster-manager
status:
version:
kubernetes: 1.28.0
properties:
version:
kubernetes: 1.28.0
properties:
- name: clusterset.k8s.io
value: some-clusterset
- name: location
value: apac
conditions:
conditions:
- type: ControlPlaneHealthy
status: True
lastTransitionTime: "2023-05-08T07:56:55Z"
Expand Down
1 change: 1 addition & 0 deletions keps/sig-network/2681-pod-host-ip/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -476,6 +476,7 @@ N/A
- 2021-05-06: Initial KEP
- 2023-08-15: Alpha release with Kubernetes 1.28
- 2023-10-30: Beta release with Kubernetes 1.29
- 2024-03-01: GA release with Kubernetes 1.30

## Drawbacks

Expand Down
2 changes: 1 addition & 1 deletion keps/sig-network/2681-pod-host-ip/kep.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ authors:
owning-sig: sig-network
participating-sigs:
- sig-node
status: implementable
status: implemented
creation-date: 2021-05-06
reviewers:
- "@thockin"
Expand Down
15 changes: 14 additions & 1 deletion keps/sig-node/4381-dra-structured-parameters/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2645,14 +2645,15 @@ More specifically for DRA code in Kubernetes (based on checking out the code):
<!--
Generated with:

go test -cover ./pkg/scheduler/framework/plugins/dynamicresources/... ./pkg/controller/resourceclaim ./pkg/kubelet/cm/dra/... ./staging/src/k8s.io/dynamic-resource-allocation/cel ./staging/src/k8s.io/dynamic-resource-allocation/structured | sed -e 's/.*\(k8s.io[a-z/-]*\).*coverage: \(.*\) of statements/- `\1`: \2/' | sort
go test -cover ./pkg/apis/resource/validation ./pkg/scheduler/framework/plugins/dynamicresources/... ./pkg/controller/resourceclaim ./pkg/kubelet/cm/dra/... ./staging/src/k8s.io/dynamic-resource-allocation/cel ./staging/src/k8s.io/dynamic-resource-allocation/structured | sed -e 's/.*\(k8s.io[a-z/-]*\).*coverage: \(.*\) of statements/- `\1`: \2/' | sort

-->

v1.31.0:

- `k8s.io/dynamic-resource-allocation/cel`: 80.0%
- `k8s.io/dynamic-resource-allocation/structured`: 82.7%
- `k8s.io/kubernetes/pkg/apis/resource/validation`: 77.1%
- `k8s.io/kubernetes/pkg/controller/resourceclaim`: 70.1%
- `k8s.io/kubernetes/pkg/kubelet/cm/dra`: 78.6%
- `k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin`: 77.9%
Expand All @@ -2663,12 +2664,24 @@ Start of v1.32 development cycle (v1.32.0-alpha.1-178-gd9c46d8ecb1):

- `k8s.io/dynamic-resource-allocation/cel`: 88.8%
- `k8s.io/dynamic-resource-allocation/structured`: 82.7%
- `k8s.io/kubernetes/pkg/apis/resource/validation`: 77.1%
- `k8s.io/kubernetes/pkg/controller/resourceclaim`: 70.0%
- `k8s.io/kubernetes/pkg/kubelet/cm/dra`: 78.6%
- `k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin`: 77.7%
- `k8s.io/kubernetes/pkg/kubelet/cm/dra/state`: 46.2%
- `k8s.io/kubernetes/pkg/scheduler/framework/plugins/dynamicresources`: 72.9%

End of v1.32 development cycle (~ https://github.com/kubernetes/kubernetes/pull/127511 = v1.32.0-alpha.3-414-g50d0f920c01):

- `k8s.io/dynamic-resource-allocation/cel`: 89.6%
- `k8s.io/dynamic-resource-allocation/structured`: 91.3%
- `k8s.io/kubernetes/pkg/apis/resource/validation`: 98.4%
- `k8s.io/kubernetes/pkg/controller/resourceclaim`: 73.5%
- `k8s.io/kubernetes/pkg/kubelet/cm/dra`: 78.7%
- `k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin`: 91.0%
- `k8s.io/kubernetes/pkg/kubelet/cm/dra/state`: 46.2%
- `k8s.io/kubernetes/pkg/scheduler/framework/plugins/dynamicresources`: 77.4%

##### Integration tests

<!--
Expand Down