Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kraft #74

Open
wants to merge 53 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
dfc0824
Add koperator/api changes for KRaft support
panyuenlau Jul 25, 2023
722fd96
Merge branch 'master' into kraft-api
panyuenlau Jul 26, 2023
30427bf
Merge branch 'master' into kraft-api
panyuenlau Jul 27, 2023
54af705
Move processRoles under brokerConfig
panyuenlau Jul 27, 2023
dbf3200
Update comments for ZK-relevant configurations in kafkacluster_types.go
panyuenlau Jul 27, 2023
219d998
Rebase origin/kraft-api
panyuenlau Jul 20, 2023
314ea41
Support running Kafka cluster in KRaft mode
panyuenlau Jul 20, 2023
7613f06
brokerRoles -> processRoles to match upstream Kafka naming
panyuenlau Jul 21, 2023
aca2396
Update pod start-up process so pods can be restarted during rolling u…
panyuenlau Jul 21, 2023
c5a4819
Rebase from origin/kraft-api
panyuenlau Jul 23, 2023
5d1b777
Update exsiting integration tests and func signatures
panyuenlau Jul 23, 2023
aa4f5f5
Fix broker configurations; add unit tests for broker configurations u…
panyuenlau Jul 24, 2023
7431c93
Remove unnecessary method from koperator/api
panyuenlau Jul 25, 2023
2e3be06
Extend integration tests to cover KRaft mode
panyuenlau Jul 25, 2023
78e63ee
make lint-fix
panyuenlau Jul 25, 2023
3b5aff9
Update static KafkaCluster yamls; add check for kraft mode before set…
panyuenlau Jul 25, 2023
f9aedac
Rebase from origin/koperator-api
panyuenlau Jul 25, 2023
5804835
Use util functions that got moved to the koperator/api module
panyuenlau Jul 25, 2023
ca16422
Remove unineteded changes during rebase
panyuenlau Jul 25, 2023
afd567b
Do not take active controller identity into consideration when reorde…
panyuenlau Jul 25, 2023
bbc0307
Update implementation to accomomdate the latest KafkaCluster API change
panyuenlau Jul 27, 2023
455ef3f
Make comments about ZK-relevant configurations more clear
panyuenlau Jul 27, 2023
6b3d616
Add ConcurrentBrokerRestartCountPerRack to RollingUpgradeConfig (#1002)
ctrlaltluc Jul 27, 2023
0a90251
Small refactoring
panyuenlau Jul 28, 2023
63e15fa
Merge branch 'master' into kraft
panyuenlau Jul 28, 2023
aa859a1
Exclude controller-only nodes from all CC operations
panyuenlau Jul 30, 2023
28a4a75
Add processRoles label key for Kafka pods in KRaft mode; export consts
panyuenlau Jul 30, 2023
514fa07
Allow concurrent broker restarts from same AZ (broker rack) (#62)
amuraru Jun 10, 2023
16a9fc2
Fix flaky test by deleting nodeports explicitly (#67)
ctrlaltluc Jun 13, 2023
cdfb6b9
Allow dashes when parsing broker rack (#68)
ctrlaltluc Jun 28, 2023
18e7253
Upgrade Kafka to 3.6.0 (#69)
ctrlaltluc Oct 11, 2023
edb7ebf
Upgrade dependencies
amuraru Dec 12, 2023
5f78c06
Fix wrong port on expectEnvoyWithConfigAz2Tls test (#70)
dobrerazvan Dec 19, 2023
7383921
Upgrade Kafka to 3.6.1 (#71)
cristianpetrache Dec 22, 2023
2da23a0
working kraft
dobrerazvan Jan 23, 2024
57d22ec
Merge origin/master
dobrerazvan Jan 29, 2024
51fb7d6
Fixing go.mod
dobrerazvan Feb 7, 2024
bea02f8
Fixing tests post merge
dobrerazvan Feb 7, 2024
7b70fa6
Merge remote-tracking branch 'origin/master' into kraft
dobrerazvan Feb 7, 2024
46cf1a8
Fix tests
dobrerazvan Feb 7, 2024
168340e
Fix tests
dobrerazvan Feb 13, 2024
f205723
Fix unnecessary append
dobrerazvan Feb 13, 2024
2bb6c76
Fix go.mod
dobrerazvan Feb 13, 2024
58aa672
Update CRD
dobrerazvan Feb 13, 2024
112173c
Update go.mod for e2e
dobrerazvan Feb 13, 2024
c927063
Update go.mod
dobrerazvan Feb 13, 2024
b64f2ed
More fixes
dobrerazvan Feb 13, 2024
8172c6e
Upgrade Kafka image to use Java v21 (#72)
amuraru Feb 16, 2024
2fe3531
Merge branch 'master' into kraft
dobrerazvan Feb 19, 2024
20be375
Remove jbod cc tests
dobrerazvan Feb 20, 2024
50c2b07
Merging Master to Kraft and Reverting Rack Removal (#89)
dvaseekara Dec 9, 2024
15f6634
Trigger E2E Test
Dec 9, 2024
b55b1f7
End to end testing with KRaft cluster (#92)
dvaseekara Dec 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions .github/actions/kind-create/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ inputs: # adding these parameters to make this reusable later on
kind_k8s_version:
description: 'The k8s version to use'
required: true
default: 'v1.24.15'
default: 'v1.27.3'
kind_config:
description: 'Path to the kind config to use'
required: true
Expand All @@ -23,7 +23,7 @@ runs:

steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v3
- uses: engineerd/[email protected]
with:
name: ${{ inputs.kind_cluster_name }}
Expand All @@ -45,3 +45,12 @@ runs:
chmod 600 $kubeconfig_path
echo "kubeconfig=$(echo $kubeconfig_path)" >> $GITHUB_OUTPUT
shell: bash

- name: Install cloud-provider-kind
id: cloud-provider-kind
run: |
echo "Install cloud-provider-kind"
go install sigs.k8s.io/cloud-provider-kind@latest
kubectl label node e2e-kind-control-plane node.kubernetes.io/exclude-from-external-load-balancers-
~/go/bin/cloud-provider-kind &
shell: bash
7 changes: 5 additions & 2 deletions .github/workflows/build-push-kafka-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,18 +24,21 @@ jobs:
echo ::set-output name=version::${VERSION}
echo ::set-output name=tags::${TAGS}
echo ::set-output name=created::$(date -u +'%Y-%m-%dT%H:%M:%SZ')
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
uses: docker/setup-buildx-action@v3
- name: Login to DockerHub
if: startsWith(github.ref, 'refs/tags/')
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and push
uses: docker/build-push-action@v2
uses: docker/build-push-action@v5
with:
context: docker/kafka
platforms: linux/amd64,linux/arm64
push: ${{ startsWith(github.ref, 'refs/tags/') }}
tags: ${{ steps.prep.outputs.tags }}
labels: |
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@ jobs:

steps:
- name: Set up Go
uses: actions/setup-go@93397bea11091df50f3d7e59dc26a7711a8bcfbe #v4.1.0
uses: actions/setup-go@v4
with:
go-version: 1.19
go-version: 1.21

- name: Checkout code
uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 #v3.5.3
uses: actions/checkout@v4

- name: Check Go modules dependency file integrity
run: |
Expand Down Expand Up @@ -49,15 +49,15 @@ jobs:

steps:
- name: Set up Go
uses: actions/setup-go@93397bea11091df50f3d7e59dc26a7711a8bcfbe #v4.1.0
uses: actions/setup-go@v4
with:
go-version: 1.19
go-version: 1.21

- name: Checkout code
uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 #v3.5.3
uses: actions/checkout@v4

- name: License cache
uses: actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 #v3.3.1
uses: actions/cache@v3
with:
path: .licensei.cache
key: license-v1-${{ hashFiles('**/go.sum') }}
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v1
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
Expand All @@ -54,7 +54,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v1
uses: github/codeql-action/autobuild@v2

# ℹ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl
Expand All @@ -68,4 +68,4 @@ jobs:
# make release

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1
uses: github/codeql-action/analyze@v2
12 changes: 9 additions & 3 deletions .github/workflows/e2e-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,18 @@ jobs:

steps:
- name: Set up Go
uses: actions/setup-go@v3
uses: actions/setup-go@v4
with:
go-version: 1.19
go-version: 1.21

# Enable Tmate Session if you'd like to Debut the E2E Kind Cluster
# - name: Setup tmate session
# uses: mxschmitt/action-tmate@v3
# with:
# detached: true

- name: Checkout code
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Build docker image
run: |
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/helm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ jobs:

steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- uses: azure/setup-helm@v1
- uses: azure/setup-helm@v3
with:
version: ${{ env.HELM_VERSION }}

- name: Add Helm repositories
run: |
helm repo add banzaicloud-stable "https://kubernetes-charts.banzaicloud.com"
# helm repo add banzaicloud-stable "https://kubernetes-charts.banzaicloud.com"
helm repo add incubator "https://charts.helm.sh/incubator"
helm repo add stable "https://charts.helm.sh/stable"

Expand Down
16 changes: 15 additions & 1 deletion .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ run:
deadline: 2m
skip-dirs:
- client
go: '1.19'
go: '1.21'
build-tags:
- e2e

Expand Down Expand Up @@ -88,3 +88,17 @@ linters-settings:
lll:
# FIXME: we should be able to reduce the length of lines to be closer to the default 120 lines
line-length: 300 # default: 120
depguard:
rules:
main:
files:
- $all
deny:
- pkg: "forbiddenpackage"
desc: Please don't use forbiddenpackage package

issues:
exclude-rules:
- path: '(.+)_test\.go'
linters:
- goconst
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Build the manager binary
FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.19 as builder
FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.21 as builder

ARG BUILDPLATFORM
ARG TARGETPLATFORM
Expand Down
26 changes: 20 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@ RELEASE_MSG ?= "koperator release"

REL_TAG = $(shell ./scripts/increment_version.sh -${RELEASE_TYPE} ${TAG})

GOLANGCI_VERSION = 1.51.2
LICENSEI_VERSION = 0.8.0
GOLANGCI_VERSION = 1.55.2
LICENSEI_VERSION = 0.9.0
GOPROXY=https://proxy.golang.org

CONTROLLER_GEN_VERSION = v0.9.2
CONTROLLER_GEN_VERSION = v0.13.0
CONTROLLER_GEN = $(PWD)/bin/controller-gen

ENVTEST_K8S_VERSION = 1.24.x!
ENVTEST_K8S_VERSION = 1.27.x!

KUSTOMIZE_BASE = config/overlays/specific-manager-version

Expand Down Expand Up @@ -102,7 +102,7 @@ test: generate fmt vet bin/setup-envtest
test-e2e:
IMG_E2E=${IMG_E2E} go test github.com/banzaicloud/koperator/tests/e2e \
-v \
-timeout 20m \
-timeout 45m \
-tags e2e \
--ginkgo.show-node-events \
--ginkgo.trace \
Expand Down Expand Up @@ -156,6 +156,20 @@ docker-build: ## Build the operator docker image.
docker-push: ## Push the operator docker image.
docker push ${IMG}

# PLATFORMS defines the target platforms for the manager image be built to provide support to multiple
# architectures. (i.e. make docker-buildx IMG=myregistry/mypoperator:0.0.1). To use this option you need to:
# - be able to use docker buildx. More info: https://docs.docker.com/build/buildx/
# - have enabled BuildKit. More info: https://docs.docker.com/develop/develop-images/build_enhancements/
# - be able to push the image to your registry (i.e. if you do not set a valid value via IMG=<myregistry/image:<tag>> then the export will fail)
# To adequately provide solutions that are compatible with multiple platforms, you should consider using this option.
PLATFORMS ?= linux/arm64,linux/amd64
.PHONY: docker-buildx
docker-buildx: ## Build and push docker image for the manager for cross-platform support
- docker buildx create --name koperator-builder
docker buildx use koperator-builder
docker buildx build --push --platform=$(PLATFORMS) --tag ${IMG} -f Dockerfile .
- docker buildx rm koperator-builder

bin/controller-gen: bin/controller-gen-$(CONTROLLER_GEN_VERSION) ## Symlink controller-gen-<version> into versionless controller-gen.
@ln -sf controller-gen-$(CONTROLLER_GEN_VERSION) bin/controller-gen

Expand Down Expand Up @@ -224,7 +238,7 @@ license-header-fix: gen-license-header bin/addlicense ## Fix missing license hea
$(ADDLICENSE_OPTS_IGNORE) \
$(ADDLICENSE_SOURCE_DIRS)

GOTEMPLATE_VERSION := 3.7.3
GOTEMPLATE_VERSION := 3.7.5

bin/gotemplate: $(BIN_DIR)/gotemplate-$(GOTEMPLATE_VERSION) ## Symlink gotemplate-<version> into versionless gotemplate.
@ln -sf gotemplate-$(GOTEMPLATE_VERSION) $(BIN_DIR)/gotemplate
Expand Down
46 changes: 30 additions & 16 deletions api/go.mod
Original file line number Diff line number Diff line change
@@ -1,36 +1,50 @@
module github.com/banzaicloud/koperator/api

go 1.19
go 1.21

require (
dario.cat/mergo v1.0.0
emperror.dev/errors v0.8.1
github.com/banzaicloud/istio-client-go v0.0.17
github.com/cert-manager/cert-manager v1.11.2
github.com/imdario/mergo v0.3.13
golang.org/x/exp v0.0.0-20230713183714-613f0c0eb8a1
github.com/cert-manager/cert-manager v1.13.2
// github.com/stretchr/testify v1.8.4
golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa
gotest.tools v2.2.0+incompatible
k8s.io/api v0.26.4
k8s.io/apimachinery v0.26.4
sigs.k8s.io/controller-runtime v0.14.6
k8s.io/api v0.28.4
k8s.io/apimachinery v0.28.4
sigs.k8s.io/controller-runtime v0.16.3
)

require (
github.com/go-logr/logr v1.2.3 // indirect
// github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/go-logr/logr v1.3.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.6.0 // indirect
golang.org/x/net v0.7.0 // indirect
golang.org/x/text v0.7.0 // indirect
// github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/net v0.18.0 // indirect
golang.org/x/text v0.14.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
k8s.io/klog/v2 v2.80.1 // indirect
k8s.io/utils v0.0.0-20221128185143-99ec85e7a448 // indirect
sigs.k8s.io/json v0.0.0-20220713155537-f223a00ba0e2 // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
// gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/klog/v2 v2.110.1 // indirect
k8s.io/utils v0.0.0-20230726121419-3b25d923346b // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
)

require github.com/stretchr/testify v1.8.4

require (
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

// remove once https://github.com/cert-manager/cert-manager/issues/5953 is fixed
replace github.com/Venafi/vcert/v4 => github.com/jetstack/vcert/v4 v4.9.6-0.20230127103832-3aa3dfd6613d
Loading
Loading