agones-{extensions,allocator}: Make servers context aware #3845

zmerlynn · 2024-05-31T13:41:30Z

adds an httpserver utility package to handle the Run function that controller/extensions use. Make that context aware using the same method as https.Run:

agones/pkg/util/https/server.go

Lines 127 to 130 in dfa414e

    
           go func() { 
        
           	<-ctx.Done() 
        
           	s.tls.Close() // nolint: errcheck,gosec 
        
           }()

also plumbs context-awareness through the allocator run{Mux,REST,GRPC} functions.
adds a gRPC health server to the allocator, calls .Shutdown() on it during graceful termination - this seems to push the client off correctly.

Along the way, basically re-wrote the TestAllocatorAfterDeleteReplica test:

moved the GetAllocatorClient up because it's fairly disruptive (new certs, etc.), so I wanted the pod stability checks to happen after
changed agones-allocator stability checks to just grab the Deployment and wait for Replicas != AvailableReplicas.
changed the test to actually disrupt the Pods in the middle of the test (vs before it just nukes the pods then starts the test.

Tested with e2e in a loop.

Towards #3853

agones-bot · 2024-05-31T13:50:00Z

Build Failed 😱

Build Id: f71cd44c-de3c-458c-9dd6-aa9f3dd9b3ee

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

zmerlynn · 2024-05-31T16:24:44Z

Hmm, upon further testing this still has a minor flake. Let me play with it a bit more, moving to draft.

agones-bot · 2024-06-05T18:52:36Z

Build Failed 😱

Build Id: 6355c010-ac46-4c04-92cf-5fee63fbdc1f

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2024-06-05T20:13:36Z

Build Succeeded 👏

Build Id: cb48cb0a-06e7-485a-87ca-f0a79097eb43

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

https://2d2e2e5-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3845/head:pr_3845 && git checkout pr_3845
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.42.0-dev-2d2e2e5-amd64

markmandel · 2024-06-07T23:47:09Z

cmd/extensions/main.go

@@ -150,7 +150,7 @@ func main() {
 	agonesInformerFactory := externalversions.NewSharedInformerFactory(agonesClient, defaultResync)
 	kubeInformerFactory := informers.NewSharedInformerFactory(kubeClient, defaultResync)

-	server := &httpServer{}
+	server := &httpserver.Server{Logger: logger}


Suggested change

server := &httpserver.Server{Logger: logger}

server := &http.Server{Logger: logger}

That wouldn't be redundant 😃

cmd/allocator/main.go

markmandel · 2024-06-07T23:52:58Z

pkg/util/httpserver/server.go

+
+// Package httpserver implements an http server that conforms to the
+// controller runner interface.
+package httpserver


Suggested change

package httpserver

package http

I'd be tempted to make the package http since we also have a https - not sure the server part is required.

I really didn't want to shadow a super commonly important base library. Calling this http would result in it getting imported as agoneshttp or something pretty often, since I would also probably not rename base http.

agones-bot · 2024-06-11T17:03:23Z

Build Succeeded 👏

Build Id: ec4f214a-d587-406d-8ee0-20be8a5a2b15

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

https://acc03f2-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3845/head:pr_3845 && git checkout pr_3845
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.42.0-dev-acc03f2-amd64

markmandel · 2024-06-13T21:56:04Z

cmd/allocator/main.go


 	go func() {
+		go func() {
+			<-ctx.Done()
+			grpcServer.Stop()


Should this be: https://pkg.go.dev/google.golang.org/grpc#Server.GracefulStop (or maybe used in lieu of a lot of this code?)

Which might help with your multi-context issue?

I went back and forth on this, and I have a version in a branch where I did exactly that. It made things worse.

Let me just go back to the drawing board here, but I think it may involve rewriting a lot of the handling basically everywhere.

But no, it doesn't solve the two-context issue. The problem is that here (and in httpserver/https), we use a context to signal shutdown, but the service handler still needs a forward context to get work done to shut down. Let me see if I can just rework the listen handlers more.

Yeah I see your point. I think you are right that we'll want two contexts, but what we could do is something akin to:

<- listenerCtx.Done() // graceful shutdown both the http and grpc listeners. grpcServer.GracefulStop() server.Shutdown() cancelInfrormer() // cancel the worker + Informer context, since now we know we aren't accepting any more connections.

This might require pulling the grpc + http listeners into a larger struct that has a Run(ctx) operation on it, and stops when the ctx is <- ctx.Done(), to provide a larger data structure to operate on (and then it could take a context, and manage it's own context for the workers and informers).

Did that make sense?

Hmm, we could do something similar to the runner slices and have shutdown slices, perhaps?

Hmm, we could do something similar to the runner slices and have shutdown slices, perhaps?

Seems like you are rebuilding Contexts 😄

cmd/allocator/main.go

…ealth check * adds an `httpserver` utility package to handle the `Run` function that controller/extensions use. Make that context aware using the same method as https.Run: https://github.com/googleforgames/agones/blob/dfa414e5e4da37798833bbf8c33919acb5f3c2ea/pkg/util/https/server.go#L127-L130 * also plumbs context-awareness through the allocator run{Mux,REST,GRPC} functions. * adds a gRPC health server to the allocator, calls .Shutdown() on it during graceful termination - this seems to push the client off correctly. Tested with e2e in a loop. Towards googleforgames#3853

…AllocatorClient

agones-bot · 2024-06-26T01:24:36Z

Build Succeeded 👏

Build Id: 4e70c09c-b912-48d4-91fb-6af56029445b

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

https://9d246a3-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3845/head:pr_3845 && git checkout pr_3845
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.42.0-dev-9d246a3-amd64

markmandel · 2024-07-11T00:07:51Z

cmd/allocator/main.go

+	// and a cascading graceful shutdown instead of multiple contexts and sleeps.
+	<-listenCtx.Done()
+	logger.Infof("Listen context cancelled")
+	time.Sleep(5 * time.Second)


I'm assuming this is because we don't actually know that the gRPC/HTTP servers have gracefully shut down yet, so we just wait for a bit?

Yeah .. fully knowing they're done takes a bit more plumbing.

agones-bot · 2024-07-11T01:15:52Z

Build Succeeded 👏

Build Id: 6e64e3f3-f0bb-4499-8566-4b744130ea83

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

https://2f06434-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3845/head:pr_3845 && git checkout pr_3845
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.42.0-dev-2f06434-amd64

markmandel

Good enough 😁 thanks for all the new plumbing!

agones-bot · 2024-07-12T00:47:50Z

Build Succeeded 👏

Build Id: 841fae19-753d-41d2-a8c5-4b3b68f7ad80

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

https://1006b0b-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3845/head:pr_3845 && git checkout pr_3845
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.42.0-dev-1006b0b-amd64

…ames#3845) * agones-{extensions,allocator}: Make servers context aware, add gRPC health check * adds an `httpserver` utility package to handle the `Run` function that controller/extensions use. Make that context aware using the same method as https.Run: https://github.com/googleforgames/agones/blob/dfa414e5e4da37798833bbf8c33919acb5f3c2ea/pkg/util/https/server.go#L127-L130 * also plumbs context-awareness through the allocator run{Mux,REST,GRPC} functions. * adds a gRPC health server to the allocator, calls .Shutdown() on it during graceful termination - this seems to push the client off correctly. Tested with e2e in a loop. Towards googleforgames#3853 * Move from context.Background() * Use Shutdown/GracefulStop * Relax deadline slightly (original had none), also delete pod from GetAllocatorClient

… apis (googleforgames#3900) * Adds a command to generate the zz_generated.deepcopy.go files for the apis * Combines CRD client and deepcopy code generation into one script * Updates boilerplate headers to 2024 * Generated code from gen-all-sdk-grpc * Small update to template comment agones-{extensions,allocator}: Make servers context aware (googleforgames#3845) * agones-{extensions,allocator}: Make servers context aware, add gRPC health check * adds an `httpserver` utility package to handle the `Run` function that controller/extensions use. Make that context aware using the same method as https.Run: https://github.com/googleforgames/agones/blob/dfa414e5e4da37798833bbf8c33919acb5f3c2ea/pkg/util/https/server.go#L127-L130 * also plumbs context-awareness through the allocator run{Mux,REST,GRPC} functions. * adds a gRPC health server to the allocator, calls .Shutdown() on it during graceful termination - this seems to push the client off correctly. Tested with e2e in a loop. Towards googleforgames#3853 * Move from context.Background() * Use Shutdown/GracefulStop * Relax deadline slightly (original had none), also delete pod from GetAllocatorClient Update Slack invite link (googleforgames#3896) Looks like the Slack invite link expired somewhere along the way, so this is a new, shiny updated one!

zmerlynn assigned markmandel May 31, 2024

github-actions bot added kind/other size/M labels May 31, 2024

zmerlynn marked this pull request as draft May 31, 2024 16:25

zmerlynn force-pushed the close-srv branch from 9387e7b to 1775f9e Compare June 5, 2024 17:52

github-actions bot added the size/L label Jun 5, 2024

zmerlynn force-pushed the close-srv branch from 1775f9e to 2d2e2e5 Compare June 5, 2024 17:54

zmerlynn marked this pull request as ready for review June 5, 2024 18:05

markmandel reviewed Jun 7, 2024

View reviewed changes

zmerlynn force-pushed the close-srv branch from 2d2e2e5 to acc03f2 Compare June 11, 2024 15:52

markmandel reviewed Jun 13, 2024

View reviewed changes

zmerlynn added 4 commits June 26, 2024 00:22

Move from context.Background()

d3d5fcb

Use Shutdown/GracefulStop

a349058

Relax deadline slightly (original had none), also delete pod from Get…

9d246a3

…AllocatorClient

zmerlynn force-pushed the close-srv branch from acc03f2 to 9d246a3 Compare June 26, 2024 00:22

markmandel reviewed Jul 11, 2024

View reviewed changes

Merge branch 'main' into close-srv

2f06434

markmandel approved these changes Jul 11, 2024

View reviewed changes

markmandel enabled auto-merge (squash) July 11, 2024 23:40

Merge branch 'main' into close-srv

1006b0b

markmandel merged commit a74a86f into googleforgames:main Jul 12, 2024
4 checks passed

kamaljeeti added kind/bug These are bugs. and removed kind/other labels Jul 15, 2024

This was referenced Jul 25, 2024

Allocator metrics missing in Agones 1.42.0 #3920

Closed

Fix allocator metrics endpoint #3921

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

agones-{extensions,allocator}: Make servers context aware #3845

agones-{extensions,allocator}: Make servers context aware #3845

zmerlynn commented May 31, 2024 •

edited

Loading

agones-bot commented May 31, 2024

zmerlynn commented May 31, 2024

agones-bot commented Jun 5, 2024

agones-bot commented Jun 5, 2024

markmandel Jun 7, 2024

markmandel Jun 7, 2024

zmerlynn Jun 10, 2024

agones-bot commented Jun 11, 2024

markmandel Jun 13, 2024

zmerlynn Jun 14, 2024

zmerlynn Jun 14, 2024

markmandel Jun 14, 2024 •

edited

Loading

zmerlynn Jun 14, 2024

markmandel Jun 14, 2024

agones-bot commented Jun 26, 2024

markmandel Jul 11, 2024

zmerlynn Jul 11, 2024

agones-bot commented Jul 11, 2024

markmandel left a comment

agones-bot commented Jul 12, 2024

	go func() {
	<-ctx.Done()
	s.tls.Close() // nolint: errcheck,gosec
	}()

	server := &httpserver.Server{Logger: logger}
	server := &http.Server{Logger: logger}

agones-{extensions,allocator}: Make servers context aware #3845

agones-{extensions,allocator}: Make servers context aware #3845

Conversation

zmerlynn commented May 31, 2024 • edited Loading

agones-bot commented May 31, 2024

zmerlynn commented May 31, 2024

agones-bot commented Jun 5, 2024

agones-bot commented Jun 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agones-bot commented Jun 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markmandel Jun 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agones-bot commented Jun 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agones-bot commented Jul 11, 2024

markmandel left a comment

Choose a reason for hiding this comment

agones-bot commented Jul 12, 2024

zmerlynn commented May 31, 2024 •

edited

Loading

markmandel Jun 14, 2024 •

edited

Loading