Changed the sidecar requests rate limiter from exponential to a constant one #4186

miai10 · 2025-05-23T12:06:19Z

What type of PR is this?
/kind bugfix

What this PR does / Why we need it:

During high allocation rate tests, I noticed that the sessions list is updated very very slow (at 15 minutes interval sometimes). The lists were updated by the game server logic following the start or termination of allocated sessions. A high failure rate of PATCH requests made the retry interval to hit the 1000 seconds limit which is very high (there is a nice article here regarding how rate limiters work: https://danielmangum.com/posts/controller-runtime-client-go-rate-limiting/#the-default-controller-rate-limiter). The Agones controllers use a fast/slow rate limiter but for some reason that I am missing the sidecar is using the default one.

Using a constant rate limiter gave better results: the sessions are updated and there is no storm of PATCH failures. 500ms are chosen to match the allocator batch wait time, the logic being that the allocator pushes allocations, the game server acknowledges somehow. This value can be configured from the controller helm values as I think it is related to the system performance. For example, my tests were better with 1 second.

Note: the one per 500 ms request is actually for each type of update that the sidecar implements: state, labels, annotations, player capacity, connected players, counter and lists. So in theory there could be up to 7 requests per 500 ms coming from the sidecar.

Which issue(s) this PR fixes:

The issue is not open yet, it is described above.

Special notes for your reviewer:

markmandel · 2025-05-24T20:50:08Z

NIce one.

The Agones controllers use a fast/slow rate limiter but for some reason that I am missing the sidecar is using the default one.

Any reason not to use this for the sidecar as well - keep everything consistent?

/gcbrun

agones-bot · 2025-05-24T20:59:19Z

Build Failed 😭

Build Id: e6f929b1-b836-4e2f-b797-6410cfd494c8

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2025-05-24T22:14:49Z

make[1]: Leaving directory '/workspace/build'
sort /workspace/install/yaml/install.yaml > /tmp/agones-install/install.current.yaml.sorted
diff /tmp/agones-install/install.yaml.sorted /tmp/agones-install/install.current.yaml.sorted
17367a17368
>           value: "1s"
17520a17522
>         - name: SIDECAR_REQUESTS_RATE_LIMIT
make: *** [Makefile:420: test-install-yaml] Error 1

Aah, need to regenerate the index.yaml install script make gen-install from memory.

miai10 · 2025-05-26T08:13:11Z

NIce one.

The Agones controllers use a fast/slow rate limiter but for some reason that I am missing the sidecar is using the default one.

Any reason not to use this for the sidecar as well - keep everything consistent?

/gcbrun

I think it is more important to be consistent with the allocator. My view of the problem is that the allocator updates the crd with new allocations, the gs detects those allocations (eventually changes lists and counters as acknowledgement) and updates again the CRD when sessions are finished. If the sidecar starts retries fast, it looks like it has priority thus generating update failures on the allocator side.

In one server one session case there should not be problems as there should not be reasons for retries.

If you want, I can expose the fast/slow rate limiter settings but it seems overkill to have 3 settings exposed :P.

markmandel · 2025-05-26T18:17:30Z

That makes sense. SGTM.

markmandel · 2025-05-26T18:17:36Z

/gcbrun

agones-bot · 2025-05-26T18:27:31Z

Build Failed 😭

Build Id: adb6713e-afb0-4d80-9288-233c448d2fff

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

0xaravindh · 2025-05-30T05:07:02Z

/gcbrun

…ant one

miai10 · 2025-05-30T09:55:31Z

I rebased onto latest, do I need to do something else to get the change merged?

markmandel · 2025-05-31T15:14:04Z

/gcbrun

markmandel · 2025-05-31T15:16:20Z

Assuming all passes (and someone should step through the code, I haven't done that yet, I usually wait for CI to pass), we'll also need an update to the Helm docs please.

https://agones.dev/site/docs/installation/install-agones/helm/#configuration

As a note - the new feature will need to be wrapped in a feature shortcode so it only publishes with the new version. You can find details here: https://agones.dev/site/docs/contribute/documentation-editing-contribution/#at-the-page-level

agones-bot · 2025-05-31T15:55:16Z

Build Failed 😭

Build Id: f2d581e5-14e5-45ff-9c66-9f93df959c8d

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2025-05-31T17:26:10Z

Flake: Step 28: submit-e2e-test-cloud-build

markmandel · 2025-05-31T17:26:24Z

/gcbrun

agones-bot · 2025-05-31T17:38:51Z

Build Failed 😭

Build Id: c9d178cb-6fa8-4c75-b1db-65fe47c0de91

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2025-06-01T00:46:59Z

ERROR: failed to solve: failed to compute cache key: failed to copy: httpReadSeeker: failed open: unexpected status code https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:cb524f6f22159378ea820d234d80ca09b79c2f0cc91315eeef11904e3ff36a21: 503 Service Unavailable
make: *** [Makefile:625: build-agones-sdk-image-windows-ltsc2019] Error 1
make: *** Waiting for unfinished jobs....
#5 extracting sha256:8edf7e005a035a07159bee8c941c629808c8de251a7a9e5e424a2648cf918299 56.4s done

Fun.

markmandel · 2025-06-01T22:05:57Z

flaaaakinesss....

markmandel · 2025-06-02T03:56:53Z

There we go - CI passed 👍🏻 time for a code review.

markmandel

Code LGTM! Just need those docs updates, and this should be good to merge!

0xaravindh · 2025-06-16T07:07:40Z

/gcbrun

agones-bot · 2025-06-16T07:41:53Z

Build Failed 😭

Build Id: 9e78dba1-5f6a-4494-8883-a66afe02e38b

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2025-06-16T08:40:42Z

Build Failed 😭

Build Id: 018e2fd7-d946-489e-aa7e-6cb5a5303bd5

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

0xaravindh · 2025-06-16T14:39:27Z

/gcbrun

agones-bot · 2025-06-16T15:38:28Z

Build Succeeded 🥳

Build Id: d24c5eb8-cb62-4e0b-854c-06801751d8bf

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.50.0-dev-990f698
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.50.0-dev-990f698
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.50.0-dev-990f698-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.50.0-dev-990f698
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.50.0-dev-990f698
Linux C++ SDK (build): agonessdk-1.50.0-dev-990f698-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.50.0-dev-990f698.zip

A preview of the website (the last 30 builds are retained):

https://990f698-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4186/head:pr_4186 && git checkout pr_4186
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.50.0-dev-990f698

miai10 · 2025-06-17T15:53:49Z

Assuming all passes (and someone should step through the code, I haven't done that yet, I usually wait for CI to pass), we'll also need an update to the Helm docs please.

https://agones.dev/site/docs/installation/install-agones/helm/#configuration

As a note - the new feature will need to be wrapped in a feature shortcode so it only publishes with the new version. You can find details here: https://agones.dev/site/docs/contribute/documentation-editing-contribution/#at-the-page-level

Sorry for the late reply, I was in vacation...

I did the change in the helm.md

markmandel · 2025-06-18T05:01:46Z

Looks like 1.50 is underway, so will come back around once the release is done 👍🏻

site/content/en/docs/Installation/Install Agones/helm.md

igooch · 2025-06-27T18:12:53Z

/gcbrun

agones-bot · 2025-06-27T19:15:16Z

Build Succeeded 🥳

Build Id: e856e041-81ce-41b4-81d0-0903417f202f

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.51.0-dev-bef214c
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.51.0-dev-bef214c
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.51.0-dev-bef214c-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.51.0-dev-bef214c
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.51.0-dev-bef214c
Linux C++ SDK (build): agonessdk-1.51.0-dev-bef214c-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.51.0-dev-bef214c.zip

A preview of the website (the last 30 builds are retained):

https://bef214c-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4186/head:pr_4186 && git checkout pr_4186
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.51.0-dev-bef214c

markmandel

LGTM!

If you fix the merge conflicts please, and we are good to merge!

# Conflicts: # pkg/gameservers/controller.go

miai10 · 2025-07-02T07:02:36Z

Kind reminder to merge this until it gets conflicted again :P

0xaravindh · 2025-07-02T07:11:46Z

/gcbrun

agones-bot · 2025-07-02T08:11:18Z

Build Succeeded 🥳

Build Id: 9436ba9d-cc17-411d-8433-076bc039e6f6

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.51.0-dev-c747b0d
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.51.0-dev-c747b0d
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.51.0-dev-c747b0d-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.51.0-dev-c747b0d
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.51.0-dev-c747b0d
Linux C++ SDK (build): agonessdk-1.51.0-dev-c747b0d-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.51.0-dev-c747b0d.zip

A preview of the website (the last 30 builds are retained):

https://c747b0d-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4186/head:pr_4186 && git checkout pr_4186
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.51.0-dev-c747b0d

github-actions bot added the kind/bug These are bugs. label May 23, 2025

github-actions bot added kind/other size/M labels May 26, 2025

miai10 added 2 commits May 30, 2025 12:47

Changed the sidecar requests rate limiter from exponential to a const…

6676944

…ant one

Generate install.yaml

87f4878

miai10 force-pushed the perf/sidecar-patches branch from 8bcac9b to 87f4878 Compare May 30, 2025 09:54

Merge branch 'main' into perf/sidecar-patches

6297175

markmandel reviewed Jun 3, 2025

View reviewed changes

Merge branch 'main' into perf/sidecar-patches

990f698

Update the helm.md

32dbde6

markmandel reviewed Jun 18, 2025

View reviewed changes

site/content/en/docs/Installation/Install Agones/helm.md Show resolved Hide resolved

Using sdkServer instead of sdk for nesting

ffffc1f

markmandel reviewed Jun 22, 2025

View reviewed changes

site/content/en/docs/Installation/Install Agones/helm.md Outdated Show resolved Hide resolved

miai10 and others added 2 commits June 25, 2025 12:16

Using sdkServer outside of the controller nesting

e53de26

Merge branch 'main' into perf/sidecar-patches

bef214c

markmandel approved these changes Jun 28, 2025

View reviewed changes

Merge branch 'main' into perf/sidecar-patches

964e3a2

github-actions bot added the size/L label Jun 30, 2025

Merge remote-tracking branch 'official/main' into perf/sidecar-patches

c747b0d

# Conflicts: # pkg/gameservers/controller.go

miai10 mentioned this pull request Jun 30, 2025

[Feature Proposal] - Remove the 1000 entries limit for lists #4192

Open

markmandel merged commit 1865ad2 into googleforgames:main Jul 2, 2025
4 checks passed

miai10 deleted the perf/sidecar-patches branch July 4, 2025 07:42

Changed the sidecar requests rate limiter from exponential to a constant one #4186

Changed the sidecar requests rate limiter from exponential to a constant one #4186

Uh oh!

Conversation

miai10 commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markmandel commented May 24, 2025

Uh oh!

agones-bot commented May 24, 2025

Uh oh!

markmandel commented May 24, 2025

Uh oh!

miai10 commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markmandel commented May 26, 2025

Uh oh!

markmandel commented May 26, 2025

Uh oh!

agones-bot commented May 26, 2025

Uh oh!

0xaravindh commented May 30, 2025

Uh oh!

miai10 commented May 30, 2025

Uh oh!

markmandel commented May 31, 2025

Uh oh!

markmandel commented May 31, 2025

Uh oh!

agones-bot commented May 31, 2025

Uh oh!

markmandel commented May 31, 2025

Uh oh!

markmandel commented May 31, 2025

Uh oh!

agones-bot commented May 31, 2025

Uh oh!

markmandel commented Jun 1, 2025

Uh oh!

markmandel commented Jun 1, 2025

Uh oh!

markmandel commented Jun 2, 2025

Uh oh!

markmandel left a comment

Choose a reason for hiding this comment

Uh oh!

0xaravindh commented Jun 16, 2025

Uh oh!

agones-bot commented Jun 16, 2025

Uh oh!

agones-bot commented Jun 16, 2025

Uh oh!

0xaravindh commented Jun 16, 2025

Uh oh!

agones-bot commented Jun 16, 2025

Uh oh!

miai10 commented Jun 17, 2025

Uh oh!

markmandel commented Jun 18, 2025

Uh oh!

Uh oh!

Uh oh!

igooch commented Jun 27, 2025

Uh oh!

agones-bot commented Jun 27, 2025

Uh oh!

markmandel left a comment

Choose a reason for hiding this comment

Uh oh!

miai10 commented Jul 2, 2025

Uh oh!

0xaravindh commented Jul 2, 2025

Uh oh!

agones-bot commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

miai10 commented May 23, 2025 •

edited

Loading

miai10 commented May 26, 2025 •

edited

Loading