Redis issue on argocd-application-controller v2.13.0, chart 7.7.3

[davidt-gh]

Checklist:

• [ V ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ V ] I've included steps to reproduce the bug.
• [ V ] I've pasted the output of argocd version.

Describe the bug
The controller was unhealthy (the UI was stuck). Then looked the logs, saw some info logs, and after moment got a lot of logs (which in case I was able to catch)
Started with:

Reconnect to redis because error: \"dial tcp: lookup argocd-redis: i/o timeout\

1 second later the whole log (Which attached in here), but starting with:

fatal error: sync: unlock of unlocked mutex

To Reproduce
Changed argocd helm chart 7.6.12 -> 7.7.3

Expected behavior
Have small downtime, and keep working :)

Screenshots

Version

argocd: v2.12.6+4dab5bd
  BuildDate: 2024-10-18T21:32:12Z
  GitCommit: 4dab5bd6a60adea12e084ad23519e35b710060a2
  GitTreeState: clean
  GoVersion: go1.23.2
  Compiler: gc
  Platform: darwin/arm64
WARN[0001] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
argocd-server: v2.13.0+347f221

Logs

Paste any relevant application logs here.

application-controller.log

[andrii-korotkov-verkada]

Looks like an issue with Redis library. They did some changes on master for the relevant code https://github.com/redis/go-redis/blob/master/redis.go#L178-L180, redis/go-redis@080e051. Let's wait for a release of 9.7.1 or 9.8.0 and upgrade.

[andrii-korotkov-verkada]

I still don't understand how it becomes unlocked.

[andrii-korotkov-verkada]

Can you file an issue for Redis and share a part of logs and this issue, please? This may help to get cherry pick into 9.7.1 earlier.

[davidt-gh]

Can you file an issue for Redis and share a part of logs and this issue, please? This may help to get cherry pick into 9.7.1 earlier.

redis/go-redis#3195 - please add comment or context if needed.

[ftmiro]

I'm not sure if this issue is limited to the application-controller. I run 6 replicas of argocd-repo-server and one specific replica restarted 3 times in the last 5 days with:

Reconnect to redis because error: "dial tcp: lookup argocd-redis-ha-haproxy: i/o timeout"
...
fatal error: sync: unlock of unlocked mutex

We run argocd 2.13.3 and redis: 770913835712.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/docker/library/redis:7.0.15-alpine

This the only component running into this issue. All other services, like controller, server, redis-ha-proxy, redis-server are stable.

[vrivellino]

@ftmiro: We're seeing this same issue in the argocd-repo-server deployment of one of our clusters. We're running v2.13.4

UPDATE: Our issue was ultimately related to coredns being overloaded from a misbehaving app

Looks like an issue with Redis library. They did some changes on master for the relevant code https://github.com/redis/go-redis/blob/master/redis.go#L178-L180, redis/go-redis@080e051. Let's wait for a release of 9.7.1 or 9.8.0 and upgrade.

@andrii-korotkov-verkada: Unless I'm misunderstanding, 2.13.x of argo-cd does use a version of go-redis that includes that change:

https://github.com/argoproj/argo-cd/blob/v2.13.4/go.mod#L71

https://github.com/redis/go-redis/blob/v9.6.1/redis.go#L178-L181

So I think that change is a red herring

[prajwalv]

We are also running into the same issue in one of our production instances, but seems to be affecting only instance (out of 13).

We patched the latest(v2.13.4) argocd version couple days back (but was seeing this issue before we patched to latest).

Any update on this would be greatly appreciated.