[chart/redis-ha][BUG] The cluster became read only #384
Description
Describe the bug
The cluster was functioning well for a few days. Then out of the blue the connection to the cluster service became read only without any noticeable reason. I have checked connection to these services:
- qa-launch-cluster-common-redis-ha; # <-- btw, is this the right service to use?
- qa-launch-cluster-common-redis-ha-haproxy.
Restarting the redis-ha-server statefulset has "fixed" the issue, the cluster service became read-write.
To Reproduce
I can't figure out how to reproduce the issue.
Expected behavior
The cluster should function properly, the cluster service should provide read-write access.
Additional context
Chart version: 4.35.3 on a kubernetes v1.33.6+k3s1 cluster of 3 control and 5 worker nodes.
There are 2 redis-ha clusters deployed in the same namespace: dev and qa. Both became read-only.
Pods list in the cluster namespace:
$ kubectl -n redis get pod
NAME READY STATUS RESTARTS AGE
dev-launch-cluster-common-redis-ha-haproxy-dccb8d799-fmsnd 1/1 Running 0 11d
dev-launch-cluster-common-redis-ha-haproxy-dccb8d799-pwq7m 1/1 Running 0 11d
dev-launch-cluster-common-redis-ha-haproxy-dccb8d799-wfn24 1/1 Running 0 11d
dev-launch-cluster-common-redis-ha-server-0 4/4 Running 0 17m
dev-launch-cluster-common-redis-ha-server-1 4/4 Running 0 19m
dev-launch-cluster-common-redis-ha-server-2 4/4 Running 0 20m
qa-launch-cluster-common-redis-ha-haproxy-6d566d7b9c-czshj 1/1 Running 0 11d
qa-launch-cluster-common-redis-ha-haproxy-6d566d7b9c-lc5pt 1/1 Running 0 11d
qa-launch-cluster-common-redis-ha-haproxy-6d566d7b9c-md7kd 1/1 Running 0 11d
qa-launch-cluster-common-redis-ha-server-0 4/4 Running 0 11d
qa-launch-cluster-common-redis-ha-server-1 4/4 Running 1 (8d ago) 11d
qa-launch-cluster-common-redis-ha-server-2 4/4 Running 0 11d
redisinsight-0 1/1 Running 0 11dCluster pod logs:
$ kubectl -n redis logs -f -l release=qa-launch-cluster-common --all-containers --max-log-requests 30
10.43.247.211 qa-launch-cluster-common-redis-ha-announce-0.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-0.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-0
10.43.38.202 qa-launch-cluster-common-redis-ha-announce-1.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-1.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-1
10.43.131.245 qa-launch-cluster-common-redis-ha-announce-2.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-2.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-2
1:S 09 Feb 2026 07:18:36.057 * 1 changes in 900 seconds. Saving...
1:S 09 Feb 2026 07:18:36.058 * Background saving started by pid 1312873
[WARNING] (8) : Server check_if_redis_is_master_0/R0 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.43.247.211')", check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] (8) : backend 'check_if_redis_is_master_0' has no server available!
10.43.247.211 qa-launch-cluster-common-redis-ha-announce-0.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-0.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-0
10.43.38.202 qa-launch-cluster-common-redis-ha-announce-1.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-1.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-1
10.43.131.245 qa-launch-cluster-common-redis-ha-announce-2.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-2.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-2
time="2026-01-28T12:49:40Z" level=info msg="Redis Metrics Exporter v1.67.0,SHA1=e2bb7fd6af3b950efa3267e4b932531098dc06b0 build date: sha1: [no-sha] Go: go1.23.4 GOOS: linux GOARCH: amd64"
time="2026-01-28T12:49:40Z" level=info msg="Providing metrics at :9121/metrics"
1312873:C 09 Feb 2026 07:18:36.098 * DB saved on disk
1312873:C 09 Feb 2026 07:18:36.099 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:S 09 Feb 2026 07:18:36.159 * Background saving terminated with success
1:S 09 Feb 2026 07:33:37.063 * 1 changes in 900 seconds. Saving...
1:S 09 Feb 2026 07:33:37.064 * Background saving started by pid 1314135
1314135:C 09 Feb 2026 07:33:37.099 * DB saved on disk
1314135:C 09 Feb 2026 07:33:37.100 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:S 09 Feb 2026 07:33:37.165 * Background saving terminated with success
1:X 04 Feb 2026 03:53:49.930 # +sdown slave 10.43.247.211:6379 10.43.247.211 6379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:53:49.930 # +sdown slave 10.43.38.202:6379 10.43.38.202 6379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:53:49.930 # +sdown sentinel e0e25ffa42a5c3eaab26cf4da3727ac257ddbc9c 10.43.247.211 26379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:53:49.930 # +sdown sentinel ae3f28e424fcf389aa2a55e805005be431280a71 10.43.38.202 26379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:55:02.600 # -sdown slave 10.43.247.211:6379 10.43.247.211 6379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:55:02.600 # -sdown slave 10.43.38.202:6379 10.43.38.202 6379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:55:02.600 # -sdown sentinel e0e25ffa42a5c3eaab26cf4da3727ac257ddbc9c 10.43.247.211 26379 @ mymaster 10.43.131.245 6379
Updating redis config..
we are slave of redis master (10.43.247.211:6379)
Updating sentinel config..
evaluating sentinel id (${SENTINEL_ID_0})
sentinel id (e0e25ffa42a5c3eaab26cf4da3727ac257ddbc9c), sentinel grp (mymaster), quorum (2)
redis master (10.43.247.211:6379)
announce (10.43.247.211:26379)
make sure 10.43.247.211 is not a slave (slaveof no one)
Setting redis auth values..
Wed Jan 28 12:48:59 UTC 2026 Ready...
[WARNING] (8) : Server bk_redis_master/R0 is DOWN, reason: Layer7 timeout, info: " at step 7 of tcp-check (expect string 'role:master')", check duration: 2003ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server bk_redis_master/R2 is DOWN, reason: Layer4 timeout, info: " at step 1 of tcp-check (connect)", check duration: 1001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] (8) : backend 'bk_redis_master' has no server available!
time="2026-01-31T21:21:50Z" level=info msg="Redis Metrics Exporter v1.67.0,SHA1=e2bb7fd6af3b950efa3267e4b932531098dc06b0 build date: sha1: [no-sha] Go: go1.23.4 GOOS: linux GOARCH: amd64"
time="2026-01-31T21:21:51Z" level=info msg="Providing metrics at :9121/metrics"
[WARNING] (8) : Server check_if_redis_is_master_0/R2 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.43.247.211')", check duration: 2002ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server check_if_redis_is_master_0/R1 is DOWN, reason: Layer7 timeout, info: " at step 5 of tcp-check (expect string '10.43.247.211')", check duration: 2002ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] (8) : backend 'check_if_redis_is_master_0' has no server available!
[WARNING] (8) : Server bk_redis_master/R0 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R0 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server bk_redis_master/R0 is DOWN, reason: Layer7 timeout, info: " at step 7 of tcp-check (expect string 'role:master')", check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R1 is DOWN, reason: Layer4 timeout, info: " at step 1 of tcp-check (connect)", check duration: 1001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R0 is DOWN, reason: Layer4 timeout, info: " at step 1 of tcp-check (connect)", check duration: 1002ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R1 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R0 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R2 is DOWN, reason: Layer4 timeout, info: " at step 1 of tcp-check (connect)", check duration: 1001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R1 is DOWN, reason: Layer4 timeout, info: " at step 1 of tcp-check (connect)", check duration: 1002ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R1 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server bk_redis_master/R2 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R2 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
time="2026-01-28T12:52:17Z" level=info msg="Redis Metrics Exporter v1.67.0,SHA1=e2bb7fd6af3b950efa3267e4b932531098dc06b0 build date: sha1: [no-sha] Go: go1.23.4 GOOS: linux GOARCH: amd64"
time="2026-01-28T12:52:17Z" level=info msg="Providing metrics at :9121/metrics"
1:X 04 Feb 2026 03:55:02.600 # -sdown sentinel ae3f28e424fcf389aa2a55e805005be431280a71 10.43.38.202 26379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:55:02.750 * Sentinel new configuration saved on disk
1:X 04 Feb 2026 03:55:02.750 # +new-epoch 2
1:M 09 Feb 2026 07:18:36.029 * 1 changes in 900 seconds. Saving...
1:M 09 Feb 2026 07:18:36.029 * Background saving started by pid 1338559
1338559:C 09 Feb 2026 07:18:36.060 * DB saved on disk
1338559:C 09 Feb 2026 07:18:36.061 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 09 Feb 2026 07:18:36.130 * Background saving terminated with success
1:M 09 Feb 2026 07:33:37.091 * 1 changes in 900 seconds. Saving...
1:M 09 Feb 2026 07:33:37.091 * Background saving started by pid 1339641
1339641:C 09 Feb 2026 07:33:37.126 * DB saved on disk
1339641:C 09 Feb 2026 07:33:37.127 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 09 Feb 2026 07:33:37.192 * Background saving terminated with success
[WARNING] (8) : Server bk_redis_master/R0 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R0 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server bk_redis_master/R0 is DOWN, reason: Layer7 timeout, info: " at step 7 of tcp-check (expect string 'role:master')", check duration: 2001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server bk_redis_master/R2 is DOWN, reason: Layer4 timeout, info: " at step 1 of tcp-check (connect)", check duration: 1001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] (8) : backend 'bk_redis_master' has no server available!
[WARNING] (8) : Server check_if_redis_is_master_2/R2 is DOWN, reason: Layer4 timeout, info: " at step 1 of tcp-check (connect)", check duration: 1002ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R0 is DOWN, reason: Layer4 timeout, info: " at step 1 of tcp-check (connect)", check duration: 1002ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R0 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server bk_redis_master/R2 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 1ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] (8) : Server check_if_redis_is_master_2/R2 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Mon Feb 9 07:41:45 UTC 2026 Found redis master (10.43.131.245)
Identifying redis master (get-master-addr-by-name)..
using sentinel (qa-launch-cluster-common-redis-ha), sentinel group name (mymaster)
Mon Feb 9 07:42:45 UTC 2026 Found redis master (10.43.131.245)
Identifying redis master (get-master-addr-by-name)..
using sentinel (qa-launch-cluster-common-redis-ha), sentinel group name (mymaster)
Mon Feb 9 07:43:45 UTC 2026 Found redis master (10.43.131.245)
Identifying redis master (get-master-addr-by-name)..
using sentinel (qa-launch-cluster-common-redis-ha), sentinel group name (mymaster)
Mon Feb 9 07:44:45 UTC 2026 Found redis master (10.43.131.245)
1:X 04 Feb 2026 03:54:41.716 * Sentinel new configuration saved on disk
1:X 04 Feb 2026 03:54:41.716 # +vote-for-leader ae3f28e424fcf389aa2a55e805005be431280a71 2
1:X 04 Feb 2026 03:54:41.716 # -sdown slave 10.43.247.211:6379 10.43.247.211 6379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:54:41.716 # -sdown sentinel e0e25ffa42a5c3eaab26cf4da3727ac257ddbc9c 10.43.247.211 26379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:54:41.722 * e0e25ffa42a5c3eaab26cf4da3727ac257ddbc9c voted for e0e25ffa42a5c3eaab26cf4da3727ac257ddbc9c 2
1:X 04 Feb 2026 03:54:52.551 # -failover-abort-not-elected master mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:54:52.653 * Next failover delay: I will not start a failover before Wed Feb 4 04:00:42 2026
1:X 04 Feb 2026 03:55:02.741 # -sdown master mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:55:02.741 # -odown master mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:55:02.813 # -sdown sentinel f3600ef2957066d701580da18db9b1ae90fac005 10.43.131.245 26379 @ mymaster 10.43.131.245 6379
I0209 10:44:56.968827 56226 request.go:697] Waited for 1.00028358s due to client-side throttling, not priority and fairness, request: GET:https://192.168.75.154:6443/api/v1/namespaces/redis/pods/qa-launch-cluster-common-redis-ha-haproxy-6d566d7b9c-lc5pt/log?container=config-init&follow=true&tailLines=10
10.43.247.211 qa-launch-cluster-common-redis-ha-announce-0.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-0.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-0
10.43.38.202 qa-launch-cluster-common-redis-ha-announce-1.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-1.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-1
10.43.131.245 qa-launch-cluster-common-redis-ha-announce-2.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-2.redis.svc.cluster.local qa-launch-cluster-common-redis-ha-announce-2
1:S 09 Feb 2026 07:18:36.041 * 1 changes in 900 seconds. Saving...
1:S 09 Feb 2026 07:18:36.042 * Background saving started by pid 1427669
1427669:C 09 Feb 2026 07:18:36.082 * DB saved on disk
1427669:C 09 Feb 2026 07:18:36.082 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:S 09 Feb 2026 07:18:36.143 * Background saving terminated with success
1:S 09 Feb 2026 07:33:37.031 * 1 changes in 900 seconds. Saving...
1:S 09 Feb 2026 07:33:37.032 * Background saving started by pid 1428932
1428932:C 09 Feb 2026 07:33:37.059 * DB saved on disk
1428932:C 09 Feb 2026 07:33:37.060 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:S 09 Feb 2026 07:33:37.133 * Background saving terminated with success
Updating redis and sentinel config..
Updating sentinel config..
evaluating sentinel id (${SENTINEL_ID_2})
sentinel id (f3600ef2957066d701580da18db9b1ae90fac005), sentinel grp (mymaster), quorum (2)
redis master (10.43.247.211:6379)
announce (10.43.131.245:26379)
Updating redis config..
we are slave of redis master (10.43.247.211:6379)
Setting redis auth values..
Wed Jan 28 12:52:06 UTC 2026 Ready...
Mon Feb 9 07:41:37 UTC 2026 Found redis master (10.43.131.245)
Identifying redis master (get-master-addr-by-name)..
using sentinel (qa-launch-cluster-common-redis-ha), sentinel group name (mymaster)
Mon Feb 9 07:42:37 UTC 2026 Found redis master (10.43.131.245)
Identifying redis master (get-master-addr-by-name)..
using sentinel (qa-launch-cluster-common-redis-ha), sentinel group name (mymaster)
Mon Feb 9 07:43:37 UTC 2026 Found redis master (10.43.131.245)
Identifying redis master (get-master-addr-by-name)..
using sentinel (qa-launch-cluster-common-redis-ha), sentinel group name (mymaster)
Mon Feb 9 07:44:37 UTC 2026 Found redis master (10.43.131.245)
1:X 04 Feb 2026 03:54:41.727 * Sentinel new configuration saved on disk
1:X 04 Feb 2026 03:54:41.727 # +vote-for-leader e0e25ffa42a5c3eaab26cf4da3727ac257ddbc9c 2
1:X 04 Feb 2026 03:54:41.727 # -sdown sentinel ae3f28e424fcf389aa2a55e805005be431280a71 10.43.38.202 26379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:54:41.727 * ae3f28e424fcf389aa2a55e805005be431280a71 voted for ae3f28e424fcf389aa2a55e805005be431280a71 2
1:X 04 Feb 2026 03:54:41.818 # -sdown slave 10.43.38.202:6379 10.43.38.202 6379 @ mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:54:52.302 # -failover-abort-not-elected master mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:54:52.360 * Next failover delay: I will not start a failover before Wed Feb 4 04:00:42 2026
1:X 04 Feb 2026 03:55:02.765 # -sdown master mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:55:02.765 # -odown master mymaster 10.43.131.245 6379
1:X 04 Feb 2026 03:55:02.765 # -sdown sentinel f3600ef2957066d701580da18db9b1ae90fac005 10.43.131.245 26379 @ mymaster 10.43.131.245 6379
Updating redis and sentinel config..
Updating sentinel config..
evaluating sentinel id (${SENTINEL_ID_1})
sentinel id (ae3f28e424fcf389aa2a55e805005be431280a71), sentinel grp (mymaster), quorum (2)
redis master (10.43.247.211:6379)
announce (10.43.38.202:26379)
Updating redis config..
we are slave of redis master (10.43.247.211:6379)
Setting redis auth values..
Wed Jan 28 12:50:26 UTC 2026 Ready...
Mon Feb 9 07:41:42 UTC 2026 Found redis master (10.43.131.245)
Identifying redis master (get-master-addr-by-name)..
using sentinel (qa-launch-cluster-common-redis-ha), sentinel group name (mymaster)
Mon Feb 9 07:42:42 UTC 2026 Found redis master (10.43.131.245)
Identifying redis master (get-master-addr-by-name)..
using sentinel (qa-launch-cluster-common-redis-ha), sentinel group name (mymaster)
Mon Feb 9 07:43:42 UTC 2026 Found redis master (10.43.131.245)
Identifying redis master (get-master-addr-by-name)..
using sentinel (qa-launch-cluster-common-redis-ha), sentinel group name (mymaster)
Mon Feb 9 07:44:42 UTC 2026 Found redis master (10.43.131.245)