Resharding Valkey Under Load on OpenEverest

Scaling a sharded database is easy to promise and hard to do. You can add a shard in one line of YAML. The data still has to move, and clients still have to find it while it moves.

provider-valkey runs Valkey Cluster on OpenEverest through the Valkey operator. The operator can change the number of shards on a live cluster. I wanted to know what this looks like from the client side: does throughput actually grow, do clients get errors, and can you lose data. I also wanted to see how it behaves when things go wrong in the middle, because they do.

So I built a cluster that was already maxed out, and started changing it.

The setup

• Kubernetes 1.36 cluster
• 4 dedicated nodes for Valkey (16 vCPU, 32 GB each) and 1 node for monitoring and test clients
• OpenEverest v2.0.0-dev.4
• provider-valkey 0.1.8 with valkey-operator 0.7.1
• Valkey 9.0.6, cluster topology, 3 shards with 1 replica each, TLS on
• kube-prometheus-stack and the OpenEverest metrics plugin for dashboards

This is where everything ran once the cluster had 6 shards. Each shard is a primary and a replica, and the operator never puts two pods of the same shard on one node. The dashed shards are the ones the scale-out added.

Every client talks to every shard, so each memtier pod sends traffic to all four nodes, not just its own. Which pod of a shard is the primary changed during the tests, because every failover swaps the roles.

This is the Instance. It is what you get from the UI, plus a node selector and two config lines:

apiVersion: core.openeverest.io/v1alpha1
kind: Instance
metadata:
  name: vk-shards
  namespace: valkey-blog
spec:
  providerRef:
    name: valkey
  version: "9.0"
  topology:
    type: cluster
    parameters:
      numShards: 3
  components:
    engine:
      type: valkey
      replicas: 1
      resources:
        limits:
          cpu: "2"
          memory: 4Gi
      storage:
        size: 10Gi
        storageClass: linode-block-storage
      parameters:
        config:
          maxmemory: 3gb
          maxmemory-policy: noeviction
    monitoring:
      type: exporter

To scale, I changed one number:

kubectl patch instance vk-shards -n valkey-blog --type merge \
  -p '{"spec":{"topology":{"parameters":{"numShards":6}}}}'

You can’t change the number of shards in the UI yet. It is a known gap, more on it below.

How I tested

I needed two clients: one to put real pressure on the cluster, and one to tell me exactly which writes failed and whether anything got lost.

Load. Four memtier_benchmark pods in cluster mode, one per Valkey node. Each runs 3 threads with 25 connections, 1 SET for every 4 GETs, 1 KB values over 2 million keys. Before the tests I preloaded all 2 million keys. memtier exposes Prometheus metrics, so throughput, errors and latency end up on the same dashboards as Valkey.

Probe. A small Python script with redis-py’s cluster client. Eight processes write sequential keys like probe:{3:5124}:5124017. The {3:5124} part is a hash tag, so every 1,000 consecutive keys land in the same hash slot. Every write is either acknowledged or recorded as failed. At the end I read back every acknowledged key. If an acknowledged key is missing, the cluster lost it. The hash tag also tells me the slot, and the slot tells me which shard lost it.

Monitoring. A loop that recorded the ValkeyCluster status every few seconds, and Prometheus scraping every 5 seconds.

The important detail: the cluster was already saturated before every test. With 3 shards, each primary’s main thread was pinned at one CPU core. Valkey runs commands on a single thread, so one core per primary is the ceiling. The cluster did about 39,000 ops/sec with p99 latency around 48 ms. If adding shards works, throughput should go up. If it doesn’t, nothing else matters.

That is the worst case, not a normal day. So at the end I repeated both scaling directions at about 45% of capacity, which is closer to how a production cluster should run.

Scaling out: 3 to 6 shards

I changed numShards from 3 to 6 and watched. The whole thing took 126 seconds, and throughput almost doubled.

Client throughput Keys per shard Reads per shard

memtier ops/sec. Flat at ~39k while the new pods start, then ~75k once the slots have moved. p99 latency stayed at ~48 ms the whole time: the cluster is still saturated, there are just twice as many cores doing the work.

Each old shard gives half of its keys to one new shard, one pair at a time: shard 0 to shard 5, then 2 to 4, then 1 to 3. About 900k keys per pair, moved in roughly 25 seconds while the cluster served full traffic.

Reads per shard, from the Valkey exporter. About 10k reads/sec per shard before and after. Each shard was already doing all it could, so the only way to get more is more shards.

What happens inside

Valkey Cluster splits the key space into 16,384 hash slots. Every key belongs to one slot, and every slot belongs to one shard. Scaling out means creating new shards and moving slots to them, together with the keys.

Valkey 9.0 brought a big change here: atomic slot migration. Before 9.0, moving a slot meant moving its keys one by one with MIGRATE. While a slot was half-moved, clients got ASK redirects and multi-key commands could fail. Now the operator sends one CLUSTER MIGRATESLOTS command to the source. The source sends a snapshot of the slots to the target, streams the changes that happen in the meantime, pauses writes for a moment and hands over ownership in one step. It works a lot like replication. Clients don’t see the slot until it has moved, and then get one MOVED for it. The operator moves up to 400 slots per batch, one batch per reconcile.

Click through the timeline:

To me this is the best part of the whole exercise. One number in the spec, two minutes, and twice the throughput, with zero lost writes and no client-visible errors except the CLUSTERDOWN burst. I don’t have an explanation for that burst yet, but a client that retries doesn’t see it.

Scaling in

Scaling in is the same in reverse: move slots off the shards you remove, then delete them. I did it twice. 6 → 3 with lighter load took 313 seconds without a single error.

The second time I made it harder: 6 → 4 shards and 1 → 2 replicas per shard in the same change, at full load. It took 776 seconds, and again 0 failed writes. The operator first added the new replicas, then drained shards 4 and 5, then rebalanced. But look at where the keys went:

Most of the drained slots went to shard 0. It peaked at 4.3 million keys, 1.6 times its final size, and only then the operator spread them out. The 6 → 3 run looked the same: shard 0 peaked at 1.7 times its final size. That’s how the drain is written right now. It picks “the first valid destination” and leaves the rest to the rebalance at the end. The comment calls this intentionally simple.

It has two effects. A lot of data moves twice. And shard 0 becomes the hot shard: it owns a big share of slots, but it still has one core. Remember, the cluster is as fast as its busiest shard. Throughput went down from 73k to about 33k ops/sec during the drain and came back to 50k at the end (4 shards × 1 core). The scary part is memory. My shards used about 0.8 GB of 3 GB, so there was plenty of room. If your shards are 60% full and you scale in, the receiving shard can hit maxmemory, and with noeviction that means failed writes. With an eviction policy it is worse: Valkey silently drops keys.

Is 776 seconds a lot? Yes. The Valkey team measured a 4 → 3 scale-in of a 40 GB cluster under heavy load in 27 seconds. That’s roughly 370 MB/s. My drain moved about 1.6 GB, counting the second move off shard 0, in about 670 seconds. That’s 2 to 3 MB/s. It isn’t a fair fight: my primaries had no CPU left, and migration has to share that one thread with clients. The operator also adds pauses: 400 slots per batch, one batch per reconcile, and a replica sync check in between. But a gap this big means most of the time is spent outside Valkey.

And at a normal load?

Nobody should run a production cluster at 100% CPU. So I lowered memtier’s request rate to about 45% of capacity. p99 latency dropped to 7 ms. Then I scaled out 4 → 6 and back to 4.

One detail if you try this: in cluster mode, memtier’s --rate-limiting applies to every connection to every shard. The offered load grows and shrinks with the number of shards, so for these runs I compare latency, not throughput.

Scale-out takes about two minutes either way. Most of that is creating pods and volumes, and the operator waiting between moves. Scale-in is six times faster with CPU to spare, even after you subtract the replica change from the saturated run. Clients see the change as a latency bump into the tens of milliseconds for a minute or two. They don’t see errors.

The hot shard didn’t go away: shard 0 still peaked at 1.5 times its final size. That comes from the drain logic, not from the load.

Draining a node

Kubernetes upgrades and node maintenance mean kubectl drain. I drained a node with two primaries and one replica on it:

kubectl drain lke665663-986688-kd8pd --ignore-daemonsets --delete-emptydir-data

The operator creates a PodDisruptionBudget with maxUnavailable: 1, so the drain evicts one Valkey pod at a time and retries the rest with “Cannot evict pod as it would violate the pod’s disruption budget”. Each evicted primary handed its role to its replica on shutdown. Each pod then started on another node, after its block volume detached and attached there (about 30 to 45 seconds per pod).

The drain took 114 seconds. The probe recorded 0 failed writes, with the slowest write at 1.25 seconds. memtier dipped to 48k ops/sec for a moment. That’s what a drain should look like.

Killing primaries

Planned operations went well. Now the unplanned part. I deleted primary pods in two ways:

• Graceful: kubectl delete pod. The provider sets shutdown-on-sigterm failover, so on SIGTERM the primary hands its slots to its replica before it exits.
• Force: kubectl delete pod --grace-period=0 --force. The process is killed without warning, like an OOM kill or a crash.

Every row is one second, from the probe’s point of view:

Graceful delete works, mostly. In 4 runs out of 5 the replica took over in about 0.7 seconds. The probe lost 0 to 2 writes, and memtier dipped to ~62k for a second. In one run the handoff didn’t happen. The replica stayed a replica, the pod restarted, and the old primary got its slots back. That was 31 seconds of failed writes for one shard. I didn’t capture the primary’s log for that run, so I don’t know why yet.

Force delete is a different story. I expected a failover. There wasn’t one, and the reason is timing:

1. The pod is killed, and Kubernetes starts a new one right away. Valkey is up again in about 2 seconds.
2. Valkey declares a node failed only after cluster-node-timeout, which is 15 seconds. The restarted node is back long before that. Nobody fails over.
3. The restarted node still has its nodes.conf, so it rejoins as primary, with whatever data it loaded from disk.
4. The replica, which has the latest data, sees a primary with a different history and does a full resync. It throws away its data and copies the primary’s.

Step 4 is where data gets lost. Without AOF (Append Only File), the restarted primary loads its last RDB snapshot. Everything written since that snapshot is gone, including what the replica had. My probe confirmed it: out of 8,730,666 acknowledged writes, 3,450 were missing, all in slots of the shard I killed. One worker lost exactly one block, probe:{0:533}:533102 to 533999, slot 12303, owned by shard 5.

3,450 out of 8.7 million sounds like nothing. It is better to look at it as time: it was every write that shard took in the roughly 30 seconds since its last snapshot. With the default save rules, under enough writes, that window can be up to a minute. Valkey replication is asynchronous, and the docs say clearly that a cluster can lose acknowledged writes. The replication docs even have a section about this exact trap: a primary without full persistence that restarts automatically wipes its replicas.

Then I enabled AOF with appendfsync everysec through the Instance config and repeated the kill. Same sequence: the pod came back as primary in 2 seconds, and the replica did a full resync. But this time the primary loaded its AOF, and the final check found 1,984,672 acknowledged writes, 0 missing. With everysec, Valkey writes the AOF to the file before it replies to the client, and only the fsync is delayed. A killed process doesn’t lose what is already in the kernel page cache. A crashed node can still lose up to one second of writes. That is one kill, so it shows how it works, not a statistic.

Why 30 seconds?

The force-killed pod was back in 2 seconds, but its slots failed for 30. I suspected DNS and tested it: I force-deleted another primary and resolved its hostname twice a second from a client pod.

-0.4s 10.2.1.15      # old pod IP
+30.5s 10.2.1.23     # new pod IP, the pod was Ready at +6s

For 30 seconds after the kill, DNS kept returning the old pod IP. With TLS, the operator announces nodes by hostname (preferredEndpointType: Hostname), so clients connect by name. CoreDNS on LKE has ttl 30 and cache 30, which is also the default in most clusters. So clients spend 30 seconds trying to reach a pod that no longer exists. With a graceful delete this doesn’t matter: the replica takes over, and the slots move to a host that is already in DNS.

One more thing for client authors: memtier got stuck. After both force kills it went to 0 ops/sec and never recovered, even with --reconnect-on-error. I had to restart it. The redis-py probe recovered by itself in both cases. Before you trust your client to survive a failover, test it.

All runs in one place

Seconds in which the probe saw failed writes. Tap or click a row for details.

In total the probe checked 10.7 million acknowledged writes*. Exactly 3,450 were lost, all from one force kill without AOF. Nothing else lost data, and no key ever had the wrong value.

What was breaking

Open the cards for the details.

Practical advice

If you run sharded Valkey on OpenEverest:

• Add shards when the primaries hit one core. Valkey runs commands on one thread, so more CPU per pod won’t help. In my test, twice the shards gave 1.9 times the throughput.
• But don’t wait until you’re at 100%. Scaling still works when the cluster is saturated, but scale-in took six times longer than at 45% load. Plan the change while you have headroom.
• Enable AOF. Add appendonly: "yes" and appendfsync: everysec to the engine config. Without it, a crashed primary can take recent writes with it, replica or not. Enabling it restarts every pod once, so do it in a quiet hour.
• Use a cluster-aware client that retries MOVED, CLUSTERDOWN and connection errors. And test it with a failover before you rely on it.
• Before you scale in, check memory. Until the drain is fixed, assume one shard can briefly get 1.7 times its share of data.
• Delete the PVCs of removed shards after a scale-in, until the provider does it for you.
• Use kubectl drain for node maintenance. It is the smoothest operation I tested.

What’s next

We will fix the PVC reclaim policy in the provider and report the re-grow loop and the drain imbalance to the valkey-operator project. I also want to dig into the CLUSTERDOWN burst and the missed graceful handoff, ideally with a reproducer.

The bigger lesson for me: the planned operations are in good shape. Doubling shards under full load in two minutes, with no lost data, is exactly what you want. Scale-in still needs work: it’s slow at full load, and it piles data onto one shard. The risky part is the unplanned one, and there the defaults matter more than the operator.

If you run these tests yourself and see something different, please open an issue in provider-valkey.

Try It Yourself

• Install: Follow the quick install guide to get OpenEverest v2 running in your cluster.
• Valkey: Learn more about Valkey on OpenEverest, or go straight to the provider-valkey repository.
• Chat: Join the conversation in the CNCF Slack (channel: #openeverest-users).