Breaking MariaDB on Purpose: Failover with MaxScale on OpenEverest

Recently we added MariaDB MaxScale support to provider-mariadb. You set enabled: true on the proxy component, and OpenEverest puts MaxScale in front of your Galera or replication cluster. Clients get one endpoint, and MaxScale sends writes to the primary and spreads reads across replicas.

Soon after, a contributor left a detailed comment on the original issue. They had tested MaxScale with mariadb-operator and saw a race: MaxScale promotes a replica, and a moment later the operator turns it back into a read-only replica. The result is a cluster with no writable primary. The advice was to keep MaxScale’s automatic failover on and to have a manual recovery plan ready.

Our provider does the opposite and turns MaxScale’s failover off. So either we got it wrong, or we are looking at different setups. I wanted to see it myself, so I deployed a cluster and started killing primaries.

The setup

• LKE cluster, 6 nodes, Kubernetes 1.36
• OpenEverest v2.0.0-dev.4
• provider-mariadb with mariadb-operator 26.10.1
• MariaDB 12.3, MaxScale 23.08
• Replication topology: 3 MariaDB nodes, 2 MaxScale pods

The Instance is the same one you would create from the UI. The only MaxScale-specific part is this:

    proxy:
      replicas: 2
      parameters:
        enabled: true

Who is in charge of failover

This is the most important question in this setup. There are two components that know how to promote a replica: mariadb-operator and MaxScale’s mariadbmon monitor. If both try to do it at the same time, you get the race from the issue.

mariadb-operator supports both models. Click through them:

So we were not looking at the same setup. The rest of the post is about whether the “operator owns failover” model holds up in practice.

How I tested

I needed two things: a client that tells me exactly when writes stop and start, and a way to check that no data is lost.

The client is a small pod that connects through the MaxScale Service every half a second. Each time it opens a new connection, inserts a row in a transaction and asks which host it landed on. It only logs changes, so the output looks like this:

08:01:18.363 OK mdb-mxs-0
08:01:45.968 ERR ERROR 1815 (HY000): Internal error: Session creation failed
08:02:08.079 OK mdb-mxs-1

Next to it I had a script polling the operator status, both MaxScale pods and read_only plus replication state on every MariaDB node. After every run I compared a checksum of the table across all three nodes.

The scenarios:

• Graceful delete of the primary pod (kubectl delete pod). MariaDB shuts down cleanly, and Kubernetes starts the pod again right away.
• Force delete of the primary pod (--grace-period=0 --force).
• Kill one of the two MaxScale pods.
• The same graceful delete without MaxScale, as a control.
• A planned switchover, which the operator does during a rolling update.

The numbers

This is the time between the first failed write and the first successful write after it. The client polls every 0.5 s, so take the decimals with a grain of salt. Tap or click a row for details.

A few things stand out:

• Failover works. In every run the operator picked the most advanced replica, promoted it, and MaxScale started routing writes to it.
• MaxScale is not the slow part. It usually followed the operator’s promotion within 1 to 3 seconds. Most of the time goes into the operator noticing the failure and running the switchover steps.
• The spread is wide. Anything from 3.6 to 25 seconds for an unplanned failover. It depends on timing: how fast the operator notices, and whether the old pod comes back in the middle of the switchover. This is a handful of runs, not a benchmark.
• Losing a MaxScale pod is a non-event with two replicas.

And the data?

For me this is the important part. In every run, the table checksum matched on all three nodes. No acknowledged write was lost, and no write ended up on one node only. The operator enables semi-synchronous replication by default, so a commit on the primary waits until at least one replica has the transaction. That is what makes promoting a replica safe. One caveat: if no replica answers within the timeout (10 seconds by default), the primary falls back to asynchronous replication. It never happened in my tests, but it is the thing to keep in mind when you think about losing data.

What was breaking

The happy path is fine, but two things happened after the failover that I did not like. The easiest way to show them is to walk through the slowest graceful delete run (the red bar above) second by second. This run was done before the fix.

So there are two separate problems here. Open the cards for the details.

Manual recovery

Until the upstream fix lands, a stuck former primary needs a hand. The recipe below worked every time in my tests, but it is only safe when two things are true. If you are not sure about either of them, don’t run it.

1. The stuck node has nothing the new primary is missing. Compare a checksum of your important tables, or at least row counts and the latest rows. If the node really diverged, this recipe would hide the problem.
2. The new primary’s binary log starts at its promotion. Look at the first GTID in its oldest binary log (SHOW BINARY LOGS, then SHOW BINLOG EVENTS IN '<oldest file>'). Its sequence number must directly follow the stuck node’s last GTID. In my case the stuck node stopped at 0-10-421 and the new primary’s log started at 0-11-422. If the new primary still has binary logs from an earlier time when it was primary, the recipe would replay old transactions.

If both hold, run this on the stuck node:

STOP SLAVE;
RESET MASTER;
SET GLOBAL gtid_slave_pos='';
CHANGE MASTER TO MASTER_USE_GTID=slave_pos;
START SLAVE;

RESET MASTER drops the node’s own binary log, which is what confuses the new primary. With an empty position, the node starts replicating from the beginning of the new primary’s binary log, which is exactly the promotion point. Within a minute the pod becomes ready and the operator takes it from there.

One thing not to do: don’t delete the stuck node’s volume and expect the operator to rebuild it. New replicas are seeded from a physical backup, and without one configured there is nothing to copy the data from.

The cleaner path is to let the operator rebuild the replica from a physical backup. mariadb-operator can do this automatically (replica recovery), but it needs backup storage configured. We are looking into turning it on in the provider when backup storage is available.

Practical advice

If you run MariaDB replication with MaxScale on OpenEverest:

• Leave MaxScale’s failover settings alone. The provider sets them on purpose. In our model the operator owns failover, and MaxScale only routes traffic.
• Upgrade the provider to get semiSyncBootAsReplica, and plan for one rolling restart.
• Watch for a MariaDB pod in CrashLoopBackOff with error 1236 in its logs after a failover. It means the cluster runs with one node less than you think.
• Run two MaxScale pods. That is the default, and losing one costs nothing.
• Expect up to 25 seconds of failed writes on an unplanned failover, and make sure your application retries.

What’s next

The most important missing piece is mariadb-operator#1948. Once it is fixed upstream, the old primary will rejoin on its own, and we will pick up the new operator version in the provider. On our side, we will fix the misleading status and look into automatic replica recovery.

If you try this yourself and see different behavior, please comment on the issue or open a new one. Failover bugs mostly show up under timing nobody has tried yet, so every report helps.

Try It Yourself

• Install: Follow the quick install guide to get OpenEverest v2 running in your cluster.
• MariaDB: Learn more about MariaDB on OpenEverest, or go straight to the provider-mariadb repository.
• Chat: Join the conversation in the CNCF Slack (channel: #openeverest-users).