Running TiDB on Kubernetes with OpenEverest

When we were at KubeCon Japan and in Taipei this year, TiDB came up more often than PostgreSQL. Teams run it in production, on VMs and on Kubernetes, and more than one person told us they run it on plain StatefulSets with no operator at all. The alternative they compare against is TiDB Cloud.

There is a middle ground: an operator that does the work, on your own hardware. We recently shipped a TiDB provider for OpenEverest, built on TiDB Operator v2. Instead of writing another “how to install” post, we rented an 8-node cluster and measured what someone planning to self-host TiDB actually wants to know:

• How long until I have a working cluster, and how small can it be?
• Can I really run analytics on the same database as my transactions?
• What happens to my application when I scale, lose a node, back up, or upgrade?
• What does TLS cost?

Along the way the tests found eight bugs and one missing feature in the provider. We fixed them and ran everything again; the numbers below are from the fixed build, and the last section lists what changed.

TL;DR

Three Helm commands and one Instance gave us a highly available TiDB cluster serving SQL in about a minute. HTAP, the main reason to pick TiDB, held up: with analytics on TiFlash, transactions kept 93% of their throughput and analytical queries ran a median 8× faster than on TiKV. The catch is that you have to pin your transactional sessions to TiKV; left to the optimizer, TPC-C lost 65%.

Day-2 work under load was uneventful. New nodes joined in under a minute, losing a TiKV pod stalled some transactions for 10–15 seconds without failing any, and through TiProxy a rolling upgrade from 7.5 to 8.5 didn’t drop a single connection. TLS everywhere costs about 7% of throughput.

Along the way the tests found eight bugs and a missing feature in the provider, all fixed in provider-tidb 0.2.2.

TiDB in one minute

TiDB is a distributed SQL database that speaks the MySQL protocol. It is four programs, and each one runs as its own group of pods:

With OpenEverest you don’t create these yourself. You create one Instance, and the provider turns it into a TiDB Operator v2 Cluster with one group per component, plus TLS certificates, a PodMonitor and backup jobs.

The test bed

Database pods ran only on the database nodes (a taint and a node selector, set through the Instance’s scheduling policy), so the load generator and monitoring never competed with them for CPU.

What the disks can do

TiKV commits every write to its Raft log with fsync, so disk latency shows up directly in write latency. Before installing anything we measured the network block storage against the node’s local NVMe with fio:

We used block storage for everything: it is what most managed Kubernetes users get by default, and it survives losing a node. Local NVMe would make writes faster, at the cost of both.

From zero to a TiDB cluster

You need a Kubernetes cluster and Helm. Install OpenEverest, then the TiDB provider. The provider chart brings TiDB Operator v2 and its CRDs with it:

helm repo add openeverest https://openeverest.github.io/helm-charts/
helm install everest-core openeverest/openeverest --devel \
  --namespace everest-system --create-namespace

helm install provider-tidb oci://ghcr.io/openeverest/charts/provider-tidb \
  --namespace everest-system

The provider ships three presets, and the UI creates a cluster from any of them in one click:

Without the UI, it is one resource. This is the tidb-standard preset written out:

apiVersion: core.openeverest.io/v1alpha1
kind: Instance
metadata:
  name: orders
spec:
  providerRef:
    name: tidb
  version: "8.5.2"
  topology:
    type: cluster
    parameters:
      tls: {cluster: true, client: true}
      monitoring:
        prometheus: {enabled: true}
  components:
    pd:   {type: pd,   replicas: 3, resources: {limits: {cpu: "2", memory: 4Gi}},  storage: {size: 20Gi}}
    tikv: {type: tikv, replicas: 3, resources: {limits: {cpu: "4", memory: 16Gi}}, storage: {size: 100Gi}}
    tidb: {type: tidb, replicas: 2, resources: {limits: {cpu: "4", memory: 8Gi}}}
    tiproxy: {type: tiproxy, replicas: 2, resources: {limits: {cpu: "2", memory: 2Gi}}}

The connection details (the TiProxy endpoint on port 6000), the root password and the CA for TLS land in the orders-conn Secret. The UI shows them on the Instance’s overview, here for a tidb-dev cluster, which has no TiProxy:

Tip: use the full resource name, kubectl get instances.core.openeverest.io. TiDB Operator v2 puts its PD, TiKV and TiDB objects in a kubectl category called instance, so kubectl get instance lists those instead.

How long it takes

On the test cluster, with images already on the nodes, a tidb-standard cluster had PD, TiKV and TiDB up and serving SQL 50–78 seconds after the Instance was applied, and tidb-analytics (adding two TiFlash nodes) in 55–57 seconds. The Instance turns Ready once TiProxy is up too, which took up to another 67 seconds; more on why in What testing changed. Most of the rest is Kubernetes work: attaching the block-storage volumes takes 10–30 seconds per pod, and in every run one or two PDs waited another 35–45 seconds for their DNS name to resolve consistently. Most of the remaining time is Kubernetes work: attaching the block-storage volumes takes 10–20 seconds per pod.

The first version of this chart looked different. On provider-tidb 0.2.0, two of the three PD pods crash-looped two or three times during every bootstrap, and the whole cluster waited for the last one: switch between the runs to see it. The cause, and the fix, are in the last section; with it, there were no restarts in any run.

HTAP: analytics on the transactional database

This is the reason most people look at TiDB. Analytical queries usually go to a separate warehouse that an ETL job feeds every hour or every night. TiFlash keeps a columnar copy of the same tables inside the cluster, updated through Raft, so a report can read data that was committed a few milliseconds ago, without slowing down the transactions that wrote it. At least, that’s the promise. We tested it with CH-benCHmark: TPC-C transactions and 22 TPC-H-style analytical queries running at the same time on the same 100-warehouse dataset (4.5 GB of data, 30 million order_line rows, before TiKV keeps three copies).

Step 1: add TiFlash replicas

TiFlash nodes start with the tidb-analytics preset, or with any Instance once you give the analytics layer some nodes:

Tables only get a columnar copy when you ask:

ALTER TABLE order_line SET TIFLASH REPLICA 2;
SELECT table_name, progress, available FROM information_schema.tiflash_replica;

For all 12 tables, two replicas each, that took 208 seconds.

Step 2: run transactions and analytics together

We ran 32 TPC-C clients for 15 minutes, with 2 more clients running the 22 analytical queries in a loop, a couple of busy dashboards, in four configurations:

• TPC-C alone did 24.2k tpmC (New-Order transactions per minute) with a p99 of 76 ms. This is the baseline.
• Analytics on TiKV cut it to 10.3k tpmC (−57%), with p99 at 285 ms. The row store has to scan and join millions of rows for every report, and the transactions queue behind it. Even so, 13 analytical queries failed, most of them stopped by TiDB’s 1 GiB per-query memory limit while joining the rows TiKV sent it.
• Analytics on TiFlash kept 22.6k tpmC (−6.5%) and an 84 ms p99, while the dashboards ran 3.1× more queries (470 queries per hour, QphH, against 152).
• Letting the optimizer pick for every session was the worst of both: 8.6k tpmC (−65%) with a 176 ms p99, even though the dashboards got 296 QphH.

The “Analytical queries” view runs each of the 22 queries once on an idle cluster. On TiKV, 19 finished and three hit the 1 GiB limit. On TiFlash, all 22 finished, a median of 8× faster: Q1 went from 2.7 s to 0.5 s, Q13 from 2.8 s to 0.14 s, and Q4 from 24.3 s to 0.8 s (32×).

Pin your OLTP sessions. The orange line is the trap. Once tables have TiFlash replicas, TiDB’s optimizer may pick TiFlash for any query that looks cheaper there, including the Stock-Level and Order-Status parts of TPC-C. With TiFlash busy serving reports, those transactions waited behind them. Compared with the pinned run, the average Stock-Level went from 19 ms to 578 ms, Order-Status from 13 ms to 235 ms, and Delivery from 102 ms to 760 ms. TiDB’s answer is engine isolation. Set it per session in your OLTP connection pool:

SET SESSION tidb_isolation_read_engines = 'tikv,tidb';

or make TiKV the default for every session through the TiDB config parameter of the Instance, and let reporting tools opt in with 'tiflash,tidb':

[isolation-read]
engines = ["tikv", "tidb"]

In our test with four analytical clients, that one setting took unpinned TPC-C from 6.6k back to 24.0k tpmC, and the analytical sessions still read from TiFlash.

Step 3: how fresh is “real time”?

TiFlash is a Raft learner: before it answers, it checks it has caught up with the leader. We wrote a row to TiKV and immediately read it back through TiFlash, 1,000 times:

Every read saw the row that had just been committed. Under load, TiFlash first finished the analytical work it was busy with, so the read waited; it never returned stale data.

What about memory?

TiFlash runs joins and aggregations in memory, and that is what to size it for. We started with 16 GiB per TiFlash node; under CH-benCHmark TiFlash hit its memory limit, failed analytical queries and, before we pinned sessions, the OLTP statements the optimizer had sent to it. The tidb-analytics preset now gives TiFlash 32 GiB.

With four analytical clients instead of two, even 32 GiB was tight: transactions on TiKV were unaffected (23.2k tpmC), but 3.5–24% of the analytical queries failed in different runs. Once the heaviest join (Q9) takes TiFlash over its limit (80% of the container’s memory), TiFlash turns away new queries until it is back under it. Two ways out:

• More memory per TiFlash node, or more TiFlash nodes, for more concurrent heavy queries.
• Query-level spilling. SET GLOBAL tiflash_mem_quota_query_per_node = 6442450944; makes TiFlash spill a query’s joins and aggregations to disk past 70% of 6 GiB. In our test it cut the failures to four, at the price of half the analytical throughput, because the spill files went to network block storage.

Scaling, losing a node, backups and upgrades, under load

Benchmarks on a quiet cluster are easy. The questions that matter are about the moments you change something while the application is running. For each test below, 32–64 TPC-C clients were running the whole time, and for the disruptive ones a probe inserted a row every 100 ms on its own connection, so every stall is measured from the client side. Pick a test:

Scaling out and in

Scaling is one field in the Instance: spec.components.tikv.replicas. With 64 TPC-C clients running against the tidb-analytics cluster (TiProxy in front):

• TiKV 3 → 5. The two new TiKV pods were up in 58 seconds. PD then moved Regions to them in the background: over the next 18 minutes the new nodes filled from 0 to 10–12 GB each while the original three went from 18 to about 12 GB, without a dip in throughput or latency. Throughput didn’t go up either, because TiKV wasn’t the bottleneck here.
• TiDB 2 → 4. The new TiDB pods were ready in 38 seconds. TiDB was the bottleneck, and the gain depends on getting sessions onto the new pods. Clients keep their connections, so without a proxy the new pods sit idle: on provider 0.2.0 they served 0 queries until the end of the test. With TiProxy, sessions moved over gradually, and throughput went from 27.1k to 32.7k New-Order/min (+20%), with p99 down from 143 ms to 119 ms.
• TiDB 4 → 2. Done in 31 seconds. TiProxy moved the sessions off the TiDB pods being removed between transactions: the clients saw no errors.
• TiKV 5 → 3. Before deleting a pod, the operator takes its store offline and PD moves every Region away. That took 217 seconds, during which throughput dropped 18% and p99 rose to 179 ms. The data was never down to fewer than three copies.
• TiKV 3 → 2 was rejected. PD keeps three copies of every Region and can’t remove a store once fewer than three would be left, so the provider refuses the change instead of leaving the operator stuck.

Across the 41-minute run, the clients saw zero failed transactions.

Losing a TiKV pod

We force-deleted the TiKV pod that led the most Regions (kubectl delete pod --grace-period=0 --force), with 32 TPC-C clients and the write probe running. Kubernetes started a new pod on the same volume in 7 seconds. Its Regions had already elected new leaders on the other two TiKV nodes, which takes up to about 10 seconds with TiKV’s default election timeout. For those 10–15 seconds, transactions touching the affected Regions waited: New-Order p99 peaked at 16 s, and throughput dipped to 15.6k for half a minute before going back to 23k.

Nothing failed. The 24 TPC-C transactions that returned an error over the 21-minute run are TPC-C’s own write conflicts, at the same rate as in runs without any fault. The probe’s writes went to Regions led by other nodes and never waited more than 33 ms. Afterwards all 12,134 writes the probe got an “OK” for were in the table.

Backups and restore

A backup is a Backup resource pointing at a BackupStorage (here the cloud provider’s S3-compatible object storage in the same region). BR asks every TiKV node to upload its own data in parallel:

During the backup, throughput dropped about 10% (22.1k to 20.0k tpmC) and p99 rose from 83 to 92 ms.

Scheduled backups with a retention count are part of the Instance, and the UI sets them up:

To restore, we created a new Instance from the tidb-standard preset with the backup as its dataSource. It was Ready with the data in 175 seconds, of which the BR restore itself took 61 seconds. Its connection Secret worked straight away; on provider 0.2.0 it didn’t, see below.

spec:
  dataSource:
    type: Backup
    backup:
      backupRef: {name: orders-nightly}
  backup:                      # the restored Instance reads the backup through its storage
    enabled: true
    classRef: {name: br}
    storages:
      - storageRef: {name: s3-store}

A rolling upgrade, with and without TiProxy

Upgrading is also one field: spec.version: "7.5.5" → "8.5.2". TiDB Operator restarts the components in the order TiDB requires, PD → TiKV → TiDB, one pod at a time, and waits for each to be healthy before the next. On a tidb-standard cluster with 50 warehouses, 32 TPC-C clients and the write probe, that took 11.5 minutes.

With clients going through TiProxy, the upgrade was hard to spot from the outside: New-Order p99 never went above 193 ms, the probe’s slowest write took 86 ms, and it got 0 errors for 11,721 writes, every one of which was in the table afterwards. TiProxy moved the sessions off each TiDB pod before it stopped, which is what graceful-wait-before-shutdown is for.

We ran the same upgrade on a tidb-standard cluster without TiProxy, with TPC-C and the probe connected straight to the TiDB Service (the Upgrade, direct tab above). It took 11.0 minutes. The PD and TiKV steps looked the same as through TiProxy: a PD leader change made a timestamp request fail in one TPC-C transaction through TiProxy and two direct, and nothing else. The difference was the TiDB step. TiDB Operator rolls TiDB with one extra pod and removes it at the end, and a client connected to a TiDB pod that goes away loses its connection: two TPC-C transactions and one probe write failed that way. New-Order p99 peaked at 369 ms instead of 193 ms, and the probe’s slowest write took 549 ms instead of 86 ms. No acknowledged write was lost; all 13,176 were in the table. Most connection pools retry such errors, so the gap is small, but it is the gap between an upgrade the application doesn’t notice and one it logs.

One more thing the upgrade showed: on the same cluster, TiDB 8.5.2 did 20–27% more New-Order transactions per minute than 7.5.5 (23.9k against 18.9k in the TiProxy run, 23.1k against 19.2k in the direct one), with a lower p99.

What TLS costs

TLS is on by default for new Instances: clients to TiProxy and TiDB, and between PD, TiKV, TiDB and TiProxy. To see what it costs, we ran two identical tidb-standard clusters, one with TLS and one without, and alternated three 10-minute TPC-C runs on each, with 64 clients through TiProxy and 50 warehouses:

The runs on each side were within 1.5% of each other, so the 7% is not noise. At this load TiDB’s CPUs were the limit, so the cost shows up as throughput: encrypting every hop leaves less CPU for transactions. With spare CPU on the TiDB nodes it would show up as a few milliseconds of latency instead.

What testing changed in the provider

We started on provider-tidb 0.2.0 and fixed what the tests found before taking the numbers above. All of it is in provider-tidb 0.2.2 (#69–#75, #78, #80):

Two things we found are outside the provider. TiDB Operator v2.0.1 starts TiProxy pods only after its next PD sync, which adds up to about 70 s before the Instance is Ready with TiProxy (the database itself accepts connections before that); the operator’s main branch has already removed that wait. And when the provider rejects a change, such as scaling TiKV below PD’s max-replicas, the reason is only in the provider log on OpenEverest v2.0.0-dev.4; the next developer preview shows it on the Instance.

What’s not there yet

• Point-in-time recovery. Backups are full snapshots; BR’s log backup isn’t wired in yet.
• TiCDC (change data capture) isn’t a component of the provider yet.
• Certificate rotation. Provider-issued certificates are valid for ten years and are not rotated; use cert-manager with your own issuer if you need shorter lifetimes.

FAQ

Is TiDB MySQL-compatible? It speaks the MySQL protocol and most of MySQL 8.0’s SQL, so MySQL drivers, ORMs and the mysql client work as they are. Stored procedures, triggers and events are not supported; check the compatibility list before migrating.

How much hardware does a self-hosted TiDB cluster need? For a laptop or CI, the tidb-dev preset asks for 3 vCPUs and 6 GiB. For production, the smallest highly available layout is 3 PD, 3 TiKV and 2 TiDB; our tidb-standard preset asks for 30 vCPUs and 80 GiB across them (including two TiProxy nodes), and adding TiFlash for analytics takes another 8 vCPUs and 64 GiB. TiKV and TiFlash should each run on their own nodes.

Do I need TiDB Operator v1 or v2? TiDB Operator v2 is a redesign: one Cluster resource plus a group per component, instead of v1’s single TidbCluster. It manages the same TiDB releases. New deployments should use v2, which is what the OpenEverest provider is built on.

Can I run TiDB on Kubernetes without an operator? You can run the binaries in StatefulSets, and some teams do. The operator earns its keep on the day-2 work: taking a TiKV store offline before removing its pod, upgrading components in the right order (PD → TiKV → TiDB), rotating certificates and running BR backups. Every scaling and upgrade number in this post is the operator doing that work.

Self-hosted TiDB or TiDB Cloud? TiDB Cloud runs and patches the cluster for you. Self-hosting keeps the data on your own infrastructure, on any Kubernetes, with your own storage and network rules, and the software is open source (Apache 2.0). With OpenEverest, the day-2 work (scaling, backups, upgrades, TLS, monitoring) goes through the same API and UI as your other databases.

Is TiFlash a replacement for a data warehouse? For analytics on live operational data, often yes: no ETL, and reads are consistent with what was just committed. For years of history from many sources, a warehouse or lakehouse is still the better place. Size TiFlash memory for your heaviest concurrent queries.

How do backups work? BR takes a consistent snapshot of the whole cluster and writes it to S3-compatible storage; each TiKV node uploads its own data in parallel. Schedules with count- or time-based retention are part of the Instance. Point-in-time recovery is not supported by the provider yet.

Try it yourself

Everything here runs on any Kubernetes cluster with Helm, a default StorageClass and, for backups, an S3-compatible bucket:

helm repo add openeverest https://openeverest.github.io/helm-charts/
helm install everest-core openeverest/openeverest --devel \
  --namespace everest-system --create-namespace
helm install provider-tidb oci://ghcr.io/openeverest/charts/provider-tidb \
  --namespace everest-system

Then create a cluster from the tidb-dev preset in the UI, or apply the Instance from the beginning of this post. The provider README covers TLS, monitoring, backups, TiFlash and TiProxy in detail.

The raw results behind every chart (ttr.json, htap.json, timeline.json) are next to this post, and the benchmark scripts and raw results are in blog-data. If you run TiDB on Kubernetes, with or without an operator, we’d like to hear how: what broke, what’s missing, and what you would measure next.

• Install: follow the quick install guide to get OpenEverest v2 running in your cluster.
• TiDB: learn more about TiDB on OpenEverest, or go straight to the provider-tidb repository.
• Chat: join the conversation in the CNCF Slack (channel: #openeverest-users).