Milvus on OpenEverest - A Vector On Your Platform

Sooner or later someone asks the platform team for a vector database. It starts with one RAG prototype, then a second team wants semantic search, and soon there are three different setups nobody wants to own.

Milvus is one of the most popular open-source answers. It is also a lot of moving parts: five kinds of Milvus processes, plus etcd for metadata, object storage for data, and a write-ahead log. That is great for scaling and not so great for the person on call.

As of provider-milvus 0.2.0, Milvus is available on OpenEverest. You get the same Instance API and UI you use for PostgreSQL, MongoDB or Valkey, and the provider takes care of the rest. I wanted to see if it holds up for real platform work, so I did what a team would do with it:

1. Created a production-shaped Milvus cluster.
2. Indexed the entire English Kubernetes documentation for semantic and full-text search.
3. Upgraded Milvus while a client kept searching, and tried to downgrade it.
4. Took scheduled backups to S3 and cloned production into a staging instance from one of them.

Everything below ran on a real cluster. The numbers are what I measured, including the parts that didn’t go well.

How Milvus fits into OpenEverest

OpenEverest v2 is built around providers. The core knows nothing about Milvus. It stores an Instance, and the provider turns it into whatever the database needs. provider-milvus doesn’t reimplement Milvus lifecycle management either. It drives the Milvus Operator, which the Milvus maintainers ship, and bundles it in the same Helm chart.

The provider watches Instance resources with spec.providerRef.name: milvus and reconciles them into a Milvus custom resource. The operator does the rest: it starts the Milvus processes, installs etcd and MinIO as bundled dependencies, and rolls upgrades. The provider adds what OpenEverest users expect from any database: generated credentials, a connection Secret, a downgrade guard, and backups.

Backups are the one place where the provider runs something itself. The Milvus Operator has no backup feature, so the provider runs milvus-backup, the Milvus project’s own tool, as Kubernetes Jobs. It doesn’t copy any data on its own.

Pick a topology

Milvus comes in two shapes, and the provider supports both. Click through to see what each one runs.

You can also point either topology at your own etcd and S3 bucket, including IAM-based access such as EKS IRSA. I used the bundled ones here.

The setup

• Kubernetes 1.36 on Linode (LKE), 3 nodes with 8 vCPU and 16 GB each. The cluster also ran Valkey and PostgreSQL instances for other tests, so Milvus didn’t have it to itself.
• OpenEverest v2.0.0-dev.4
• provider-milvus 0.2.0, which bundles the Milvus Operator 1.3.9
• Milvus 2.6.11 at first, upgraded to 2.6.15 later
• An S3 bucket on a separate MinIO in the cluster, used as the backup target

Install the provider

The provider chart bundles the Milvus Operator, so one helm install is enough:

helm install provider-milvus \
  oci://ghcr.io/openeverest/charts/provider-milvus \
  --version 0.2.0 \
  --namespace everest-system

kubectl get providers milvus

Create a Milvus cluster

In the UI, Milvus shows up as one more engine. The form is generated from the provider’s own schema, so it only offers what the provider supports: topology and version first, then a section per Milvus role (replicas, CPU, RAM), then object storage, the metadata store and the message stream. Exposure, monitoring and pod scheduling sit under advanced configuration.

The UI creates an Instance. This is the one I used, with two proxies and two query nodes so a single pod restart doesn’t take searches down:

apiVersion: core.openeverest.io/v1alpha1
kind: Instance
metadata:
  name: docs-search
  namespace: milvus-demo
spec:
  providerRef:
    name: milvus
  version: "2.6.11"
  topology:
    type: cluster
  components:
    proxy:
      type: milvus
      replicas: 2
      resources:
        requests: {cpu: 250m, memory: 512Mi}
        limits: {cpu: "1", memory: 1Gi}
    mixCoord:
      type: milvus
      replicas: 1
    dataNode:
      type: milvus
      replicas: 1
    queryNode:
      type: milvus
      replicas: 2
      resources:
        requests: {cpu: 500m, memory: 1Gi}
        limits: {cpu: "2", memory: 2Gi}
    streamingNode:
      type: milvus
      replicas: 1
  backup:
    enabled: true
    classRef:
      name: milvus-backup
    storages:
      - storageRef:
          name: s3-backups

I trimmed the resources of the other roles; examples/instance-cluster.yaml in the provider repo has a complete cluster manifest.

It was Ready 2 minutes 42 seconds after kubectl apply, with 11 pods spread across the 3 nodes. Most of that time was etcd and MinIO starting and getting their volumes. Here is what one Instance turned into:

Authentication is on from the start, with a random root password. Applications read everything from the connection Secret: host, port, username, password, a ready-made uri, and a token in the root:<password> form pymilvus expects. The password never appears in the Milvus custom resource or in the config Milvus renders. To change it, you edit the credentials Secret and the provider rotates it in Milvus.

Put something real in it: semantic search over the Kubernetes docs

A vector database without vectors proves little. Platform engineers have a corpus everybody knows: the Kubernetes documentation. So I indexed all of it.

• Data: every page under content/en/docs of kubernetes/website at commit 703e31c: 1,721 pages, split by ##/### headings into 15,582 chunks, 14 MiB of text.
• Embeddings: BAAI/bge-small-en-v1.5 (384 dimensions) through fastembed, on CPU, in a Kubernetes Job. No GPU anywhere.
• Search: an HNSW index on the embeddings, plus Milvus 2.6’s built-in BM25 full-text search on the same text. Milvus computes the BM25 sparse vectors itself, so the client only sends the text.

This is the collection schema:

schema = client.create_schema(auto_id=True)
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field("title", DataType.VARCHAR, max_length=512)
schema.add_field("url", DataType.VARCHAR, max_length=512)
schema.add_field("text", DataType.VARCHAR, max_length=8192, enable_analyzer=True)
schema.add_field("dense", DataType.FLOAT_VECTOR, dim=384)
schema.add_field("sparse", DataType.SPARSE_FLOAT_VECTOR)
schema.add_function(Function(name="bm25", function_type=FunctionType.BM25,
                             input_field_names=["text"], output_field_names=["sparse"]))

index = client.prepare_index_params()
index.add_index(field_name="dense", index_type="HNSW", metric_type="COSINE",
                params={"M": 16, "efConstruction": 200})
index.add_index(field_name="sparse", index_type="SPARSE_INVERTED_INDEX", metric_type="BM25")

The client connects with the values from docs-search-conn, mounted as environment variables. Loading took 30 minutes, and Milvus was the fast part:

Then I asked it questions. Each one ran three ways: dense (vectors only), BM25 (keywords only) and hybrid (both, merged with reciprocal rank fusion in one hybrid_search call). Top 3 hits:

Day 2, with traffic on

Creating a database is the easy part. I wanted to see what happens to the applications when the platform team does its job: upgrade, back up, clone.

So before touching anything, I started a probe: a Job with 4 threads running hybrid searches back to back, recording every result and its latency. On the untouched cluster it did about 900 searches per second, p50 4 ms, p99 10 ms. It ran for 20 minutes, through everything below.

Upgrading Milvus 2.6.11 → 2.6.15

In the UI you change the version on the instance. With kubectl it’s one field:

kubectl -n milvus-demo patch instance docs-search --type merge \
  -p '{"spec":{"version":"2.6.15"}}'

The provider sees that every component uses the same image and switches the operator to a rolling upgrade. The operator then upgrades one role at a time, in the order Milvus requires. Click through the timeline:

Zero failed searches out of 468,420 during the upgrade, and 1,092,480 over the whole 20 minutes. The slowest single search took 46 ms.

One thing I can’t explain yet: about a minute after the proxies finished, p99 went from ~10 ms to ~15 ms and stayed there, while p50 stayed at 4 ms and throughput went up a bit. It could be a change in 2.6.15 or how segments landed on the new query nodes. I didn’t dig into it for this post. The two dips around 16 and 18 minutes are mine: two short Jobs loading the embedding model on a shared node.

To see what happened inside, you don’t need kubectl. The Inspector plugin shows every pod behind the instance, which node it runs on and how often it restarted, and streams its logs. Here, after the upgrade: the Milvus pods spread across the nodes with 0 restarts, and the coordinator’s log assigning the WAL channels to the streaming node.

Trying to go back

Downgrades are where vector databases bite you: the newer version may already have written metadata the older one doesn’t understand. So the provider refuses them:

kubectl -n milvus-demo patch instance docs-search --type merge \
  -p '{"spec":{"version":"2.6.11"}}'

Milvus didn’t change: every pod stayed on 2.6.15, and the probe didn’t notice. The provider log says exactly why:

component "dataNode" cannot be downgraded from 2.6.15 to 2.6.11; choose 2.6.15 or newer

The rough part is where you see it. The Instance went to Failed, but on OpenEverest v2.0.0-dev.4 the reason isn’t copied to the Instance status. “Failed” with no message is scary when Milvus itself is fine. Setting the version back to 2.6.15 brought the Instance back to Ready within seconds.

Backups to S3

OpenEverest describes backup targets with a BackupStorage. The provider reads it and hands its bucket and credentials to milvus-backup. Credentials are passed as Secret references, not written into any config:

apiVersion: backup.openeverest.io/v1alpha1
kind: BackupStorage
metadata:
  name: s3-backups
  namespace: milvus-demo
spec:
  type: s3
  s3:
    bucket: milvus-backups
    region: us-east-1
    endpointURL: http://backup-minio.backup-storage.svc:9000
    credentialsSecretRef:
      name: s3-backups
---
apiVersion: backup.openeverest.io/v1alpha1
kind: Backup
metadata:
  name: docs-search-before-reembed
  namespace: milvus-demo
spec:
  origin:
    type: Instance
    instanceRef:
      name: docs-search
  classRef:
    name: milvus-backup
  storageRef:
    name: s3-backups

The backup was Succeeded in 5 seconds and wrote 24 MB under everest/milvus/milvus-demo/docs-search/ in the bucket. The probe didn’t notice it.

Schedules live on the instance. Each one becomes a CronJob, and every run shows up as a regular Backup that you can restore from or delete:

spec:
  backup:
    storages:
      - storageRef:
          name: s3-backups
        schedules:
          - name: nightly
            enabled: true
            cron: "0 2 * * *"          # UTC
            retention: {type: count, count: 7}
          - name: every-15m
            enabled: true
            cron: "*/15 * * * *"
            retention: {type: time, duration: 1d}

The first every-15m run started at 12:15:00 and was done at 12:15:05. Retention deletes the old runs, and their data in the bucket with them.

The UI shows times in my local timezone (UTC+3), so 12:05 UTC is 15:05 here. The Size column is still empty; the provider doesn’t report backup sizes yet.

Cloning production into staging

This is the feature I’d show a team first. Staging that has the same data as production, created by the platform and not by a script someone wrote two years ago. A new Instance with a dataSource starts empty, restores the backup, and only then reports Ready. It doesn’t have to match production’s topology: staging here is a single standalone pod.

apiVersion: core.openeverest.io/v1alpha1
kind: Instance
metadata:
  name: docs-search-staging
  namespace: milvus-demo
spec:
  providerRef:
    name: milvus
  topology:
    type: standalone
  components:
    standalone:
      type: milvus
      replicas: 1
  backup:
    enabled: true
    classRef:
      name: milvus-backup
    storages:
      - storageRef:
          name: s3-backups
  dataSource:
    type: Backup
    backup:
      backupRef:
        name: docs-search-before-reembed

2 minutes 43 seconds from nothing to a searchable copy. The restore itself took 31 seconds; the rest was Milvus starting. While seeding, the instance reports Restoring and doesn’t publish its connection details, so nothing connects to a half-restored database.

Is it really the same? I ran the six questions against both instances and compared the top 5 results by primary key. They matched exactly, and so did the row count.

What’s not there yet

In 0.2.0, but not in this post

• GPU builds. A 2.6.15-gpu version bundle runs every role on the GPU image.
• Node pools per role. Run one role, for example query nodes, as several groups with their own node selectors, such as one group on L40S nodes and one on H200.
• Prometheus. One toggle creates a PodMonitor for every Milvus component.
• External access. LoadBalancer or NodePort on the client-facing component, with the external address in the connection Secret.
• Password rotation. Change the password in the credentials Secret and the provider applies it in Milvus.

Practical advice

If you’re about to offer Milvus to your teams:

• Start teams on standalone, move to cluster when they need it. Standalone is 3 pods; the cluster above is 11.
• Run at least two proxies and two query nodes in production. That’s the shape that kept the probe at zero errors through the upgrade. mixCoord and the streaming node ran one replica each here and were replaced without errors, but I didn’t kill them by force.
• Plan for the embedding pipeline, not the database. Loading 15,582 chunks took 35 seconds in Milvus and 29 minutes in a CPU embedding model. Your teams’ bottleneck is GPUs for embeddings, not vector storage.
• Turn on scheduled backups from day one, to real object storage, not the bundled MinIO.
• Give teams staging copies from backups instead of letting them load production data by hand. It took under 3 minutes here.

Try It Yourself

• Install: Follow the quick install guide to get OpenEverest v2 running in your cluster.
• Milvus: Go to the provider-milvus repository for the README, examples and the backup design.
• Chat: Join the conversation in the CNCF Slack (channel: #openeverest-users).

If you try this and see something different, please open an issue in provider-milvus.