Monitoring
OpenEverest installs no monitoring system, agents or metrics stack. Instead, it connects instances to the monitoring system you already run — PMM, Prometheus, VictoriaMetrics or another one — through monitoring extensions. What an instance can send, and to which systems, depends on its provider.
Monitoring extensions are new in OpenEverest 2.0.0 Alpha 1. Names, fields and behavior may change before General Availability.
How it works
| Resource | Scope | Created by | What it is |
|---|---|---|---|
MonitoringClass | Cluster | A monitoring extension | A monitoring technology, such as pmm or prometheus, and the settings and authentication methods its destinations accept. |
MonitoringDestination | Namespace | You | One monitoring server: its address, credentials and TLS settings. Instances in the same namespace can send to it. |
Instance.spec.monitoring.destinations | Instance | You | The destinations an instance sends to — up to 8 — with optional per-instance parameters. |
MonitoringBinding | Namespace | OpenEverest | One per instance and destination. It records whether the link is accepted and configured. Read-only. |
A class is served in one of two ways:
- Provider-managed. The provider configures its Operator's native integration — for example, the Percona providers run the PMM agent next to the database. The monitoring extension only checks the destination.
- Extension-managed. The provider publishes the metrics endpoints of each component in the instance status, and the monitoring extension configures the collection — for example, by creating Prometheus Operator
PodMonitorresources.
OpenEverest decides the mode per provider. kubectl get monitoringclass <name> -o yaml shows, under status.providers, which installed providers are compatible with the class, in which mode, and how disruptive enabling it is.
Install a monitoring extension
Monitoring extensions are installed with Helm, like providers. Find them under Integrations in the Plugin Hub, or browse the Hub catalog. The catalog entry shows the chart version, prerequisites, compatibility and install command.
For example, to install the Prometheus extension into everest-system:
helm install monitoring-prometheus \
oci://ghcr.io/openeverest/charts/monitoring-prometheus \
--version 0.1.0 \
--namespace everest-system
Install the prerequisites named by the extension as well. Monitoring extensions connect to an external monitoring system; they do not install a PMM server, Prometheus, Coroot, or VictoriaMetrics backend unless their README explicitly says otherwise.
Check that the class is installed and claimed by its controller:
kubectl get monitoringclasses
NAME ACCEPTED AGE
pmm Accepted 1m
Example: send metrics to PMM
This example sends the metrics and Query Analytics of an instance to a PMM 3 server. It needs the PMM monitoring extension and a provider that supports PMM.
-
In PMM, create a service account with the Admin role and a token for it. The providers register database nodes with PMM using this token.
-
Store the token in a Secret in the instance's namespace, under the key
token:kubectl create secret generic pmm-token -n demo --from-literal=token=glsa_... -
Create the destination:
apiVersion: monitoring.openeverest.io/v1alpha1kind: MonitoringDestinationmetadata:name: pmm-prodnamespace: demospec:classRef:name: pmmparameters:url: https://pmm.example.comauthentication:mode: ServiceAccountTokencredentialsSecretRef:name: pmm-tokenFor a server with a certificate from a private CA, add
tls.ca.secretKeyRefpointing at the CA bundle. -
Check that the destination is ready:
kubectl get monitoringdestination pmm-prod -n demoNAME CLASS AUTH ACCEPTED READY VERSION AGEpmm-prod pmm ServiceAccountToken Accepted Ready 3.8.0 1m -
Add the destination to the instance:
spec:monitoring:destinations:- name: pmmdestinationRef:name: pmm-prodThe
nameidentifies the entry within the instance. Some classes accept per-instance settings inparameters— for PMM, the resources of the agent container.
Enabling or disabling monitoring may add or remove an agent or exporter container, which restarts the instance's pods one at a time. If the instance's maintenance settings do not allow rolling restarts, the change waits for approval and the binding reports MaintenancePending.
To stop sending to a destination, remove its entry from the instance.
Check the status
The instance's MonitoringConfigured condition summarizes all its destinations, and status.monitoring.destinations shows each one:
kubectl get instance orders-db -n demo -o jsonpath='{.status.monitoring.destinations}'
Each link between an instance and a destination is a MonitoringBinding:
kubectl get monitoringbindings -n demo
NAME INSTANCE DESTINATION CLASS ACCEPTED CONFIGURED AGE
orders-db-pmm-7c9d5 orders-db pmm-prod pmm Accepted Configured 2m
Troubleshoot monitoring
Each resource reports a condition that tells you where the problem is:
| Resource | Condition | Common reasons |
|---|---|---|
MonitoringClass | Accepted — the extension's controller serves the class. | Pending: the extension's controller is not running. |
MonitoringDestination | Accepted — the destination is valid. | ClassNotFound, InvalidParameters, AuthModeUnsupported, CredentialsNotFound, InvalidCABundle |
Ready — the extension reached the server with the credentials. | Unreachable, Unauthorized, UnsupportedServerVersion, InvalidParameters, Pending | |
MonitoringBinding | Accepted — the instance's provider can send to this class. | DestinationNotFound, ClassNotAccepted, IntegrationUnsupported, Incompatible, AuthModeUnsupported |
Configured — the collection is set up. | Pending, MaintenancePending, RenderFailed, ApplyFailed, AgentNotReady, EndpointNotServing, ServerTooOld |
Then check the logs of the monitoring extension's controller and of the instance's provider in everest-system.
Access control
With RBAC enabled, users need read on monitoring-destinations to use a destination in an instance, and create or update to manage destinations. Creating a destination that references a Secret, or changing the spec of one that does, also needs use on that Secret; see Secrets and credentials. Creating a destination lets the monitoring extension send a Secret from the namespace to the destination's address, so grant it only to users who are trusted with the namespace's Secrets. monitoring-classes and monitoring-bindings are read-only.
Monitoring endpoints (removed)
The Settings > Monitoring endpoints page and the MonitoringConfig resource of the previous monitoring model were removed in OpenEverest 2.0.0 Alpha 1. Use monitoring destinations instead: Settings > Monitoring destinations in the UI.