VictoriaMetrics — High-Performance Metrics Storage, PromQL-Compatible, and Cost Reduction vs Prometheus

When Prometheus Runs Out of Room

A single Prometheus server is wonderful until it is not. It scrapes, it stores, it answers PromQL — all in one process — and for one team that is exactly the right amount of machinery. The trouble starts at scale: memory grows with active series, retention is bounded by one disk, and there is no built-in horizontal path.

VictoriaMetrics is a time series database built to be the storage and query layer that Prometheus was never trying to be. It speaks the Prometheus remote-write protocol and the Prometheus query API, so it slots in behind your existing stack, then keeps working at cardinalities and retention windows that would sink a single Prometheus.

This guide walks the whole surface: the single binary, ingestion, the MetricsQL query language, scraping with vmagent, alerting with vmalert, the cluster version, and the deduplication and downsampling levers that make long retention affordable.

Why Teams Reach For It

The pitch is resource efficiency. VictoriaMetrics’ own documentation lists a set of “prominent features” comparing it to the incumbents, and the numbers are worth quoting exactly rather than paraphrasing.

• Memory. It claims to use “up to 7x less RAM than Prometheus, Thanos or Cortex” when handling millions of unique series.
• Storage.The docs state “up to 7x less storage space is required compared to Prometheus, Thanos or Cortex” thanks to high compression.
• Cardinality.It is “optimized for time series with high churn rate” and includes a series limiter for high-cardinality workloads.

Treat these as vendor benchmarks, not laws of physics — your mileage depends on cardinality, scrape interval, and query mix. The honest takeaway is that VictoriaMetrics is engineered for the exact failure modes that push a single Prometheus over the edge, and the numbers are big enough to be worth measuring on your own data.

The Single Binary

Start with the single-node build. It is one static binary, victoria-metrics, published as the victoriametrics/victoria-metrics Docker image. There is no external dependency — no Kafka, no ZooKeeper, no separate metadata store — which is a large part of why it is pleasant to operate.

It listens for the querying and ingestion API on port 8428 by default. Two flags matter on day one: -storageDataPath (where data lives, defaulting to a victoria-metrics-data directory in the working directory) and -retentionPeriod (how long to keep it, defaulting to 1 month).

# Run the single-node server with Docker,
# persisting data and keeping 12 months of history.
docker run -it --rm \
  -p 8428:8428 \
  -v $(pwd)/vmdata:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest \
  -storageDataPath=/victoria-metrics-data \
  -retentionPeriod=12

The -retentionPeriod value is interpreted in months by default (so 12 means twelve months); the documentation notes the minimum retention is 24h, which you can write as 1d. Everything else has a sane default, so this is genuinely enough to get a working, durable metrics store running.

Getting Data In

VictoriaMetrics is deliberately promiscuous about ingestion. It accepts the Prometheus remote_write protocol, the Prometheus exposition format, InfluxDB line protocol over HTTP, TCP and UDP, the Graphite plaintext protocol, OpenTSDB, DataDog, OpenTelemetry, CSV, and its own JSON and native binary formats.

The most common path is Prometheus remote write. If you already run Prometheus, add a remote_write block pointing at the VictoriaMetrics /api/v1/write endpoint and you are dual-writing immediately — no re-instrumentation, no data-model translation.

# prometheus.yml — forward everything to VictoriaMetrics.
global:
  scrape_interval: 15s

remote_write:
  - url: http://victoria-metrics:8428/api/v1/write

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]

For teams migrating off InfluxDB, the same server accepts line protocol directly, so a Telegraf agent can write to VictoriaMetrics with only a URL change. This multi-protocol front door is what lets it act as a single consolidation point for a heterogeneous estate.

# Send a point via InfluxDB line protocol over HTTP.
curl -d 'measurement,tag=value field=1.5' \
  -X POST 'http://victoria-metrics:8428/write'

# ...then read it back through the Prometheus query API.
curl 'http://victoria-metrics:8428/api/v1/query?query=measurement_field'

Querying — PromQL, and Then Some

Queries go through the familiar Prometheus handlers: /api/v1/query for instant queries and /api/v1/query_range for range queries, both of which also work under a /prometheus prefix. This is why Grafana, alerting tools, and your own scripts need no changes.

The query language is MetricsQL, which the docs describe as “backwards-compatible with PromQL.” Existing PromQL expressions run unchanged, and MetricsQL then adds features that remove common PromQL papercuts.

• WITH templates. Factor repeated sub-expressions out of a large query, the way you would with a local variable.
• default operator. q1 default q2 fills gaps in q1 with values from q2 — no more or vector(0) gymnastics.
• keep_metric_names. A modifier that stops functions from dropping the metric name, so results stay labelled.
• if / ifnot. Filter one series set by the presence of another, inline.

# MetricsQL: a WITH template plus keep_metric_names.
WITH (
  cpu = rate(node_cpu_seconds_total{mode!="idle"}[5m])
)
(sum(cpu) by (instance) keep_metric_names) default 0

The docs are explicit that there are intentional behavioural differences too — notably how rate and increase treat samples before the lookbehind window, with no result extrapolation. When you port dashboards, read that section rather than assuming byte-for-byte identical output. For a broader take on building alerting and dashboards on top of a metrics store, our note on observability-driven development covers SLOs and alert design that this query layer feeds directly.

vmagent — Scraping and Shipping

You do not have to keep Prometheus around just to scrape. vmagent is a tiny agent that discovers and scrapes Prometheus targets, relabels and filters, and remote-writes the result to one or more destinations. It is a drop-in replacement for Prometheus’ scraping half.

It reads a standard Prometheus scrape config via -promscrape.config — honouring the global and scrape_configs sections — and sends data to each -remoteWrite.url you pass. It defaults to port 8429.

# vmagent scrapes prometheus.yml and fans out to two backends.
/path/to/vmagent \
  -promscrape.config=/etc/vmagent/prometheus.yml \
  -remoteWrite.url=http://vm-a:8428/api/v1/write \
  -remoteWrite.url=http://vm-b:8428/api/v1/write

The feature that saves incidents is on-disk buffering. If a remote storage becomes unavailable, vmagent buffers collected metrics at -remoteWrite.tmpDataPath and replays them once the connection returns, with the buffer size bounded by -remoteWrite.maxDiskUsagePerURL. A backend restart no longer means a hole in your graphs.

vmalert — Rules and Alerts

Alerting is handled by vmalert, which executes alerting and recording rules against a datasource and is, by design, compatible with Prometheus rule syntax. It queries -datasource.url, sends firing alerts to Alertmanager at -notifier.url, and persists recording-rule results via -remoteWrite.url.

Rules are loaded from -rule (which accepts wildcards and even HTTP URLs), and vmalert serves HTTP on port 8880 by default. The rule file is the Prometheus format you already know — groups of alert/expr/for and record/expr entries.

# alerts.yml — plain Prometheus rule format, run by vmalert.
groups:
  - name: node
    rules:
      - alert: HighCPU
        expr: >
          100 * (1 - avg by (instance)
          (rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CPU above 90% on {{ $labels.instance }}"

# Wire vmalert to the database and to Alertmanager.
/path/to/vmalert \
  -rule=/etc/vmalert/alerts.yml \
  -datasource.url=http://victoria-metrics:8428 \
  -notifier.url=http://alertmanager:9093 \
  -remoteWrite.url=http://victoria-metrics:8428

Because it targets Alertmanager, vmalert drops into a routing tree you may already run for logs and other signals. If you are consolidating alerting across metrics and logs, our walkthrough of Grafana Loki with Alertmanager shows the notification side of the same picture.

The Cluster Version

When one node is not enough, the cluster version splits the single binary into three stateless-plus-stateful roles that scale independently.

• vmstorage stores raw data and answers range/label queries. Default port 8482. This is the stateful tier.
• vminsert accepts ingested data and spreads it across vmstorage nodes by consistent hashing. Default port 8480.
• vmselect runs queries by fetching from all configured vmstorage nodes. Default port 8481.

Both vminsert and vmselect are told where the storage nodes live through the -storageNode flag. Scale reads by adding vmselect replicas, scale writes by adding vminsert replicas, and scale capacity by adding vmstorage nodes.

# Minimal three-node cluster (one of each), same host.
vmstorage -retentionPeriod=12 -storageDataPath=/vmdata

vminsert  -storageNode=127.0.0.1:8400
vmselect  -storageNode=127.0.0.1:8401

The cluster also brings native multitenancy. Writes go to /insert/<accountID>/... on vminsert and reads to /select/<accountID>/prometheus/... on vmselect, where accountID may be written as accountID:projectID. Durability is tuned with -replicationFactor. This tenant-per-URL model is a clean way to isolate teams or customers on shared infrastructure — a pattern we explore more generally in multi-tenant architecture.

Deduplication and Downsampling

Two features exist specifically to bound cost. The first is deduplication: with high-availability Prometheus pairs or redundant vmagents, you receive two nearly identical samples for every scrape. Setting -dedup.minScrapeInterval keeps a single sample per interval, roughly halving storage for HA setups.

# Keep one sample per 15s even when two HA scrapers write.
victoria-metrics \
  -storageDataPath=/victoria-metrics-data \
  -dedup.minScrapeInterval=15s

The second is downsampling, configured with -downsampling.period in offset:interval form — for example, keep raw resolution recently, then thin older data to one point per 5 minutes. It is the right tool for multi-year retention where per-second detail on year-old data is worthless.

# Thin data older than 30 days to 5m, older than 1y to 1h.
-downsampling.period=30d:5m
-downsampling.period=1y:1h

A Production Checklist

• Pin the image tag. Deploy a specific version, not latest, and read the changelog before upgrades — defaults and flags do move between releases.
• Start single-node. One binary handles far more than most teams expect. Add the cluster only when a single vmstorage cannot keep up.
• Buffer with vmagent. Put vmagent in front so a backend restart replays from disk instead of dropping samples.
• Deduplicate HA pairs. If you run redundant scrapers, set -dedup.minScrapeInterval to the scrape interval.
• Bound cardinality. Watch active series and use relabeling to drop labels you never query — cardinality, not raw volume, is what hurts.
• Benchmark the claims. The 7x figures are vendor benchmarks; measure RAM and disk on your own traffic before committing capacity plans.

VictoriaMetrics’ appeal is that it changes almost nothing about how your monitoring looks — the same remote_write, the same query API, the same PromQL, the same Alertmanager — while changing a great deal about what it costs to run at scale. You keep the Prometheus ecosystem and trade a single overloaded process for a database engineered for cardinality, compression, and long retention.

Metrics are one signal of three. VictoriaMetrics stores the numbers, but traces and logs complete the picture — pair it with OpenTelemetry for unified traces, metrics, and logs and distributed tracing via Jaeger to correlate a latency spike in a trace with the resource pressure your VictoriaMetrics dashboards already show.