Command Palette

Search for a command to run...

GitHubGitHub

Prometheus Installation Guide

Install Prometheus 3.15 with node_exporter 1.12, configure scraping and alerting in the 3.x format, and understand the breaking changes when upgrading from 2.x.

Prometheus is an open-source monitoring and alerting toolkit. It scrapes metrics, stores them as time series, and evaluates rules against them.

System Requirements

  • A 64-bit operating system: Linux, macOS or Windows
  • No runtime dependencies — Prometheus is a single Go binary
  • 2 GB RAM is enough for a small installation; disk usage is roughly 1–2 GB per retained week per million active series
  • Ports 9090 (web UI and API) and 9091 (node_exporter)

Installation Instructions

Linux (recommended)

# Set the version
PROM_VERSION=3.15.0
 
# Create the user and directories
sudo useradd --system --no-create-home --shell /bin/false prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
 
# Download the release
cd /tmp
curl -LO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
tar xzf "prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
 
# Install the binaries
sudo install -o root -g root -m 0755 "prometheus-${PROM_VERSION}.linux-amd64/prometheus" /usr/local/bin/prometheus
sudo install -o root -g root -m 0755 "prometheus-${PROM_VERSION}.linux-amd64/promtool" /usr/local/bin/promtool
 
# Clean up
rm -rf "prometheus-${PROM_VERSION}.linux-amd64" "prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
 
# Verify
prometheus --version
promtool --version

On ARM64 use prometheus-${PROM_VERSION}.linux-arm64.tar.gz.

Create the systemd service

sudo tee /etc/systemd/system/prometheus.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
 
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus/ \
    --storage.tsdb.retention.time=15d \
    --storage.tsdb.retention.size=50GB \
    --web.enable-lifecycle
Restart=always
RestartSec=5
 
[Install]
WantedBy=multi-user.target
EOF
 
# Permissions
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
 
# Start it
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus

Configuration

prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
 
  # Prometheus 3 validates names as UTF-8 by default. Set this to "legacy" to keep
  # the Prometheus 2.x behaviour of rejecting names with dots or dashes.
  metric_name_validation_scheme: utf8
 
rule_files:
  - /etc/prometheus/rules/*.yml
 
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
 
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']
 
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
      - role: endpoints
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    # Prometheus 3 fails the scrape when the target does not send a supported
    # Content-Type. This is the safe fallback for older exporters.
    fallback_scrape_protocol: PrometheusText0.0.4

Authentication for Scrape Targets

Prometheus 3 uses the authorization block. basic_auth still works, but authorization covers both basic auth and bearer tokens:

scrape_configs:
  - job_name: api
    authorization:
      type: Basic
      credentials: <base64 of username:password>
    static_configs:
      - targets: ['api.example.com:9100']
 
  - job_name: cloud-metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/token
    static_configs:
      - targets: ['metrics.example.com:443']
    tls_config:
      ca_file: /etc/prometheus/ca.pem

Alerting Rules

Create /etc/prometheus/rules/alerts.yml:

groups:
  - name: instance-health
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.job }} on {{ $labels.instance }} has been down for 5 minutes."
 
      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
# Validate the configuration before reloading
sudo promtool check config /etc/prometheus/prometheus.yml
sudo promtool check rules /etc/prometheus/rules/alerts.yml
 
# Reload without restarting
sudo systemctl reload prometheus
# or: curl -X POST http://localhost:9090/-/reload

Adding node_exporter

# Create the user
sudo useradd --system --no-create-home --shell /bin/false node_exporter
 
# Download node_exporter 1.12.1
cd /tmp
curl -LO "https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz"
tar xzf "node_exporter-1.12.1.linux-amd64.tar.gz"
sudo install -o root -g root -m 0755 "node_exporter-1.12.1.linux-amd64/node_exporter" /usr/local/bin/node_exporter
rm -rf node_exporter-1.12.1*
 
# Create the systemd service
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<'EOF'
[Unit]
Description=Node Exporter
After=network.target
 
[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=5
 
[Install]
WantedBy=multi-user.target
EOF
 
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Verification

# The service is running
sudo systemctl status prometheus
 
# The targets are up
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
 
# The metrics endpoint serves data
curl -s http://localhost:9090/metrics | head
 
# Query it
curl -s 'http://localhost:9090/api/v1/query?query=up' | jq .
 
# Check the TSDB
promtool tsdb analyze /var/lib/prometheus

The web UI at http://localhost:9090 has the Status → Targets page — every target should be green there.

Common Operations

# Reload the configuration
curl -X POST http://localhost:9090/-/reload
sudo systemctl reload prometheus
 
# Check which flags are in effect
curl -s http://localhost:9090/api/v1/status/flags | jq .
 
# Check the runtime information
curl -s http://localhost:9090/api/v1/status/runtimeinfo | jq .
 
# TSDB analysis — find the biggest cardinality offenders
promtool tsdb analyze /var/lib/prometheus --top 10
 
# Download a TSDB block for offline analysis
promtool tsdb create-blocks-from rules /path/to/prometheus /path/to/output

Retention and Storage

# In the systemd unit
ExecStart=/usr/local/bin/prometheus \
    --storage.tsdb.path=/var/lib/prometheus/ \
    --storage.tsdb.retention.time=30d \
    --storage.tsdb.retention.size=80GB

Retention is applied when either limit is reached, so set both a time and a size limit to avoid filling the disk.

Troubleshooting

Upgrading from Prometheus 2

The changes in Prometheus 3 that break existing setups:

ChangeWhat to do
UTF-8 metric names allowed by defaultSet metric_name_validation_scheme: legacy to keep the old validation, or rename the affected metrics
le and quantile labels normalised to float formUpdate alerts and dashboards that reference le="1" to le="1.0"
Stricter Content-Type handling on scrapeFix the exporters, or set fallback_scrape_protocol per job
Alertmanager v1 API removedUpgrade to Alertmanager 0.16+ and use api_version: v2
Range selectors are now left-openRewrite subqueries that relied on the extra boundary sample, for example foo[1m:1m] to foo[2m:1m]
TSDB format changedOnly Prometheus 2.55+ can read a 3.x data directory
Log format switched to log/slogUpdate any log parsers that expected the old format
# 1. Upgrade to 2.55 first and confirm everything works
# 2. Stop Prometheus
sudo systemctl stop prometheus
 
# 3. Install the new release
sudo tar xzf prometheus-3.15.0.linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin
 
# 4. Validate the configuration with the new binary
sudo promtool check config /etc/prometheus/prometheus.yml
 
# 5. Start it again
sudo systemctl start prometheus

Best Practices

  1. Use promtool check config in CI so a broken configuration never reaches the server.
  2. Set both retention limits — time and size — so a spike in series cannot fill the disk.
  3. Record rules for expensive queries instead of running them on every dashboard refresh.
  4. Use metric_relabel_configs to drop high-cardinality labels at scrape time.
  5. Enable --web.enable-lifecycle so you can reload instead of restarting.
  6. Use the authorization block with a credentials_file rather than putting secrets in the configuration.
  7. Run Prometheus behind a load balancer with two replicas and a remote-write target for durability.
  8. Monitor Prometheus itself — the prometheus_* metrics tell you when scraping or the TSDB is in trouble.

Next Steps

Overview

Install and configure Prometheus, Grafana, the ELK stack and Nagios with scrape configurations, dashboards, alerting rules and centralised logging you can rely on in production.

Grafana

Install Grafana 13.2 on Linux, Windows, macOS, Docker and Kubernetes, connect Prometheus as a data source, and size the deployment for production.