Command Palette

Search for a command to run...

GitHubGitHub

ELK Stack Installation Guide

Install the Elastic Stack 9.5 — Elasticsearch, Logstash and Kibana — from the official apt and rpm repositories, enable TLS security and ship your first logs.

The Elastic Stack is three products that work together: Elasticsearch stores and searches your data, Logstash ingests and transforms it, and Kibana visualises it.

System Requirements

  • 64-bit Linux, macOS or Windows (Linux is the supported production platform)
  • At least 4 GB RAM for Elasticsearch, 8 GB or more for a production node
  • Java is bundled with the packages — you do not need to install a JDK
  • vm.max_map_count set to 262144 (Elasticsearch refuses to start otherwise)
  • Ports 9200 (Elasticsearch), 9300 (transport), 5044 (Logstash Beats input), 9600 (Logstash monitoring), 5601 (Kibana)
  • 20 GB free disk for the data volume

Installation Instructions

Ubuntu / Debian (apt)

# 1. Import the Elastic signing key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch \
  | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
 
# 2. Add the 9.x repository
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
 
# 3. Install the stack
sudo apt-get update
sudo apt-get install -y elasticsearch logstash kibana
 
# 4. Raise the memory map limit
sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | sudo tee /etc/sysctl.d/99-elastic.conf
sudo sysctl --system
 
# 5. Start the services
sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch
sudo systemctl enable --now kibana
sudo systemctl enable --now logstash

Security Setup

Elasticsearch 9 configures security automatically on first start: it generates TLS certificates for the transport and HTTP layers, creates an enrollment token for Kibana and enables the elastic superuser.

1. Set the elastic password

Because the service runs under systemd, the password is not printed in the terminal. Set it with the reset tool:

cd /usr/share/elasticsearch
sudo -u elasticsearch bin/elasticsearch-reset-password -u elastic
# Store it for the rest of the session
export ELASTIC_PASSWORD='the-password-you-just-set'

2. Verify Elasticsearch

curl --cacert /etc/elasticsearch/certs/http_ca.crt \
  -u "elastic:$ELASTIC_PASSWORD" https://localhost:9200
{
  "name": "Cp8oag6",
  "cluster_name": "elasticsearch",
  "version": { "number": "9.5.5" },
  "tagline": "You Know, for Search"
}

3. Connect Kibana

# Get the enrollment token from Elasticsearch
cd /usr/share/elasticsearch
sudo -u elasticsearch bin/elasticsearch-create-enrollment-token --ca-fingerprint /etc/elasticsearch/certs/http_ca.crt
# On the Kibana host, start Kibana with the token
sudo systemctl stop kibana
sudo /usr/share/kibana/bin/kibana \
  --elasticsearch.hosts=https://localhost:9200 \
  --elasticsearch.ssl.certificateAuthorities=/etc/kibana/config/http_ca.crt \
  --elasticsearch.username=kibana_system \
  --elasticsearch.password=<enrollment-token> \
  --encryption.key="$(openssl rand -base64 32)" \
  &

The token is valid for 30 minutes. Kibana writes the security configuration to /etc/kibana/kibana.yml, so this only has to be done once.

Logstash Pipeline

Create /etc/logstash/pipeline/logstash.conf:

input {
  beats {
    port => 5044
  }
}
 
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      tag_on_failure => ["_syslog_parse_failure"]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "ISO8601" ]
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    user => "elastic"
    password => "${ELASTIC_PASSWORD}"
    ssl_certificate => "/etc/elasticsearch/certs/http_ca.crt"
    index => "logs-%{+YYYY.MM.dd}"
  }
 
  stdout {
    codec => rubydebug
  }
}
# Validate the pipeline before reloading
sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
 
# Reload
sudo systemctl reload logstash
sudo journalctl -u logstash -f

Kibana Dashboards

  1. Open http://localhost:5601
  2. Sign in with elastic and the password you set
  3. Stack Management → Index Patterns → create logs-*
  4. Discover → search your data
  5. Dashboard → New → add panels
# Check that Kibana is healthy
curl -s http://localhost:5601/api/status | jq .

Multi-Node Cluster

For a real cluster you need at least three master-eligible nodes:

# On the first node
sudo tee -a /etc/elasticsearch/elasticsearch.yml > /dev/null <<'EOF'
cluster.name: my-cluster
node.name: node-1
network.host: 0.0.0.0
transport.host: 0.0.0.0
discovery.seed.hosts: ["10.0.0.11", "10.0.0.12", "10.0.0.13"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
EOF
# On the other nodes, enrol them with a token (valid for 30 minutes)
sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token>

Troubleshooting

Best Practices

  1. Enable the security features — TLS, the elastic superuser and role-based access are on by default in 9.x. Never disable them in production.
  2. Use an ILM policy so old indices are deleted automatically:
    curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u "elastic:$ELASTIC_PASSWORD" \
      -X PUT "https://localhost:9200/_ilm/policy/logs_policy" -H 'Content-Type: application/json' -d '{
        "policy": {
          "phases": {
            "hot": { "actions": { "rollover": { "max_primary_shard_size": "50gb" } } },
            "delete": { "min_age": "30d", "actions": { "delete": {} } }
          }
        }
      }'
  3. Take snapshots to a shared repository — a snapshot repository is not a backup unless it is off-cluster.
  4. Ship logs with Filebeat or Elastic Agent rather than running Logstash on every host.
  5. Never put credentials in logstash.conf — use a keystore or environment variables.
  6. Pin the version across all three components; Elasticsearch, Kibana and Logstash must be on compatible versions.
  7. Monitor the JVM heap — a heap over 75% means the node is under pressure.

Next Steps