Command Palette

Search for a command to run...

GitHubGitHub

ELK Stack Installation Guide

A comprehensive guide for installing and configuring the ELK Stack (Elasticsearch, Logstash, Kibana)

The ELK Stack is a collection of three open-source products: Elasticsearch, Logstash, and Kibana. It allows you to collect any data from any source, search, analyze, and visualize it in real time.

Prerequisites

  • A system running Linux, macOS, or Windows
  • Administrative/sudo privileges
  • Terminal/Command Prompt access
  • Minimum 8GB RAM (16GB recommended)
  • Java 11 or later installed
  • Sufficient disk space (minimum 10GB)
  • Internet connection for downloading packages

Installation Instructions

Ubuntu/Debian Installation

# Install Java
sudo apt update
sudo apt install default-jre -y
 
# Add Elastic GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
 
# Add Elastic repository
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | \
  sudo tee /etc/apt/sources.list.d/elastic-8.x.list
 
# Update package list
sudo apt update
 
# Install Elasticsearch
sudo apt install elasticsearch -y
 
# Start Elasticsearch
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
 
# Install Kibana
sudo apt install kibana -y
 
# Start Kibana
sudo systemctl enable kibana
sudo systemctl start kibana
 
# Install Logstash
sudo apt install logstash -y
 
# Start Logstash
sudo systemctl enable logstash
sudo systemctl start logstash

Post-Installation Configuration

Elasticsearch Configuration (elasticsearch.yml)

# Network settings
network.host: 0.0.0.0
http.port: 9200
 
# Discovery settings
discovery.type: single-node
 
# Memory settings
bootstrap.memory_lock: true
 
# Security settings
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

Kibana Configuration (kibana.yml)

# Server settings
server.port: 5601
server.host: "0.0.0.0"
 
# Elasticsearch connection
elasticsearch.hosts: ["http://localhost:9200"]
 
# Security settings
elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password"

Logstash Pipeline Configuration (logstash.conf)

input {
  beats {
    port => 5044
  }
  
  tcp {
    port => 5000
    codec => json
  }
}
 
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
    
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

Verification

Elasticsearch

  1. Check service status:
sudo systemctl status elasticsearch
  1. Test the REST API:
curl -X GET "localhost:9200/"

Kibana

  1. Check service status:
sudo systemctl status kibana
  1. Access web interface:
http://localhost:5601

Logstash

  1. Check service status:
sudo systemctl status logstash
  1. Test configuration:
sudo -u logstash /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/logstash.conf

Common Configuration Options

Setting Up SSL/TLS

  1. Generate certificates:
# Generate CA
bin/elasticsearch-certutil ca
 
# Generate certificates
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
  1. Configure Elasticsearch:
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

Configuring Authentication

  1. Set up passwords:
# For Elasticsearch
bin/elasticsearch-setup-passwords interactive
 
# For Kibana (in kibana.yml)
elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password"

Troubleshooting

Common Issues

  1. Elasticsearch Won't Start

    • Check Java version compatibility
    • Verify system resources (memory, disk space)
    • Check logs: sudo journalctl -u elasticsearch
    • Verify file permissions

  2. Kibana Can't Connect to Elasticsearch

    • Verify Elasticsearch is running
    • Check connection settings in kibana.yml
    • Verify network connectivity
    • Check authentication credentials

  3. Logstash Pipeline Issues

    • Validate configuration syntax
    • Check input/output connectivity
    • Monitor resource usage
    • Review pipeline logs

Security Considerations

  1. Change default passwords
  2. Enable X-Pack security
  3. Implement SSL/TLS encryption
  4. Use role-based access control
  5. Regular security updates
  6. Network segmentation
  7. Audit logging
  8. Data encryption at rest

Additional Resources

Nagios

A comprehensive guide for installing and configuring Nagios monitoring system

Jenkins

Comprehensive guide for installing Jenkins CI/CD server across different operating systems with package manager specific commands.