Ondrej Sika (sika.io) | ondrej@sika.io | course ->
Write me mail to ondrej@sika.io
- Kubernetes Training in Czech Republic
- Loki Training in Czech Republic (Log management in Grafana)
- Kubernetes Training in Europe & Middle East
- Loki Training in Europe & Middle East (Log management in Grafana)
- Demo ELK on Digital Ocean - https://github.com/ondrejsika/terraform-demo-elk
About Me - Ondrej Sika
Freelance DevOps Engineer, Consultant & Lecturer
- Complete DevOps Pipeline
- Open Source / Linux Stack
- Cloud & On-Premise
- Technologies: Git, Gitlab, Gitlab CI, Docker, Kubernetes, Terraform, Prometheus, ELK / EFK, Rancher, Proxmox, DigitalOcean, AWS
https://ondrej-sika.cz/devopslive
Feel free to star this repository or fork it.
If you found bug, create issue or pull request.
Also feel free to propose improvements by creating issues.
For sharing links & "secrets".
- Zoom Chat
- Slack - https://sikapublic.slack.com/
- Microsoft Teams
- https://sika.link/chat (tlk.io)
- Introduction to ELK
- Install Elasticsearch & Kibana
- Debian
- Using Docker
- Kubernetes (preferred)
- Filebeat
- Install
- Configuration
- Kibana
- Overview
- Discover
- Visualize
- Dashboard
- Management
- Elasticsearch
- Kibana
- Beats
Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack. Beats facilitate collecting, aggregating, and enriching your data and storing it in Elasticsearch. Kibana enables you to interactively explore, visualize, and share insights into your data and manage and monitor the stack. Elasticsearch is where the indexing, search, and analysis magic happens.
Visualize and analyze your data and manage all things Elastic Stack.
- https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
- https://www.elastic.co/guide/en/kibana/current/deb.html
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
sudo apt-get install apt-transport-https
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install elasticsearch
Initial node
Set cluter.name and listen on all interfaces (network.host) in /etc/elasticsearch/elasticsearch.yml
vim /etc/elasticsearch/elasticsearch.yml
sudo /bin/systemctl enable elasticsearch.service --now
Reset elastic user password
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic --batch
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
or join cluster
Set cluter.name in /etc/elasticsearch/elasticsearch.yml
vim /etc/elasticsearch/elasticsearch.yml
export TOKEN=
/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token $TOKEN
sudo /bin/systemctl enable elasticsearch.service --now
Install Kibana
sudo apt-get update && sudo apt-get install kibana
Listen on all interfaces (server.host) in /etc/kibana/kibana.yml
vim /etc/kibana/kibana.yml
Create token for Kibana (on ES node)
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
TOKEN=
/usr/share/kibana/bin/kibana-setup --enrollment-token $TOKEN
sudo /bin/systemctl enable kibana.service --now
HTTP CA is on /etc/elasticsearch/certs/http_ca.crt
You can try ES from curl
ELASTIC_PASSWORD=
curl --cacert /etc/elasticsearch/certs/http_ca.crt https://elastic:$ELASTIC_PASSWORD@127.0.0.1:9200
Simple single node installation for development (without auth)
cd examples/docker/elk_local
docker compose up -d
Simple single node installation with auth and https
cd examples/docker/elk_local_secure
docker compose up -d
and see Makefile
Source: ondrejsika/terraform-demo-elk
See:
- http://elk-docker.sikademo.com:9200
- http://elk-docker.sikademo.com:5601
- https://es.elk-docker.sikademo.com
- https://kb.elk-docker.sikademo.com
- Intro - https://www.elastic.co/elastic-cloud-kubernetes
- Docs / Tutorial - https://www.elastic.co/guide/en/cloud-on-examples/k8s/current/index.html
- Deploy ECK - https://www.elastic.co/guide/en/cloud-on-examples/k8s/current/k8s-deploy-eck.html
- Github - https://github.com/elastic/cloud-on-k8s
https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/install-using-helm-chart
helm install \
elastic-operator \
--repo https://helm.elastic.co \
eck-operator \
-n elastic-system \
--create-namespace \
--wait
kubectl apply -f ./examples/k8s/ns.yml
kubectl apply -f ./examples/k8s/elk_single_node
Wait until Elasticsearch and Kibana will be GREEN
kubectl get -f ./examples/k8s/elk_single_node
Get password for user elastic
kubectl -n elk get secret elk-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode; echo
or using slu:
slu eck elastic-password -n elk -e elk
See:
Test it:
export ELASTIC_PASSWORD=$(kubectl -n elk get secret main-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)
filebeat -c `pwd`/filebeat/filebeat-k8s-test.yml -e
kubectl get -f ./examples/k8s/elk_cluster
Wait until Elasticsearch and Kibana will be GREEN
kubectl get -f ./examples/k8s/elk_cluster
See:
- Provider - https://registry.terraform.io/providers/elastic/elasticstack/latest/docs
- User - https://registry.terraform.io/providers/elastic/elasticstack/latest/docs/resources/elasticsearch_security_user
- API Key - https://registry.terraform.io/providers/elastic/elasticstack/latest/docs/resources/elasticsearch_security_api_key
- Role - https://registry.terraform.io/providers/elastic/elasticstack/latest/docs/resources/elasticsearch_security_role
cd ./examples/terraform/users
Create main.auto.tfvars ...
terraform init
terraform apply
See: https://kb.k8s.sikademo.com/app/management/security/users
Docs | Quick Start Installation
brew install elastic/tap/filebeat-full
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.0.1-amd64.deb
sudo dpkg -i filebeat-9.0.1-amd64.deb
or using slu:
slu install-bin filebeat
Plaintext log generator
slu loggen
JSON log generator
slu loggen --json
Source: https://github.com/mingrammer/flog
Install on Mac
brew install mingrammer/flog/flog
Run
flog -d 100ms --loop
Source: https://pkg.go.dev/k8s.io/Kubernetes/test/images/logs-generator
Run in Docker
docker run -i \
-e "LOGS_GENERATOR_LINES_TOTAL=10" \
-e "LOGS_GENERATOR_DURATION=1s" \
gcr.io/google_containers/logs-generator:v0.1.1
Run in Kubernetes
kubectl run logs-generator \
--image=gcr.io/google_containers/logs-generator:v0.1.1 \
--restart=Never \
--env "LOGS_GENERATOR_LINES_TOTAL=1000" \
--env "LOGS_GENERATOR_DURATION=1m"
- Log
- Stdin
- Container
- Docker
- Syslog
- Kafka
Run slu loggen
slu loggen --log-file /tmp/default.log --log-prefix loggen-file
Run filebeat
filebeat -c $(pwd)/examples/filebeat/filebeat-input-log.yml -e
Run slu loggen & filebeat
slu loggen --log-prefix loggen-stdin | filebeat -c $(pwd)/examples/filebeat/filebeat-input-stdin.yml -e
Run some Docker container
docker run --name slu-loggen -d sikalabs/slu:v0.74.0 slu loggen --log-prefix loggen-container
docker run --name loop -d ondrejsika/infinite-counter
filebeat -c $(pwd)/examples/filebeat/filebeat-input-container.yml -e
- Elasticsearch
- Kafka
- File
- Console
slu loggen | filebeat -c $(pwd)/examples/filebeat/filebeat-output-file.yml -e
slu loggen | filebeat -c $(pwd)/examples/filebeat/filebeat-output-console.yml -e
Only logs
slu loggen | filebeat -c $(pwd)/examples/filebeat/filebeat-output-console.yml
Only message
slu loggen | filebeat -c $(pwd)/examples/filebeat/filebeat-output-console.yml | jq .message
cat examples/logs/multiline_python.txt | filebeat -c $(pwd)/examples/filebeat/filebeat-multiline-python.yml | jq .message
cat examples/logs/multiline_java.txt | filebeat -c $(pwd)/examples/filebeat/filebeat-multiline-java.yml | jq .message
- Conditions - https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html#conditions
Labels and Tags
echo x | filebeat -c $(pwd)/examples/filebeat/labels-and-tags.yml
Labels
echo x | filebeat -c $(pwd)/examples/filebeat/labels-and-tags.yml | jq .labels
Tags
echo x | filebeat -c $(pwd)/examples/filebeat/labels-and-tags.yml | jq .tags
echo hello | filebeat -c $(pwd)/examples/filebeat/filebeat-processor-host.yml | jq .host
echo hello | filebeat -c $(pwd)/examples/filebeat/filebeat-processor-host.yml | jq .host.hostname
echo hello | filebeat -c $(pwd)/examples/filebeat/filebeat-processor-host.yml | jq .host.os
slu loggen | filebeat -c $(pwd)/examples/filebeat/filebeat-processor-drop-fields.yml
slu loggen | filebeat -c $(pwd)/examples/filebeat/filebeat-processor-drop
B421
-event.yml
filebeat -c $(pwd)/examples/filebeat/filebeat-processor-docker.yml -e
- https://dissect-tester.jorgelbg.me/ - Dissect Tester
echo 'xx 2023/01/12 17:42:40 WARN A warning that should be ignored is usually at this level and should be actionable. (i=1)' | filebeat -c $(pwd)/examples/filebeat/filebeat-processor-dissect-2.yml | jq .dissect
Docs (List of Modules) | Docs (Configure Modules)
Filebeat modules simplify the collection, parsing, and visualization of common log formats.
filebeat -c $(pwd)/examples/filebeat/filebeat-module-traefik.yml -e
kubectl apply -f ./examples/k8s/filebeat/filebeat.yml
or install on cluster without ECK ES and send data to external ES
kubectl apply -f ./examples/k8s/filebeat/filebeat_ext.yml
Install Strimzi - Kafka Operator
kubectl create namespace kafka
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
See operator's pod
kubectl get pod -n kafka
Install Kafka Cluster
kubectl apply -f ./examples/k8s/kafka
See Kafka Cluster
kubectl get -n kafka kafka
Get Bootstrap Node
kubectl describe -f examples/k8s/kafka | grep "Bootstrap Servers"
Setup kaf - Kafka CLI
BOOTSTRAP_NODE=
Example
BOOTSTRAP_NODE=134.122.89.34:32473
kaf config add-cluster $BOOTSTRAP_NODE -b $BOOTSTRAP_NODE
kaf config use-cluster $BOOTSTRAP_NODE
Get nodes
kaf nodes
Get topics
kaf topics
Send logs to Kafka
export KAFKA_NODE=
Example
export KAFKA_NODE=134.122.89.34:31031
or
export KAFKA_NODE=$BOOTSTRAP_NODE
Run filebeat
slu loggen --log-prefix loggen-kafka | filebeat -c $(pwd)/examples/filebeat/filebeat-output-kafka.yml -e
Describe topic logs
kaf topic describe logs
Read logs from Kafka
filebeat -c $(pwd)/examples/filebeat/filebeat-input-kafka.yml
Read only message form Kafka logs
filebeat -c $(pwd)/examples/filebeat/filebeat-input-kafka.yml | jq -r '.message'
See groups
kaf groups
describe filebeat group
kaf group describe filebeat
ERROR
"and should be actionable"
agent.hostname:sika-mac
event.dataset:*
user_agent.os.name: Mac*
source.address: 176.114.249.139 and http.response.status_code: 500
http.response.status_code: 200 or http.response.status_code: 302
http.response.status_code: (200 or 302)
http.response.status_code:* and not http.response.status_code: (200 or 302)
- Announcing Elastic’s piped query language, ES|QL (blog post)
- ES|QL: The Elasticsearch Query Language (docs)
- Getting started with ES|QL
FROM logs-*-*
FROM logs-*-* | limit 10
FROM logs-*-* | WHERE container.name == "traefik-traefik-1" | limit 10
FROM logs-*-* | WHERE container.name == "traefik-traefik-1" | SORT @timestamp DESC | limit 10
That's it. Do you have any questions? Let's go for a beer!
- email: ondrej@sika.io
- web: https://sika.io
- twitter: @ondrejsika
- linkedin: /in/ondrejsika/
- Newsletter, Slack, Facebook & Linkedin Groups: https://join.sika.io
Do you like the course? Write me recommendation on Twitter (with handle @ondrejsika) and LinkedIn (add me /in/ondrejsika and I'll send you request for recommendation). Thanks.
Wanna to go for a beer or do some work together? Just book me :)
- ArgoCD with ECK Example - https://github.com/sika-training-examples/2025-05-10_argocd-apps-cez-example
- ELK Terraform Example - https://github.com/sika-training-examples/2025-04-11_cez-elk-terraform-example
- Terraform Example - https://github.com/sika-training-examples/2024-11-05_elastic-terraform-example
- Elastic Stack managed by Terraform - https://github.com/sika-training-examples/2024-02-22-rpc-elastic-terraform-example
- Filebeat examples - https://github.com/sika-training-examples/2024-02-21-rpc-elk-examples