Elastic Security turns the open-source ELK stack (Elasticsearch + Kibana) into a full SIEM: log storage, a detection engine with 1,000-plus prebuilt rules, EDR through Elastic Defend, and a hunting workbench — all on the free Basic tier. This guide installs and configures Elastic Stack 9.5 three ways, enrolls agents from Windows and Linux, and gets you from an empty cluster to a firing detection rule.
You will pick a deployment method (native package, Docker Compose, or a one-command trial), secure the cluster, stand up Fleet, ship Windows Event Logs and Linux auditd, then install and tune detection rules. Every command below is copy-pasteable against a real 9.5 cluster.
🛡️
01 — What Elastic Security Is
PHASE 01 / 11
Elastic Security is the SIEM and endpoint-security solution built on top of the Elastic Stack. Three components do the work: Elasticsearch stores and indexes every event; Kibana hosts the Security app — alerts, Timeline, cases and rules; and Elastic Agent, managed centrally by Fleet, collects logs and runs the Elastic Defend endpoint sensor. The SIEM, detection engine, and prebuilt rules are free on the Basic licence — you only pay for advanced machine learning and entity analytics.
Data lands in the Elastic Common Schema (ECS), a normalised field set that lets one detection rule match Windows, Linux, cloud, and network events without per-source rewriting. That schema is the reason a defender should care: rules, dashboards, and hunts written once run everywhere the agent reaches.
🔎
Log SIEM
Central store for Windows, Linux, firewall, cloud and DNS logs, all normalised to ECS and searchable in near real time.
🚨
Detection Engine
1,000+ prebuilt, MITRE ATT&CK-mapped rules plus custom query, EQL, threshold, and ML rule types that generate alerts on a schedule.
💻
Endpoint (EDR)
Elastic Defend adds process, file, network and malware telemetry with prevention — turning agents into an EDR sensor, no extra product.
🧭
Threat Hunting
Timeline, ES|QL, and osquery-on-demand let analysts pivot through raw events and build investigations into shareable cases.
The whole SIEM feature set — detections, Timeline, cases, and Elastic Defend — runs on the free Basic tier. You do not need a Platinum trial to build a working home SOC; you only lose cross-cluster ML anomaly jobs and the built-in threat-intel entity store.
📋
02 — Prerequisites & Requirements
PHASE 02 / 11
Elasticsearch is a JVM service that memory-maps its indices, so RAM and the vm.max_map_count kernel setting matter more than CPU. Below is a realistic single-node lab spec; scale RAM first when ingest grows.
| Resource | Minimum | Recommended |
|---|
| CPU | 2 vCPU | 4+ vCPU |
| RAM | 4 GB tight | 8–16 GB lab |
| Disk | 50 GB SSD | 100+ GB SSD (logs grow fast) |
| Host OS | Ubuntu 22.04 / Debian 12 | Ubuntu 24.04 LTS |
| JVM heap | 2 GB | ≤ 50% of RAM, max 31 GB |
| Browser | Modern Chromium / Firefox | Latest for Kibana 9.5 |
Ports and components you will open and talk to across this guide:
| Component | Port | Role |
|---|
| Elasticsearch (HTTP) | 9200 | REST API, ingest, search — TLS on by default in 9.x |
| Elasticsearch (transport) | 9300 | Node-to-node in a multi-node cluster |
| Kibana | 5601 | Web UI — Security app lives here |
| Fleet Server | 8220 | Agent enrollment and policy check-in |
| Elastic Agent | outbound | Ships to Elasticsearch/Fleet; no inbound port needed |
Elasticsearch 9.x runs a hard bootstrap check: if vm.max_map_count is below 262144 the node refuses to start in production mode. Set it before the first launch (Step 1 and Step 7 below) or the service will die immediately.
Run Kibana and Elasticsearch on the same major and minor version. Kibana 9.5 will not connect to Elasticsearch 9.4. Pin an explicit version string in every install command rather than trusting latest.
📦
03 — Method 1: Native Install (Ubuntu / Debian)
PHASE 03 / 11
The APT package install is the most production-like path: systemd-managed services, security enabled automatically, and a clean upgrade story. Do this on a fresh Ubuntu 24.04 host.
1
Prepare the OS & kernel
host prep
Update packages, install the HTTPS transport helper, and raise the memory-map limit persistently so it survives reboots.
$ sudo apt update && sudo apt install -y apt-transport-https gnupg curl
# raise mmap limit now and make it permanent
$ sudo sysctl -w vm.max_map_count=262144
$ echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-elastic.conf
2
Add the Elastic 9.x APT repository
repo
Import the signing key into its own keyring and register the 9.x repo. Elastic keeps a separate repo per major version.
$ curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch \
| sudo gpg --dearmor -o /usr/share/keyrings/elastic-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/elastic-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" \
| sudo tee /etc/apt/sources.list.d/elastic-9.x.list
$ sudo apt update
3
Install & start Elasticsearch
elasticsearch
Install the package, then watch the install output — on first start Elasticsearch prints the generated elastic superuser password and TLS certificates are auto-created. Enable and start the service under systemd.
$ sudo apt install -y elasticsearch
$ sudo systemctl enable --now elasticsearch
# confirm it is listening on 9200 (TLS)
$ sudo systemctl status elasticsearch --no-pager
The auto-generated elastic password is shown once during package install. If you missed it, do not reinstall — reset it in the next step instead.
4
Capture credentials & a Kibana token
auth
Reset the superuser password (captures it cleanly) and mint a one-time enrollment token that ties Kibana to this cluster with the right certificates.
$ sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
# -> prints: New value: <SAVE-THIS-PASSWORD>
$ sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
# -> prints a long base64 enrollment token for Kibana
# smoke test the API with the new password (accept the self-signed CA)
$ curl -sk -u elastic https://localhost:9200 | grep cluster_name
Install from the same repo. If Kibana runs on a different host, bind it to a reachable interface in /etc/kibana/kibana.yml with server.host.
$ sudo apt install -y kibana
# optional: expose Kibana beyond localhost
$ echo 'server.host: "0.0.0.0"' | sudo tee -a /etc/kibana/kibana.yml
$ sudo systemctl enable --now kibana
6
Enroll Kibana & verify
enroll
Browse to Kibana, paste the enrollment token from Step 4, then supply the 6-digit verification code the CLI prints. This completes the TLS handshake between Kibana and Elasticsearch.
# generate the browser verification code
$ sudo /usr/share/kibana/bin/kibana-verification-code
# then open http://SERVER_IP:5601 and paste token + code
Native install complete when
- systemctl shows elasticsearch and kibana both active (running)
- curl to https://localhost:9200 returns the cluster_name JSON
- You can log in to Kibana at :5601 as the elastic user
- Security app loads under the ☰ menu → Security → Overview
Keep the auto-generated CA at /etc/elasticsearch/certs/http_ca.crt. You will hand this file to Elastic Agent and to any external forwarder so it trusts the cluster without disabling verification.
🐳
04 — Method 2: Docker Compose
PHASE 04 / 11
Compose is the fastest repeatable single-node lab: one file describes Elasticsearch and Kibana, a short-lived setup container generates certificates and sets the kibana_system password, and everything comes up with one command.
7
Install Docker & set the kernel limit
docker prep
Docker containers share the host kernel, so vm.max_map_count is set on the host, not in the image. Install the Compose plugin alongside the engine.
$ curl -fsSL https://get.docker.com | sudo sh
$ sudo sysctl -w vm.max_map_count=262144
$ mkdir elastic-siem && cd elastic-siem
8
Create the .env file
config
Keep versions and passwords out of the compose file. Set strong passwords — the setup container uses KIBANA_PASSWORD for the internal kibana_system account.
.env
STACK_VERSION=9.5.0
ELASTIC_PASSWORD=ChangeMe_Elastic_2026!
KIBANA_PASSWORD=ChangeMe_Kibana_2026!
CLUSTER_NAME=cyberhawk-siem
ES_PORT=9200
KIBANA_PORT=5601
ES_MEM_LIMIT=2147483648
LICENSE=basic
9
Write docker-compose.yml
compose
The setup service builds the CA and node certificate, waits for Elasticsearch to be healthy, then sets the Kibana service-account password before exiting. Elasticsearch and Kibana mount the shared cert volume.
docker-compose.yml
services:
setup:
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes: [certs:/usr/share/elasticsearch/config/certs]
user: "0"
command: >
bash -c '
if [ ! -f config/certs/ca.zip ]; then
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f config/certs/certs.zip ]; then
echo -e "instances:\n - name: es01\n dns: [es01,localhost]\n ip: [127.0.0.1]" > config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip
--in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
chown -R 1000:0 config/certs;
until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing auth"; do sleep 3; done;
curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD}
-H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password
-d "{\"password\":\"${KIBANA_PASSWORD}\"}";
echo "setup done";
'
es01:
depends_on: [setup]
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes: [certs:/usr/share/elasticsearch/config/certs, esdata:/usr/share/elasticsearch/data]
ports: ["${ES_PORT}:9200"]
environment:
- node.name=es01
- cluster.name=${CLUSTER_NAME}
- discovery.type=single-node
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es01/es01.key
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${ES_MEM_LIMIT}
ulimits: { memlock: { soft: -1, hard: -1 } }
kibana:
depends_on: [es01]
image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
volumes: [certs:/usr/share/kibana/config/certs, kibanadata:/usr/share/kibana/data]
ports: ["${KIBANA_PORT}:5601"]
environment:
- ELASTICSEARCH_HOSTS=https://es01:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
volumes: { certs: {}, esdata: {}, kibanadata: {} }
10
Bring the stack up
launch
Start detached. The setup container runs once and exits 0; that is expected. Follow logs if Elasticsearch takes a while to pass its health check.
$ docker compose up -d
$ docker compose ps
# tail Elasticsearch if it is slow to go green
$ docker compose logs -f es01
11
Verify the cluster
verify
Hit the API from the host with the elastic password and confirm cluster health, then log in to Kibana.
$ curl -sk -u elastic:$ELASTIC_PASSWORD https://localhost:9200/_cluster/health?pretty
# status should be "green" or "yellow" (yellow is normal on single-node)
$ xdg-open http://localhost:5601
A single-node cluster reports yellow, not green, because replica shards have nowhere to go. That is healthy for a lab — do not chase green by adding replicas on one node.
⚡
05 — Method 3: Quick Trial & Elastic Cloud
PHASE 05 / 11
Two low-friction paths when you just want to try the Security app: Elastic's local dev script, or a hosted trial where Elastic runs the cluster for you.
12
One-command local trial
start-local
The official start-local script pulls Elasticsearch and Kibana, wires them together, and prints the URL and credentials. It is for evaluation only — security is relaxed and data is not meant to persist across upgrades.
$ curl -fsSL https://elastic.co/start-local | sh
# -> creates ./elastic-start-local with .env holding the generated password
$ cat elastic-start-local/.env
Never expose a start-local instance to the internet or use it in production. It trades TLS strictness for speed of setup — treat it as a throwaway sandbox.
13
Elastic Cloud hosted trial
cloud
Sign up for the Elastic Cloud trial, create a deployment, and Elastic provisions a secured cluster with Kibana in minutes. You still install Elastic Agent locally; it ships to the Cloud endpoint over TLS. Grab the Cloud ID and Kibana URL from the deployment page.
1Create a deployment at cloud.elastic.co; choose the Security use case template.
2Copy the elastic password shown once at creation into a password manager.
3Open Kibana from the deployment; Fleet and integrations work identically to self-managed.
☁️
Elastic Cloud — free trial
cloud.elastic.co · hosted Elasticsearch + Kibana
↗
| Method | Best for | Effort | Persistence |
|---|
| Native package | Production-like lab, upgrades | Medium | Full |
| Docker Compose | Repeatable single-node SIEM | Low | Volumes |
| start-local | 5-minute evaluation | Trivial | Throwaway |
| Elastic Cloud | No infra to run | Trivial | Managed |
🔧
06 — Initial Configuration
PHASE 06 / 11
Whatever method you used, the first login lands you as the all-powerful elastic user. Lock that down and carve out a role you can actually work as.
14
First login & secure the superuser
hardening
Log in to Kibana at :5601 as elastic. Treat that account like root: use it to create working users, not for daily analysis. Rotate its password if it was ever printed to a shared terminal.
Where to go
Kibana ☰ → Stack Management → Security → Users
# CLI alternative to rotate elastic from the ES host
$ sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i
15
Create a SOC analyst user
rbac
Use the security API to create a role scoped to the Security app plus read access to the data indices, then a user bound to it. This is the account your analysts log in with day to day.
$ curl -sk -u elastic:$ELASTIC_PASSWORD -X POST https://localhost:9200/_security/role/soc_analyst \
-H 'Content-Type: application/json' -d '{
"indices": [{ "names": ["logs-*",".alerts-security.*"], "privileges": ["read","view_index_metadata"] }],
"applications": [{ "application":"kibana-.kibana","privileges":["feature_siem.all"],"resources":["*"] }]
}'
$ curl -sk -u elastic:$ELASTIC_PASSWORD -X POST https://localhost:9200/_security/user/analyst1 \
-H 'Content-Type: application/json' -d '{
"password":"Analyst_2026!","roles":["soc_analyst"],"full_name":"SOC Analyst 1"
}'
Prefer the built-in t1_analyst and soc_manager roles that ship with the Security solution for realistic tiering. Clone one in Stack Management if you need to widen a single privilege rather than building from scratch.
16
Tour the Security app
orientation
Open ☰ → Security. The landing Overview is empty until data flows — that is expected. Learn the four surfaces you will live in before wiring agents.
1Alerts — where detection-rule hits land, grouped and triageable.
2Rules — install and manage prebuilt and custom detections.
3Timelines — the hunting canvas for pivoting across raw events.
4Cases — package findings into a shareable investigation record.
🛰️
07 — Fleet & Elastic Agent
PHASE 07 / 11
Fleet is the control plane: it hands each Elastic Agent a policy that says which integrations to run. Stand up a Fleet Server, define a policy, then enroll agents on the hosts you want to monitor.
FIG. 2 — ELASTIC AGENT ENROLLS VIA FLEET, PULLS A POLICY, THEN SHIPS EVENTS TO ELASTICSEARCH
17
Add a Fleet Server
fleet server
In Kibana go to Fleet, add a Fleet Server, and set the host URL agents will reach (for a lab, the SIEM host itself on 8220). Kibana generates a service token and the exact install command; run it on the host that will act as Fleet Server.
Where to go
Kibana ☰ → Fleet → Settings → Add Fleet Server
# command shape Kibana generates (token + certs filled in for you)
$ sudo ./elastic-agent install --fleet-server-es=https://ES_HOST:9200 \
--fleet-server-service-token=$SERVICE_TOKEN \
--fleet-server-policy=fleet-server-policy \
--fleet-server-es-ca=/etc/elasticsearch/certs/http_ca.crt \
--fleet-server-port=8220
18
Create an agent policy
policy
A policy is a named bundle of integrations. Create one per host class — e.g. "windows-endpoints" and "linux-servers" — so you can attach different data collection to each.
1Fleet → Agent policies → Create agent policy, name it windows-endpoints.
2Leave System integration enabled — it collects host metrics and base logs by default.
3Save; you will attach the Windows and Elastic Defend integrations to it in Phase 08.
19
Install Elastic Agent on Linux
linux agent
Fleet → Agents → Add agent shows the enrollment token and URL. Download the agent tarball on the target and enroll it against the policy. The --insecure flag is only for a lab with self-signed certs; in production pass the CA instead.
$ curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.5.0-linux-x86_64.tar.gz
$ tar xzf elastic-agent-9.5.0-linux-x86_64.tar.gz && cd elastic-agent-9.5.0-linux-x86_64
$ sudo ./elastic-agent install --url=https://FLEET_HOST:8220 \
--enrollment-token=$ENROLL_TOKEN --ca-sha256=$CA_FINGERPRINT
# confirm the agent is healthy and checking in
$ sudo elastic-agent status
20
Install Elastic Agent on Windows
windows agent
On the Windows host, run an elevated PowerShell. Download the ZIP, expand it, and enroll against the windows-endpoints policy. The installer registers a service that survives reboots.
PowerShell (Administrator)
PS> Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.5.0-windows-x86_64.zip -OutFile agent.zip
PS> Expand-Archive .\agent.zip -DestinationPath .
PS> cd .\elastic-agent-9.5.0-windows-x86_64
PS> .\elastic-agent.exe install --url=https://FLEET_HOST:8220 --enrollment-token=$ENROLL_TOKEN
The agent must reach Fleet on 8220 and Elasticsearch on 9200 outbound. If an agent shows "enrolled" but never sends data, a firewall is usually blocking 9200 while 8220 is open.
📥
08 — Ingesting Data (Integrations)
PHASE 08 / 11
Integrations are pre-built input + parser + dashboard packages. Adding one to a policy tells every agent on that policy to collect and normalise a source. Start with System, add Windows, then turn agents into EDR sensors with Elastic Defend.
| Integration | Collects | Feeds detections for |
|---|
| System | Host metrics, auth logs, syslog | Brute force, sudo abuse, service changes |
| Windows | Security, System, PowerShell, Sysmon | Credential theft, lateral movement, persistence |
| Elastic Defend | Process/file/network + malware | Endpoint behaviour, malware prevention |
| Auditd Manager | Linux auditd rule events | Execution, file integrity, privilege abuse |
| Network / Fortinet / etc. | Firewall & NGFW logs | C2 egress, port scans, policy hits |
21
Add the System integration
baseline
If it is not already on your policy, add System from the Integrations catalog. It ships auth, syslog, and metric data with dashboards — your baseline visibility on any host.
Where to go
Kibana ☰ → Integrations → System → Add → assign to policy
22
Add the Windows integration
windows logs
Add Windows to the windows-endpoints policy. Enable the Security, System, PowerShell, and — if deployed — Sysmon channels. These four channels feed most Windows detection rules. Confirm data with an ES|QL count once an agent checks in.
CONFIRMS: Windows event logs are arriving in the last 15 minutes
# run in Kibana Dev Tools or via _query API
FROM logs-windows.*
| WHERE @timestamp > NOW() - 15 minutes
| STATS events = COUNT(*) BY event.code
| SORT events DESC
23
Deploy Elastic Defend (EDR)
endpoint
Add the Elastic Defend integration to the policy and choose a protection level. "Detect" logs behaviour without blocking; "Prevent" stops malware and ransomware. Start in Detect on production hardware, move to Prevent once you trust the baseline.
1Integrations → Elastic Defend → Add; pick the "Complete EDR" preset.
2Set malware, ransomware, and memory-threat protections to detect first.
3Save; agents on the policy pull the endpoint sensor automatically within a minute.
Elastic Defend in Prevent mode can quarantine files and kill processes. Never roll it straight to Prevent on domain controllers or build servers without an exception list — test on a canary host first.
24
Verify data is flowing
verify
Confirm each source has documents before you rely on rules. Check index counts from the API and eyeball the Security → Explore → Hosts view for live entities.
$ curl -sk -u elastic:$ELASTIC_PASSWORD "https://localhost:9200/_cat/indices/logs-*?v&s=docs.count:desc"
Ingest is working when
- _cat/indices shows growing docs.count on logs-system.* and logs-windows.*
- Security → Explore → Hosts lists your enrolled hosts
- Fleet shows every agent Healthy with a recent check-in
- logs-endpoint.events.* appears once Elastic Defend is active
🚨
09 — Detection Rules & Hunting
PHASE 09 / 11
With data flowing, turn on detection. Install Elastic's prebuilt rules, write a custom one to learn the mechanics, tune out false positives, then hunt interactively with ES|QL and Timeline.
25
Install prebuilt rules
rules
The Rules page ships 1,000-plus curated, ATT&CK-mapped detections. Install them all, then enable a starter set — Windows credential access, suspicious PowerShell, and Defend endpoint alerts are high-signal first picks.
Where to go
Security ☰ → Rules → Detection rules (SIEM) → Add Elastic rules
1Click Add Elastic rules; the badge shows how many are available.
2Install all, then filter by tag OS: Windows and enable the Credential Access set.
3Enable "Malware Prevention Alert" and "Behavior Detected" to surface Elastic Defend hits.
26
Write a custom EQL rule
custom detection
EQL (Event Query Language) is Elastic's sequence-aware detection language. This rule fires when whoami.exe runs — a classic post-exploitation recon check. Create it via the detection engine API so it is reproducible; the same query works in the Rules UI.
DETECTS: execution of whoami on Windows endpoints (recon)
$ curl -sk -u elastic:$ELASTIC_PASSWORD -X POST \
https://localhost:9200/../api/detection_engine/rules \
-H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{
"name":"Whoami Execution (Recon)",
"description":"Detects whoami.exe, a common host recon command",
"risk_score":47, "severity":"medium", "type":"eql", "language":"eql",
"index":["logs-endpoint.events.*","logs-windows.*"],
"query":"process where event.type == \"start\" and process.name == \"whoami.exe\"",
"from":"now-6m","interval":"5m","enabled":true
}'
Point custom rules through the Kibana API (/api/detection_engine/rules), not the Elasticsearch API — detection rules are Kibana saved objects. Always include the kbn-xsrf header or the request is rejected.
27
Tune false positives with exceptions
tuning
A noisy rule is worse than no rule. When a legitimate admin tool trips a detection, add a rule exception rather than disabling the rule. Exceptions are field-scoped and keep the detection live for everyone else.
1Open the alert → Take action → Add rule exception.
2Scope narrowly, e.g. process.parent.name : "MonitoringAgent.exe".
3For endpoint prevention, add a trusted application under Defend policy instead.
Never build an exception on a wildcard like host.name : * or a broad path — you will silently blind the rule fleet-wide. Anchor every exception to a specific parent, hash, or signer.
28
Hunt with ES|QL & Timeline
hunting
Detections catch the known; hunting finds the rest. ES|QL is the piped query language for ad-hoc analysis. This hunt surfaces rare parent-child process pairs — a cheap way to spot living-off-the-land abuse. Pivot any interesting row into a Timeline to reconstruct the session.
FINDS: uncommon parent→child process relationships across endpoints
FROM logs-endpoint.events.process
| WHERE event.type == "start"
| STATS hits = COUNT(*) BY process.parent.name, process.name
| WHERE hits < 5
| SORT hits ASC
| LIMIT 50
FINDS: Windows failed-logon spikes by source (brute force / spray)
FROM logs-windows.*
| WHERE event.code == "4625"
| STATS failures = COUNT(*) BY source.ip, user.name
| WHERE failures > 20
| SORT failures DESC
You now have a working SIEM when
- Prebuilt rules are installed and a starter set is enabled
- Your custom whoami rule generates an alert on test execution
- Alerts triage cleanly with scoped exceptions, not disabled rules
- ES|QL hunts return results and pivot into Timeline
🩺
10 — Monitoring, Maintenance & Troubleshooting
PHASE 10 / 11
A SIEM that silently stops ingesting is a liability. Keep an eye on cluster health and disk, manage retention with index lifecycle policies, and know the handful of errors that cause 90% of Elastic outages.
29
Health checks & retention
operations
Watch cluster health, node disk, and shard allocation. Elastic's data streams ship with an Index Lifecycle Management (ILM) policy — tune it so logs roll over and delete on a schedule instead of filling the disk.
$ curl -sk -u elastic:$ELASTIC_PASSWORD https://localhost:9200/_cluster/health?pretty
$ curl -sk -u elastic:$ELASTIC_PASSWORD "https://localhost:9200/_cat/allocation?v"
# inspect the built-in logs ILM policy before editing retention
$ curl -sk -u elastic:$ELASTIC_PASSWORD https://localhost:9200/_ilm/policy/logs?pretty
30
Snapshots & upgrades
backup
Register a snapshot repository and take backups before every upgrade. Upgrade Elasticsearch before Kibana, one minor at a time, and never skip a major.
$ curl -sk -u elastic:$ELASTIC_PASSWORD -X PUT https://localhost:9200/_snapshot/backup \
-H 'Content-Type: application/json' -d '{"type":"fs","settings":{"location":"/var/backups/es"}}'
$ curl -sk -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_snapshot/backup/snap-1?wait_for_completion=true"
Add path.repo: ["/var/backups/es"] to elasticsearch.yml (or the volume in Docker) before registering an fs repository, or the PUT fails with "location doesn't match any registered repository paths".
The errors you will actually hit — and the fix for each:
| Symptom | Cause | Fix |
|---|
| ES exits at boot, "max virtual memory too low" | vm.max_map_count < 262144 | sysctl -w vm.max_map_count=262144 and persist it |
| Indices go read-only, ingest stops | Disk past 95% flood-stage watermark | Free disk, then clear read_only_allow_delete block |
| Kibana: "Unable to connect to Elasticsearch" | Wrong kibana_system password or CA | Reset kibana_system password; check CA path in kibana.yml |
| Agent enrolled but no data | Outbound 9200 blocked; CA not trusted | Open 9200 egress; pass --ca-sha256 fingerprint |
| Fleet Server "unhealthy" | Service token or policy misconfigured | Re-generate token; confirm 8220 reachable |
| Cluster stuck yellow on one node | Unassigned replica shards (expected) | Set index replicas to 0 for single-node labs |
# clear the read-only block after freeing disk
$ curl -sk -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_all/_settings" \
-H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete":null}'
Do not expose Elasticsearch 9200 or Kibana 5601 directly to the internet. Put Kibana behind a reverse proxy with authentication and restrict 9200 to the agent subnet — unauthenticated internet-facing clusters are mass-scanned and ransomed constantly.
📚
11 — Sources & References
PHASE 11 / 11
Primary documentation used to build and verify every step in this guide:
You have a free, self-hosted SIEM with EDR and a detection engine — now feed it real threat context.
Pull the latest indicators into your Elastic detections and validate what you are seeing against live intelligence. Start with the CyberHawk IOC Scanner, browse hands-on SOC SOPs for triage playbooks that pair with these rules, and keep current on emerging campaigns via CyberHawk Threat Intel.
◈ Stay Connected
Follow CyberHawk Threat Intel for threat intelligence, deployment guides and hands-on SOC tooling content.
"They can't exploit you if you are the Exploit."