267 lines
9.7 KiB
Markdown
267 lines
9.7 KiB
Markdown
---
|
||
page-title: "Setting Up Elasticsearch and Kibana Single-Node with Docker Compose | by Karthik S | Medium"
|
||
url: https://karthiksdevopsengineer.medium.com/setting-up-elasticsearch-and-kibana-single-node-with-docker-compose-329776fa3aee
|
||
date: "2024-09-27 10:53:26"
|
||
---
|
||
[
|
||
|
||
|
||
|
||
|
||
|
||
](https://karthiksdevopsengineer.medium.com/?source=post_page-----329776fa3aee--------------------------------)
|
||
|
||
|
||
|
||
Setting up Elasticsearch and Kibana on a single-node cluster can be a straightforward process with Docker Compose. In this guide, we’ll walk through the steps to get your Elasticsearch and Kibana instances up and running smoothly.
|
||
|
||
## Hardware Prerequisites
|
||
|
||
According to the Elastic Cloud Enterprise documentation, here are the hardware requirements for running Elasticsearch and Kibana
|
||
|
||
- **CPU**: A minimum of 2 CPU cores is recommended, but the actual requirement depends on your workload. More CPU cores may be required for intensive tasks or larger datasets.
|
||
- **RAM**: Elastic recommends a minimum of 8GB of RAM for Elasticsearch, but 16GB or more is recommended for production use, especially when running both Elasticsearch and Kibana on the same machine.
|
||
- **Storage**: SSD storage is recommended for better performance, especially for production use. The amount of storage required depends on your data volume and retention policies.
|
||
|
||
For more detailed hardware requirements and recommendations, refer to the [Elastic Cloud Enterprise documentation](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-hardware-prereq.html#ece-hardware-prereq).
|
||
|
||
## Software Prerequisites
|
||
|
||
Before getting started, make sure you have Docker installed on your system. You can download and install Docker from the [official website](https://docs.docker.com/engine/install/).
|
||
|
||
## Setting Up Instructions
|
||
|
||
In this guide, I will perform these operations with the following specifications.
|
||
|
||
- **OS**: Ubuntu 22.04
|
||
- **RAM**: 8GB
|
||
- **Storage**: 30GB SSD
|
||
|
||
## 1\. Adjust Kernel Settings
|
||
|
||
The `vm.max_map_count` kernel setting must be set to at least `262144`
|
||
|
||
How you set `vm.max_map_count` depends on your platform. For [more information](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_set_vm_max_map_count_to_at_least_262144)
|
||
|
||
I’m using the Linux operating system, so I will set `vm.max_map_count` using as follows
|
||
|
||
Open the ‘**/etc/sysctl.conf**’ file in a text editor with root privileges. You can use the following command
|
||
|
||
sudo nano /etc/sysctl.conf
|
||
|
||
Navigate to the end of the file or search for the line containing `vm.max_map_count`, If the line exists, modify it to set the desired value
|
||
|
||
vm.max\_map\_count\=262144
|
||
|
||
If the line doesn’t exist, add it at the end of the file
|
||
|
||
|
||
vm.max\_map\_count\=262144
|
||
|
||
Save the file and exit the text editor. Apply the changes by running the following command
|
||
|
||
sudo sysctl -p
|
||
|
||
This command reloads the sysctl settings from the configuration file. Now, the value of `vm.max_map_count` should be updated to **262144**.
|
||
|
||
## 2\. Prepare Environment Variables
|
||
|
||
Create or navigate to an empty directory for the project.
|
||
|
||
Inside this directory, create a `.env` file and set up the necessary environment variables.
|
||
|
||
Copy the following content and paste it into the `.env` file.
|
||
|
||
|
||
ELASTIC\_PASSWORD=
|
||
|
||
|
||
KIBANA\_PASSWORD=
|
||
|
||
|
||
STACK\_VERSION={version}
|
||
|
||
|
||
CLUSTER\_NAME=docker-cluster
|
||
|
||
|
||
LICENSE=basic
|
||
|
||
|
||
ES\_PORT=9200
|
||
|
||
|
||
KIBANA\_PORT=5601
|
||
|
||
|
||
MEM\_LIMIT=2147483648
|
||
|
||
In the `.env` file, specify a password for the `ELASTIC_PASSWORD` and `KIBANA_PASSWORD` variables.
|
||
|
||
The passwords must be alphanumeric and can’t contain special characters, such as `!` or `@`. The bash script included in the `compose.yml` file only works with alphanumeric characters. Example:
|
||
|
||
|
||
ELASTIC\_PASSWORD=Secure123
|
||
|
||
|
||
KIBANA\_PASSWORD=Secure123
|
||
|
||
...
|
||
|
||
In the `.env` file, set `STACK_VERSION` to the Elastic Stack version. Example:
|
||
|
||
...
|
||
|
||
\# Version of Elastic products
|
||
STACK\_VERSION=8.13.2
|
||
|
||
...
|
||
|
||
## 3\. Create Docker Compose Configuration
|
||
|
||
Now, create a `compose.yml` file in the same directory and copy the following content and paste it into the `compose.yml` file.
|
||
|
||
version: "2.2"
|
||
|
||
services:
|
||
setup:
|
||
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK\_VERSION}
|
||
volumes:
|
||
- certs:/usr/share/elasticsearch/config/certs
|
||
user: "0"
|
||
command: >
|
||
bash -c '
|
||
if \[ x${ELASTIC\_PASSWORD} == x \]; then
|
||
echo "Set the ELASTIC\_PASSWORD environment variable in the .env file";
|
||
exit 1;
|
||
elif \[ x${KIBANA\_PASSWORD} == x \]; then
|
||
echo "Set the KIBANA\_PASSWORD environment variable in the .env file";
|
||
exit 1;
|
||
fi;
|
||
if \[ ! -f config/certs/ca.zip \]; then
|
||
echo "Creating CA";
|
||
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
|
||
unzip config/certs/ca.zip -d config/certs;
|
||
fi;
|
||
if \[ ! -f config/certs/certs.zip \]; then
|
||
echo "Creating certs";
|
||
echo -ne \\
|
||
"instances:\\n"\\
|
||
" - name: es01\\n"\\
|
||
" dns:\\n"\\
|
||
" - es01\\n"\\
|
||
" - localhost\\n"\\
|
||
" ip:\\n"\\
|
||
" - 127.0.0.1\\n"\\
|
||
> config/certs/instances.yml;
|
||
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
|
||
unzip config/certs/certs.zip -d config/certs;
|
||
fi;
|
||
echo "Setting file permissions"
|
||
chown -R root:root config/certs;
|
||
find . -type d -exec chmod 750 \\{\\} \\;;
|
||
find . -type f -exec chmod 640 \\{\\} \\;;
|
||
echo "Waiting for Elasticsearch availability";
|
||
until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
|
||
echo "Setting kibana\_system password";
|
||
until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC\_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/\_security/user/kibana\_system/\_password -d "{\\"password\\":\\"${KIBANA\_PASSWORD}\\"}" | grep -q "^{}"; do sleep 10; done;
|
||
echo "All done!";
|
||
'
|
||
healthcheck:
|
||
test: \["CMD-SHELL", "\[ -f config/certs/es01/es01.crt \]"\]
|
||
interval: 1s
|
||
timeout: 5s
|
||
retries: 120
|
||
|
||
es01:
|
||
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK\_VERSION}
|
||
volumes:
|
||
- certs:/usr/share/elasticsearch/config/certs
|
||
- esdata:/usr/share/elasticsearch/data
|
||
ports:
|
||
- ${ES\_PORT}:9200
|
||
environment:
|
||
- node.name=es01
|
||
- cluster.name=${CLUSTER\_NAME}
|
||
- discovery.type=single-node
|
||
- ELASTIC\_PASSWORD=${ELASTIC\_PASSWORD}
|
||
- bootstrap.memory\_lock=true
|
||
- xpack.security.enabled=true
|
||
- xpack.security.http.ssl.enabled=true
|
||
- xpack.security.http.ssl.key=certs/es01/es01.key
|
||
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
|
||
- xpack.security.http.ssl.certificate\_authorities=certs/ca/ca.crt
|
||
- xpack.security.transport.ssl.enabled=true
|
||
- xpack.security.transport.ssl.key=certs/es01/es01.key
|
||
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
|
||
- xpack.security.transport.ssl.certificate\_authorities=certs/ca/ca.crt
|
||
- xpack.security.transport.ssl.verification\_mode=certificate
|
||
- xpack.license.self\_generated.type=${LICENSE}
|
||
mem\_limit: ${MEM\_LIMIT}
|
||
ulimits:
|
||
memlock:
|
||
soft: -1
|
||
hard: -1
|
||
healthcheck:
|
||
test:
|
||
\[
|
||
"CMD-SHELL",
|
||
"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
|
||
\]
|
||
interval: 10s
|
||
timeout: 10s
|
||
retries: 120
|
||
|
||
kibana:
|
||
depends\_on:
|
||
es01:
|
||
condition: service\_healthy
|
||
image: docker.elastic.co/kibana/kibana:${STACK\_VERSION}
|
||
volumes:
|
||
- certs:/usr/share/kibana/config/certs
|
||
- kibanadata:/usr/share/kibana/data
|
||
ports:
|
||
- ${KIBANA\_PORT}:5601
|
||
environment:
|
||
- SERVERNAME=kibana
|
||
- ELASTICSEARCH\_HOSTS=https://es01:9200
|
||
- ELASTICSEARCH\_USERNAME=kibana\_system
|
||
- ELASTICSEARCH\_PASSWORD=${KIBANA\_PASSWORD}
|
||
- ELASTICSEARCH\_SSL\_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
|
||
- SERVER\_PUBLICBASEURL=http://localhost:5601
|
||
mem\_limit: ${MEM\_LIMIT}
|
||
healthcheck:
|
||
test:
|
||
\[
|
||
"CMD-SHELL",
|
||
"curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
|
||
\]
|
||
interval: 10s
|
||
timeout: 10s
|
||
retries: 120
|
||
|
||
volumes:
|
||
certs:
|
||
driver: local
|
||
esdata:
|
||
driver: local
|
||
kibanadata:
|
||
driver: local
|
||
|
||
## 4\. Start Docker Compose
|
||
|
||
Now you can start Elasticsearch and Kibana using Docker Compose. Run the following command from your project directory
|
||
|
||
docker compose up -d
|
||
|
||
**5\. Access Elasticsearch and Kibana**
|
||
|
||
Once Docker Compose has started the services, you can access Elasticsearch at `https://<localhost or serverip>:9200` and Kibana at `http://<localhost or serverip>:5601` in your web browser.
|
||
|
||
Log in to Elasticsearch or Kibana as the `elastic` user and the password is the one you set earlier in the `.env` file.
|
||
|
||
## Conclusion
|
||
|
||
You’ve successfully set up Elasticsearch and Kibana on a single-node using Docker Compose.
|
||
|
||
**Reference** [https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html) |