267 lines
9.7 KiB
Markdown
267 lines
9.7 KiB
Markdown
|
---
|
|||
|
|
page-title: "Setting Up Elasticsearch and Kibana Single-Node with Docker Compose | by Karthik S | Medium"
|
|||
|
|
url: https://karthiksdevopsengineer.medium.com/setting-up-elasticsearch-and-kibana-single-node-with-docker-compose-329776fa3aee
|
|||
|
|
date: "2024-09-27 10:53:26"
|
|||
|
|
---
|
|||
|
|
[
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
](https://karthiksdevopsengineer.medium.com/?source=post_page-----329776fa3aee--------------------------------)
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
Setting up Elasticsearch and Kibana on a single-node cluster can be a straightforward process with Docker Compose. In this guide, we’ll walk through the steps to get your Elasticsearch and Kibana instances up and running smoothly.
|
|||
|
|
|
|||
|
|
## Hardware Prerequisites
|
|||
|
|
|
|||
|
|
According to the Elastic Cloud Enterprise documentation, here are the hardware requirements for running Elasticsearch and Kibana
|
|||
|
|
|
|||
|
|
- **CPU**: A minimum of 2 CPU cores is recommended, but the actual requirement depends on your workload. More CPU cores may be required for intensive tasks or larger datasets.
|
|||
|
|
- **RAM**: Elastic recommends a minimum of 8GB of RAM for Elasticsearch, but 16GB or more is recommended for production use, especially when running both Elasticsearch and Kibana on the same machine.
|
|||
|
|
- **Storage**: SSD storage is recommended for better performance, especially for production use. The amount of storage required depends on your data volume and retention policies.
|
|||
|
|
|
|||
|
|
For more detailed hardware requirements and recommendations, refer to the [Elastic Cloud Enterprise documentation](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-hardware-prereq.html#ece-hardware-prereq).
|
|||
|
|
|
|||
|
|
## Software Prerequisites
|
|||
|
|
|
|||
|
|
Before getting started, make sure you have Docker installed on your system. You can download and install Docker from the [official website](https://docs.docker.com/engine/install/).
|
|||
|
|
|
|||
|
|
## Setting Up Instructions
|
|||
|
|
|
|||
|
|
In this guide, I will perform these operations with the following specifications.
|
|||
|
|
|
|||
|
|
- **OS**: Ubuntu 22.04
|
|||
|
|
- **RAM**: 8GB
|
|||
|
|
- **Storage**: 30GB SSD
|
|||
|
|
|
|||
|
|
## 1\. Adjust Kernel Settings
|
|||
|
|
|
|||
|
|
The `vm.max_map_count` kernel setting must be set to at least `262144`
|
|||
|
|
|
|||
|
|
How you set `vm.max_map_count` depends on your platform. For [more information](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_set_vm_max_map_count_to_at_least_262144)
|
|||
|
|
|
|||
|
|
I’m using the Linux operating system, so I will set `vm.max_map_count` using as follows
|
|||
|
|
|
|||
|
|
Open the ‘**/etc/sysctl.conf**’ file in a text editor with root privileges. You can use the following command
|
|||
|
|
|
|||
|
|
sudo nano /etc/sysctl.conf
|
|||
|
|
|
|||
|
|
Navigate to the end of the file or search for the line containing `vm.max_map_count`, If the line exists, modify it to set the desired value
|
|||
|
|
|
|||
|
|
vm.max\_map\_count\=262144
|
|||
|
|
|
|||
|
|
If the line doesn’t exist, add it at the end of the file
|
|||
|
|
|
|||
|
|
|
|||
|
|
vm.max\_map\_count\=262144
|
|||
|
|
|
|||
|
|
Save the file and exit the text editor. Apply the changes by running the following command
|
|||
|
|
|
|||
|
|
sudo sysctl -p
|
|||
|
|
|
|||
|
|
This command reloads the sysctl settings from the configuration file. Now, the value of `vm.max_map_count` should be updated to **262144**.
|
|||
|
|
|
|||
|
|
## 2\. Prepare Environment Variables
|
|||
|
|
|
|||
|
|
Create or navigate to an empty directory for the project.
|
|||
|
|
|
|||
|
|
Inside this directory, create a `.env` file and set up the necessary environment variables.
|
|||
|
|
|
|||
|
|
Copy the following content and paste it into the `.env` file.
|
|||
|
|
|
|||
|
|
|
|||
|
|
ELASTIC\_PASSWORD=
|
|||
|
|
|
|||
|
|
|
|||
|
|
KIBANA\_PASSWORD=
|
|||
|
|
|
|||
|
|
|
|||
|
|
STACK\_VERSION={version}
|
|||
|
|
|
|||
|
|
|
|||
|
|
CLUSTER\_NAME=docker-cluster
|
|||
|
|
|
|||
|
|
|
|||
|
|
LICENSE=basic
|
|||
|
|
|
|||
|
|
|
|||
|
|
ES\_PORT=9200
|
|||
|
|
|
|||
|
|
|
|||
|
|
KIBANA\_PORT=5601
|
|||
|
|
|
|||
|
|
|
|||
|
|
MEM\_LIMIT=2147483648
|
|||
|
|
|
|||
|
|
In the `.env` file, specify a password for the `ELASTIC_PASSWORD` and `KIBANA_PASSWORD` variables.
|
|||
|
|
|
|||
|
|
The passwords must be alphanumeric and can’t contain special characters, such as `!` or `@`. The bash script included in the `compose.yml` file only works with alphanumeric characters. Example:
|
|||
|
|
|
|||
|
|
|
|||
|
|
ELASTIC\_PASSWORD=Secure123
|
|||
|
|
|
|||
|
|
|
|||
|
|
KIBANA\_PASSWORD=Secure123
|
|||
|
|
|
|||
|
|
...
|
|||
|
|
|
|||
|
|
In the `.env` file, set `STACK_VERSION` to the Elastic Stack version. Example:
|
|||
|
|
|
|||
|
|
...
|
|||
|
|
|
|||
|
|
\# Version of Elastic products
|
|||
|
|
STACK\_VERSION=8.13.2
|
|||
|
|
|
|||
|
|
...
|
|||
|
|
|
|||
|
|
## 3\. Create Docker Compose Configuration
|
|||
|
|
|
|||
|
|
Now, create a `compose.yml` file in the same directory and copy the following content and paste it into the `compose.yml` file.
|
|||
|
|
|
|||
|
|
version: "2.2"
|
|||
|
|
|
|||
|
|
services:
|
|||
|
|
setup:
|
|||
|
|
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK\_VERSION}
|
|||
|
|
volumes:
|
|||
|
|
- certs:/usr/share/elasticsearch/config/certs
|
|||
|
|
user: "0"
|
|||
|
|
command: >
|
|||
|
|
bash -c '
|
|||
|
|
if \[ x${ELASTIC\_PASSWORD} == x \]; then
|
|||
|
|
echo "Set the ELASTIC\_PASSWORD environment variable in the .env file";
|
|||
|
|
exit 1;
|
|||
|
|
elif \[ x${KIBANA\_PASSWORD} == x \]; then
|
|||
|
|
echo "Set the KIBANA\_PASSWORD environment variable in the .env file";
|
|||
|
|
exit 1;
|
|||
|
|
fi;
|
|||
|
|
if \[ ! -f config/certs/ca.zip \]; then
|
|||
|
|
echo "Creating CA";
|
|||
|
|
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
|
|||
|
|
unzip config/certs/ca.zip -d config/certs;
|
|||
|
|
fi;
|
|||
|
|
if \[ ! -f config/certs/certs.zip \]; then
|
|||
|
|
echo "Creating certs";
|
|||
|
|
echo -ne \\
|
|||
|
|
"instances:\\n"\\
|
|||
|
|
" - name: es01\\n"\\
|
|||
|
|
" dns:\\n"\\
|
|||
|
|
" - es01\\n"\\
|
|||
|
|
" - localhost\\n"\\
|
|||
|
|
" ip:\\n"\\
|
|||
|
|
" - 127.0.0.1\\n"\\
|
|||
|
|
> config/certs/instances.yml;
|
|||
|
|
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
|
|||
|
|
unzip config/certs/certs.zip -d config/certs;
|
|||
|
|
fi;
|
|||
|
|
echo "Setting file permissions"
|
|||
|
|
chown -R root:root config/certs;
|
|||
|
|
find . -type d -exec chmod 750 \\{\\} \\;;
|
|||
|
|
find . -type f -exec chmod 640 \\{\\} \\;;
|
|||
|
|
echo "Waiting for Elasticsearch availability";
|
|||
|
|
until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
|
|||
|
|
echo "Setting kibana\_system password";
|
|||
|
|
until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC\_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/\_security/user/kibana\_system/\_password -d "{\\"password\\":\\"${KIBANA\_PASSWORD}\\"}" | grep -q "^{}"; do sleep 10; done;
|
|||
|
|
echo "All done!";
|
|||
|
|
'
|
|||
|
|
healthcheck:
|
|||
|
|
test: \["CMD-SHELL", "\[ -f config/certs/es01/es01.crt \]"\]
|
|||
|
|
interval: 1s
|
|||
|
|
timeout: 5s
|
|||
|
|
retries: 120
|
|||
|
|
|
|||
|
|
es01:
|
|||
|
|
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK\_VERSION}
|
|||
|
|
volumes:
|
|||
|
|
- certs:/usr/share/elasticsearch/config/certs
|
|||
|
|
- esdata:/usr/share/elasticsearch/data
|
|||
|
|
ports:
|
|||
|
|
- ${ES\_PORT}:9200
|
|||
|
|
environment:
|
|||
|
|
- node.name=es01
|
|||
|
|
- cluster.name=${CLUSTER\_NAME}
|
|||
|
|
- discovery.type=single-node
|
|||
|
|
- ELASTIC\_PASSWORD=${ELASTIC\_PASSWORD}
|
|||
|
|
- bootstrap.memory\_lock=true
|
|||
|
|
- xpack.security.enabled=true
|
|||
|
|
- xpack.security.http.ssl.enabled=true
|
|||
|
|
- xpack.security.http.ssl.key=certs/es01/es01.key
|
|||
|
|
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
|
|||
|
|
- xpack.security.http.ssl.certificate\_authorities=certs/ca/ca.crt
|
|||
|
|
- xpack.security.transport.ssl.enabled=true
|
|||
|
|
- xpack.security.transport.ssl.key=certs/es01/es01.key
|
|||
|
|
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
|
|||
|
|
- xpack.security.transport.ssl.certificate\_authorities=certs/ca/ca.crt
|
|||
|
|
- xpack.security.transport.ssl.verification\_mode=certificate
|
|||
|
|
- xpack.license.self\_generated.type=${LICENSE}
|
|||
|
|
mem\_limit: ${MEM\_LIMIT}
|
|||
|
|
ulimits:
|
|||
|
|
memlock:
|
|||
|
|
soft: -1
|
|||
|
|
hard: -1
|
|||
|
|
healthcheck:
|
|||
|
|
test:
|
|||
|
|
\[
|
|||
|
|
"CMD-SHELL",
|
|||
|
|
"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
|
|||
|
|
\]
|
|||
|
|
interval: 10s
|
|||
|
|
timeout: 10s
|
|||
|
|
retries: 120
|
|||
|
|
|
|||
|
|
kibana:
|
|||
|
|
depends\_on:
|
|||
|
|
es01:
|
|||
|
|
condition: service\_healthy
|
|||
|
|
image: docker.elastic.co/kibana/kibana:${STACK\_VERSION}
|
|||
|
|
volumes:
|
|||
|
|
- certs:/usr/share/kibana/config/certs
|
|||
|
|
- kibanadata:/usr/share/kibana/data
|
|||
|
|
ports:
|
|||
|
|
- ${KIBANA\_PORT}:5601
|
|||
|
|
environment:
|
|||
|
|
- SERVERNAME=kibana
|
|||
|
|
- ELASTICSEARCH\_HOSTS=https://es01:9200
|
|||
|
|
- ELASTICSEARCH\_USERNAME=kibana\_system
|
|||
|
|
- ELASTICSEARCH\_PASSWORD=${KIBANA\_PASSWORD}
|
|||
|
|
- ELASTICSEARCH\_SSL\_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
|
|||
|
|
- SERVER\_PUBLICBASEURL=http://localhost:5601
|
|||
|
|
mem\_limit: ${MEM\_LIMIT}
|
|||
|
|
healthcheck:
|
|||
|
|
test:
|
|||
|
|
\[
|
|||
|
|
"CMD-SHELL",
|
|||
|
|
"curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
|
|||
|
|
\]
|
|||
|
|
interval: 10s
|
|||
|
|
timeout: 10s
|
|||
|
|
retries: 120
|
|||
|
|
|
|||
|
|
volumes:
|
|||
|
|
certs:
|
|||
|
|
driver: local
|
|||
|
|
esdata:
|
|||
|
|
driver: local
|
|||
|
|
kibanadata:
|
|||
|
|
driver: local
|
|||
|
|
|
|||
|
|
## 4\. Start Docker Compose
|
|||
|
|
|
|||
|
|
Now you can start Elasticsearch and Kibana using Docker Compose. Run the following command from your project directory
|
|||
|
|
|
|||
|
|
docker compose up -d
|
|||
|
|
|
|||
|
|
**5\. Access Elasticsearch and Kibana**
|
|||
|
|
|
|||
|
|
Once Docker Compose has started the services, you can access Elasticsearch at `https://<localhost or serverip>:9200` and Kibana at `http://<localhost or serverip>:5601` in your web browser.
|
|||
|
|
|
|||
|
|
Log in to Elasticsearch or Kibana as the `elastic` user and the password is the one you set earlier in the `.env` file.
|
|||
|
|
|
|||
|
|
## Conclusion
|
|||
|
|
|
|||
|
|
You’ve successfully set up Elasticsearch and Kibana on a single-node using Docker Compose.
|
|||
|
|
|
|||
|
|
**Reference** [https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html)
|