Setting up ELK Stack for Log Collection and Analysis

Logging and monitoring are crucial for understanding and troubleshooting the behavior of any application. Logs provide valuable insight into your systems’ performance and can help you quickly identify and fix issues. This blog post will show you how to set up a robust logging and monitoring solution using Kibana, Elasticsearch, and Logstash. We’ll cover the basics of installing and configuring these tools, sending and viewing logs in Kibana, and how to set up a local development environment with the help of Docker Compose.

TL;DR: give me the code.

Source: missioncloud

Setting up Elasticsearch

Elasticsearch is a distributed search and analytics engine designed to store, search, and analyze large volumes of text data in real-time. You can either download Elasticsearch from the Elastic website and run it on your local machine or in a production environment or use the Elasticsearch image from the Docker hub. To use the Elasticsearch image from the Docker hub, you need to have Docker installed on your machine and set up Elasticsearch using a docker-compose file. Once Elasticsearch is up and running, you can access the Elasticsearch API to index and search for log data.

Setting up Kibana

After installing Elasticsearch, it’s time to set up Kibana for data visualization and exploration. Kibana is a web-based tool that allows you to interact with and make sense of your logs and metrics. It enables you to create customized dashboards, charts, and alerts. You can download Kibana from the Elastic website and run it. Like Elasticsearch, you can also use the Kibana image from the Docker hub and set it up using a docker-compose file. Once Kibana is up and running, you’ll need to establish a connection to your Elasticsearch instance by specifying the host and port in the Kibana configuration file.

Collecting Logs with Logstash

Kibana and Elasticsearch are potent tools for storing, searching, and analyzing log data, but they need to gather it; you’ll need to use a log shipper such as Logstash or Filebeat. These tools can collect log data from various sources and send it to Elasticsearch for indexing.

Logstash is a powerful data processing pipeline that can collect, parse, and transform log data before sending it to a central location, such as Elasticsearch. To set up Logstash, you’ll need to perform the following steps:

1. Create a Logstash pipeline configuration: Logstash uses pipeline configuration files to define how it processes data. These configuration files are written in the Logstash Configuration Language (LCF) and define inputs, filters, and outputs for log data.
2. Define inputs: The input section of the pipeline configuration file defines where Logstash should look for log data. You can configure Logstash in many ways, such as reading data from a file, syslog, or TCP/UDP socket.
3. Define filters: The filter section of the pipeline configuration file defines how Logstash should process the data it receives. You can use filters to parse, modify, or drop specific fields in the log data.
4. Define outputs: The output section of the pipeline configuration file defines where Logstash should send the processed data. For example, you can configure Logstash to send data to Elasticsearch, a file, or a message queue.

Setup ELK stack on Docker Compose

To simplify the setup process, we can use a docker-compose file to set up a local development environment with Kibana, Elasticsearch, and Logstash. The following is an example of a Docker Compose file that sets up the environment:

In this example, we’re using the official Elasticsearch, Kibana, and Logstash images from the Docker hub, and we’re disabling security for Elasticsearch for simplicity. We’re also mapping the ports for Elasticsearch (9200), Kibana (5601), and Logstash (5000) so that we can access them from the host machine. Keeping your Elasticsearch and Kibana versions in sync is essential. This ensures that all features and functionalities work as expected.

Additionally, we’re copying the Logstash pipeline configuration file logstash.conf in the logstash service.

logstash.conf file is a simple configuration that sets up Logstash to listen for incoming JSON data on TCP port 5000 and send that to an Elasticsearch instance.

Here’s a breakdown of the configuration:

1. input: the input that Logstash should listen to. In this case, it’s a TCP input with the following options:

• port: the port that Logstash should listen on. In this case, it’s port 5000.
• codec: the codec that Logstash should use to parse the incoming data.

2. output: This block specifies the output that Logstash should send the processed data. In this case, it’s an Elasticsearch output with the following options:

• hosts: the hosts that Logstash should send the data to. It’s the “elasticsearch” host on port 9200.

Log Generator Service

In this example, we’re using the docker container with a shell script to produce random logs and to see that everything works well. In real life, these could be your services where you want to produce records such as microservices, docker container logs, IoT logs, etc.

In the following code blocks, you can see the shell script and a Docker file to run the script in a container.

With this setup, you can start collecting, analyzing, and visualizing logs in Kibana using Elasticsearch and Logstash. The environment can be started using the following command on the terminal

docker-compose up

and to stop the docker-compose command is:

docker-compose down

Finally

Once everything goes well, you can access the Kibana on the localhost:5601 , and you can see the logs on localhost:5601/app/logs/streamunder the Observability > Logs > Stream tab.

You can access the codebase here.

Cheers!