Collecting and visualizing Kubernetes events using Loki and Kubernetes Event Exporter for effective
Prerequisites
The Kubernetes Event Exporter is a powerful tool that solves the problem of missed Kubernetes events. It exports these events to various outputs, improving observability and enabling custom alerts. With this exporter, exporting events to other tools becomes effortless, enhancing event observability, creating custom alerts, and aggregating event data effectively.
In this tutorial, we will use Loki as a log aggregator and Grafana for log visualization. Loki efficiently collects and stores log data, making it perfect for aggregating Kubernetes events. Grafana provides a user-friendly platform to create insightful dashboards and visualizations using the collected log data. By integrating these tools with the Kubernetes Event Exporter, you can achieve comprehensive observability, monitoring, and alerting for your Kubernetes cluster, enabling data-driven decision-making and efficient management of your applications and infrastructure.
In our case, we’ve got Grafana up and running! So, if you’re not familiar with the deployment process, I recommend checking out the documentation for deploying Grafana in a Kubernetes cluster.
Loki Deployment
To begin, we will deploy Loki as a lightweight single binary using Helm. As a first step, let’s create a values file and name it loki-values.yaml. Additionally, it’s important to mention that I have disabled authentication because Grafana cannot establish a connection with authentication enabled. This information was obtained from the following issue: link. Also, I encountered an issue when deploying the Helm chart in single binary mode. The error message “loki-memberlist can’t be resolved” prompted us to add the property “memberlist.service.publishNotReadyAddresses: true” to prevent it. We found the solution in this issue.
To deploy our Helm chart add Grafana’s chart repository to Helm:
After that we run a command to deploy our Loki, in our case it would be in the “monitoring” namespace. Debug flag I use to make sure that all configuration is perfect
Kubernetes Event Exporter
Once Loki is deployed, we will go ahead to deploy the Kubernetes event exporter. Just like with Loki, we will utilize the Helm chart from this repository, along with our custom values file named values.yaml.
In addition to this file, you need to include the “config” property, which I will describe below. Within the “config” property, you can specify the destination for the event exporter’s data. In our case, we will use Loki, but you can integrate many other applications as listed in the documentation. Since Loki does not have a direct integration, we will utilize it as a simple webhook, making requests to the Loki API. Furthermore, we will format our request to display comprehensive event data. The receiver is listed as a “dump” to easily access and read the logs via the command line using the command kubectl logs
.
Just like in the Loki deployment, we add the repo:
And deploy with our values.yaml file in the “monitoring” namespace:
Data visualization with Grafana
To conclude, let’s ensure that everything is functioning properly.
Now, we can conveniently visualize our logs in Grafana. To achieve this, let’s add Loki as a data source in Grafana.
Explore the logs using the query `{source=”event-exporter”} |= ``` to retrieve all logs from the event exporter.
Сonclusion
In conclusion, this tutorial demonstrated the process of collecting Kubernetes events and visualizing them on a Grafana dashboard. By utilizing the Kubernetes Event Exporter and Grafana Loki, event monitoring and visualization were achieved.
Last updated