Beats Part 2 – Installing Beats on a Kubernetes cluster (Filebeat)

This article is the second part of the Beats series. It is recommended to previously read Beats Part 1 – Do you know Elasticsearch Beats? Introduction and use cases.

We are going to see how certain Beats can be deployed in a Kubernetes cloud infrastructure in order to extract all useful performance and availability metrics from all elements of the cluster.

Some Beats are designed to be installed in Kubernetes clusters in order to extract data from the whole cluster in a simple and centralised way, either at system level (cpu, memory, etc…), at application level (jvm heap, nginx requests, etc…), or k8s own resources (response time, DNS metrics, …).

One of the ways to install it effectively is as a Daemonset, so that a Pod with the desired Beat is raised for each node in the cluster so that each Pod extracts the data from the k8s Node on which it has been deployed.

Doing it this way allows us to use metricbeat, filebeat or heartbeat with the Autodiscover functionalities that, once they are correctly configured, monitoring an application will be as simple as putting k8s annotations at pod or deployment level to indicate what information we want to extract.

We will discuss the use of Autodiscover in the third part of this series of articles.

Installing Filebeat on Kubernetes

Kubernetes resources required

These resources need to be adapted and applied on a k8s cluster.
First identify which namespace will be used for the installation.

ServiceAccount and ClusterRole objects will be created to allow Filebeat to access the other namespaces in the cluster, list their pods, extract their metrics and metadata.

Apply these resources, either through kubectl apply or any Kubernetes resource manager.

Service Account

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat-dynamic
  namespace: iometrics-pro #Adaptar
  labels:
    k8s-app: filebeat-dynamic

We create the ServiceAccount on the namespace to allow filebeat to interact with the kubernetes API.

ClusterRole

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat-dynamic
  labels:
    k8s-app: filebeat-dynamic
rules:
- apiGroups: [""] # "" Indica que se solicita acceder al "Core" de la API
  resources:
  - namespaces
  - pods
  - nodes
  verbs:
  - get
  - watch
  - list

We create a ClusterRole with permissions limited to the API resources we need to access.

ClusterRoleBinding

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat-dynamic
subjects:
- kind: ServiceAccount
  name: filebeat-dynamic
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat-dynamic
  apiGroup: rbac.authorization.k8s.io

We apply a ClusterRoleBinding, to associate the ServiceAccount with previously created ClusterRoles.

DaemonSet

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: filebeat-dynamic
 labels:
   k8s-app: filebeat-dynamic
spec:
 selector:
   matchLabels:
     k8s-app: filebeat-dynamic
 template:
   metadata:
     labels:
       k8s-app: filebeat-dynamic
     name: filebeat-dynamic
   spec:
     serviceAccountName: filebeat-dynamic
     terminationGracePeriodSeconds: 10
     hostNetwork: true
     dnsPolicy: ClusterFirstWithHostNet
     containers:
     - name: filebeat
       image: docker.elastic.co/beats/filebeat:7.16.2 # Versión deseada
       imagePullPolicy: Always
       args: [
         "-c", "/filebeat/filebeat.yml",
         "-e"]
       env:
       - name: NODE_NAME
         valueFrom:
           fieldRef:
             fieldPath: spec.nodeName
       securityContext:
         runAsUser: 0
       resources:
         limits:
           cpu: 200m
           memory: 200Mi
         requests:
           cpu: 10m
           memory: 50Mi
       volumeMounts:
       - name: data
         mountPath: /usr/share/filebeat/data
       - name: config
         mountPath: /filebeat/filebeat.yml
         readOnly: false
         subPath: filebeat.yml
     volumes:
     - name: data
       hostPath:
         path: /var/lib/filebeat-data
         type: DirectoryOrCreate
     - name: config
       configMap:
         defaultMode: 0640
         name: filebeat-config

We apply the Daemonset, which will be in charge of raising a Pod for each available node in the cluster. It indicates the origin of the filebeat image, process start arguments, the NODE_NAME environment variable that filebeat will use internally to know on which node it is running, CPU and memory limits for pod resource management and volume for filebeat data persistence, as well as the filebeat.yml that contains all the configuration that filebeat uses and that is mounted as a configMap that we define below.

ConfigMap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    k8s-app: filebeat-dynamic
data:
  filebeat.yml: |-
    setup.template.enabled: true
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    output.elasticsearch:
    ... # See elasticsearch documentation to configure as needed.

It is necessary to adapt the configuration to the needs of the use case. See the official filebeat documentation for advanced configuration.

With the setup.template.enabled directive, when filebeat is started for the first time, the setup will be performed, which consists of sending to Elasticsearch the index templates that will carry the mappings and ILM policies necessary to store and structure the data correctly in the filebeat-* indexes.

There is also another option setup.dashboards.enabled that allows you to upload the predefined dashboards that can be viewed in Part 1. It needs additional configuration to point to Kibana, as the dashboards are uploaded to that component.

With filebeat.inputs, we are already telling filebeat to pull logs from all containers so as soon as it is up it will start sending them to

Verify the deployment

Once all the kubernetes resources have been correctly applied, we will proceed to verify that a pod has been successfully deployed for each node in the cluster, and that its status is “Running”.

It is also good practice to describe the Daemonset and a Pod, and to check the logs of a Pod to rule out error messages and verify that everything has worked correctly.

Verify data reception

Locate in Kibana, at the index configured for reception (default filebeat-*). Verify that the logs of the different containers deployed in the Kubernetes cluster are received correctly.