This article is the second part of the Beats series. It is recommended to previously read Beats Part 1 – Do you know Elasticsearch Beats? Introduction and use cases.
We are going to see how certain Beats can be deployed in a Kubernetes cloud infrastructure in order to extract all useful performance and availability metrics from all elements of the cluster.
Some Beats are designed to be installed in Kubernetes clusters in order to extract data from the whole cluster in a simple and centralised way, either at system level (cpu, memory, etc…), at application level (jvm heap, nginx requests, etc…), or k8s own resources (response time, DNS metrics, …).
One of the ways to install it effectively is as a Daemonset, so that a Pod with the desired Beat is raised for each node in the cluster so that each Pod extracts the data from the k8s Node on which it has been deployed.
Doing it this way allows us to use metricbeat, filebeat or heartbeat with the Autodiscover functionalities that, once they are correctly configured, monitoring an application will be as simple as putting k8s annotations at pod or deployment level to indicate what information we want to extract.
We will discuss the use of Autodiscover in the third part of this series of articles.
Installing Filebeat on Kubernetes
Kubernetes resources required
These resources need to be adapted and applied on a k8s cluster.
First identify which namespace will be used for the installation.
ServiceAccount and ClusterRole objects will be created to allow Filebeat to access the other namespaces in the cluster, list their pods, extract their metrics and metadata.
Apply these resources, either through kubectl apply or any Kubernetes resource manager.
Service Account
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat-dynamic
namespace: iometrics-pro #Adaptar
labels:
k8s-app: filebeat-dynamic
We create the ServiceAccount on the namespace to allow filebeat to interact with the kubernetes API.
ClusterRole
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: filebeat-dynamic
labels:
k8s-app: filebeat-dynamic
rules:
- apiGroups: [""] # "" Indica que se solicita acceder al "Core" de la API
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
We create a ClusterRole with permissions limited to the API resources we need to access.
ClusterRoleBinding
--- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: filebeat-dynamic subjects: - kind: ServiceAccount name: filebeat-dynamic namespace: kube-system roleRef: kind: ClusterRole name: filebeat-dynamic apiGroup: rbac.authorization.k8s.io
We apply a ClusterRoleBinding, to associate the ServiceAccount with previously created ClusterRoles.
DaemonSet
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat-dynamic
labels:
k8s-app: filebeat-dynamic
spec:
selector:
matchLabels:
k8s-app: filebeat-dynamic
template:
metadata:
labels:
k8s-app: filebeat-dynamic
name: filebeat-dynamic
spec:
serviceAccountName: filebeat-dynamic
terminationGracePeriodSeconds: 10
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.16.2 # Versión deseada
imagePullPolicy: Always
args: [
"-c", "/filebeat/filebeat.yml",
"-e"]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 10m
memory: 50Mi
volumeMounts:
- name: data
mountPath: /usr/share/filebeat/data
- name: config
mountPath: /filebeat/filebeat.yml
readOnly: false
subPath: filebeat.yml
volumes:
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
- name: config
configMap:
defaultMode: 0640
name: filebeat-config
We apply the Daemonset, which will be in charge of raising a Pod for each available node in the cluster. It indicates the origin of the filebeat image, process start arguments, the NODE_NAME environment variable that filebeat will use internally to know on which node it is running, CPU and memory limits for pod resource management and volume for filebeat data persistence, as well as the filebeat.yml that contains all the configuration that filebeat uses and that is mounted as a configMap that we define below.
ConfigMap
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
labels:
k8s-app: filebeat-dynamic
data:
filebeat.yml: |-
setup.template.enabled: true
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
processors:
- add_cloud_metadata:
- add_host_metadata:
output.elasticsearch:
... # See elasticsearch documentation to configure as needed.
It is necessary to adapt the configuration to the needs of the use case. See the official filebeat documentation for advanced configuration.
With the setup.template.enabled directive, when filebeat is started for the first time, the setup will be performed, which consists of sending to Elasticsearch the index templates that will carry the mappings and ILM policies necessary to store and structure the data correctly in the filebeat-* indexes.
There is also another option setup.dashboards.enabled that allows you to upload the predefined dashboards that can be viewed in Part 1. It needs additional configuration to point to Kibana, as the dashboards are uploaded to that component.
With filebeat.inputs, we are already telling filebeat to pull logs from all containers so as soon as it is up it will start sending them to
Verify the deployment
Once all the kubernetes resources have been correctly applied, we will proceed to verify that a pod has been successfully deployed for each node in the cluster, and that its status is “Running”.
It is also good practice to describe the Daemonset and a Pod, and to check the logs of a Pod to rule out error messages and verify that everything has worked correctly.
Verify data reception
Locate in Kibana, at the index configured for reception (default filebeat-*). Verify that the logs of the different containers deployed in the Kubernetes cluster are received correctly.
