Kubernetes Logging with fluentd and logz.io
Contents
tl;dr: By using logz.io, it is relatively easy to outsource Kubernetes logfiles that do not contain sensitive data to an external service and analyze them there with Kibana.
With the self-hosted Kubernetes Cluster in place the next step is the setup of a centralized logging system.
This example will focus on Fluentd1 as log collector and Logz.io2 as backend.
Basics
Fluentd is an open source collector for log files and a good choice when deployed into Kubernetes. We will create a DaemonSet3 and use the fluentd-kubernetes-daemonset4 docker image. A DaemonSet ensures, that the configured pods run on each node in the cluster and new notes are automatically provisioned.
You can find multiple tags of the image which provide support for different backends (e.g. Elasticsearch, Cloudwatch or Stackdriver).
As a backend we will use Logz.io which is build on the ELK stack and offers a free Community plan with 3GB data daily.
Setup
All required Kubernetes files are available in the GitHub repository5 and should be cloned first.
Namespace
To separate the logging from the existing namespaces, we create a new namespace kube-logging first.
kubectl create -f logging-namespace.yml
Secret
Logz.io requires a Token and Type as authentication parameters. You can find the values after a sucessful registration in your settings.
Both values are encoded in base64 and added to the fluentd-secret.yml.
echo -n YOUR_TOKEN | base64
echo -n YOUR_TYPE | base64
Now we can create the secret in our cluster:
kubectl create -f fluentd-secret.yml
Daemonset, ServiceAccount and ClusterRole
The last step is the creation of a ServiceAccount used by the fluentd pods and a matching ClusterRole.
kubectl create -f fluentd-daemonset.yml
Check installation
You can now verify the process of the DaemonSet deployment:
kubectl get ds -n kube-logging
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
fluentd 2 2 2 2 2 <none> 27s
And get the Pods:
kubectl get pods -n kube-logging
NAME READY STATUS RESTARTS AGE
fluentd-t644b 1/1 Running 0 67m
fluentd-t6sxw 1/1 Running 0 67m
Inside the logs of fluentd you should see the parsed log-files:
kubectl logs -f -n kube-logging fluentd-5wg4c
2019-01-10 08:08:54 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-scheduler-kubernetes_kube-system_kube-scheduler-20ba3a20f492e72da577408089cde0122046483f954f5e41ee2e1698dc34633f.log
2019-01-10 08:08:54 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-flannel-ds-grnn2_kube-system_install-cni-e97eb766bc841909bdb412c7b6b580855950175d62bac1fd81e518fa0371a49e.log
2019-01-10 08:08:54 +0000 [info]: #0 fluentd worker is now running worker=0
2019-01-10 08:09:36 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 2, pod_cache_size: 4, pod_cache_watch_misses: 5, pod_cache_watch_ignored: 1, pod_cache_watch_delete_ignored: 1, namespace_cache_api_updates: 4, pod_cache_api_updates: 4, id_cache_miss: 4
2019-01-10 08:10:07 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 2, pod_cache_size: 4, pod_cache_watch_misses: 5, pod_cache_watch_ignored: 1, pod_cache_watch_delete_ignored: 1, namespace_cache_api_updates: 4, pod_cache_api_updates: 4, id_cache_miss: 4
...
Summary
If everything works it will take only seconds until the first logs become available in the Kibana Dashboard6.
Comments