fluent / fluentd-kubernetes-daemonset

Fluentd Daemonset for Kubernetes > [!CAUTION] > README.md is generated from templates/README.md.erb Supported tags and respective links See also dockerhub tags page: https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset/tags Debian Current stable > [!TIP] > Since v1.17.0, the container image build process has been migrated from automated builds on hub.docker.com to GitHub Actions. > This is because there were limitation about the number of automated builds on hub.docker.com. > Now, there is no limitation about the number of build pipelines. > > Note that there were some restrictions to ship daemonset images for v1.16.5 or older ones before: > * , images (x86_64/arm64) won't be published anymore > * , , , arm64 images won't be published anymore (x86_64 only supported) > If you want to use above non published images, build it by yourself. Dockerfile itself is still maintained in this repository. Multi-Arch images • - • • - • • - • - • - • • - • • - • • - • • - • • - • • - • • - • • - • • - • • - • • - • • - • • - • • - • x86_64 images • Dockerfile • - • Dockerfile • - • Dockerfile • - • • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - arm64 images • Dockerfile • - • Dockerfile • - • Dockerfile • - • • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - • Dockerfile • - You can also use tag to refer latest v1 image, e.g. . On production, strict tag is better to avoid unexpected update. See dockerhub's tags page for older tags. What is Fluentd? Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. > www.fluentd.org Image versions Fluentd versioning is as follows: | Series | Description | |--------|-------------------------------------| | v1.x | current stable | Settings Default image version Default YAML uses latest v1 images like . If you want to avoid unexpected image update, specify exact version for like . Run as root This is for v0.12 images. In Kubernetes and default setting, fluentd needs root permission to read logs in and write to . To avoid permission error, you need to set environment variable to in your Kubernetes configuration. Use your configuration These images have default configuration and support some environment variables for parameters but it sometimes doesn't fit your case. If you want to use your configuration, use ConfigMap feature. Each image has following configurations: • fluent.conf: Destination setting, Elaticsearch, kafka and etc. • kubernetes.conf: k8s specific setting. input for log files and filter • tail_container_parse.conf: parser setting for . See also "Use CRI parser for containerd/cri-o" logs section • prometheus.conf: prometheus plugin for fluentd monitoring • systemd.conf: systemd plugin for collecting systemd-journal log. See also "Disable systemd input" section. Overwrite conf file via ConfigMap. See also several examples: • Cluster-level Logging in Kubernetes with Fluentd • https://github.com/fluent/fluentd-kubernetes-daemonset/pull/349#issuecomment-579097659 Use CRI parser for containerd/cri-o logs This feature is available since v1.12.0-xxx-1.1. By default, these images use parser for files because docker generates json formatted logs. On the other hand, containerd/cri-o use different log format. To parse such logs, you need to use parser instead. You can use parser by overwriting via ConfigMap. See also CRI parser README Use FLUENT_CONTAINER_TAIL_PATH to change container logs folder You can update the default path for the container logs i.e /var/log/container/*.log and also one can add multiple path as defined in this fluentd document https://docs.fluentd.org/input/tail#path Use FLUENT_CONTAINER_TAIL_EXCLUDE_PATH to exclude specific container logs Since v1.9.3 or later images. You can exclude container logs from with . If you have a trouble with specific log, use this envvar, e.g. . • parameter document: https://docs.fluentd.org/input/tail#exclude_path • Fluentd log issue with backslash: https://github.com/fluent/fluentd/issues/2545 Use LD_PRELOAD to enable jemalloc memory allocator Since v1.17.0-1.3/v1.16.5-1.3, jemalloc memory allocator is disabled by default. This is because that combination of systemd plugin and jemalloc memory allocator causes a crash bug. ( in typical) If you don't use systemd plugin at all, you can enable jemalloc memory allocator explicitly via env: parameter. Disable systemd input If you don't setup systemd in the container, fluentd shows following messages by default configuration. You can suppress these messages by setting to environment variable in your kubernetes configuration. Disable prometheus input plugins By default, latest images launch plugins to monitor fluentd. You can disable prometheus input plugin by setting to environment variable in your kubernetes configuration. Disable sed execution on elasticsearch image This is for older images. Latest elasticsearch images don't use sed. By historical reason, elasaticsearch image executes command during startup phase when or is specified. This sometimes causes a problem with read only mount. To avoid this problem, set "true" to environment variable in your kubernetes configuration. Running on OpenShift This daemonset setting mounts as service account so you need to run containers as privileged container. Here is command example: ` oc project kube-system oc create -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml oc adm policy add-scc-to-user privileged -z fluentd oc pa…