The art of log forwarding and log collection (at scale)
Cloud native orchestration systems such as Kubernetes, are smart enought smart to apply scale-on-demand logic for containerized applications. Internally the scheduling process is not so simple, and other areas also bring their own complexity such as Logging.
Since applications requires a logging interface , either for troubleshooting, monitoring or gather statistics, the process of data collection and forwarding requires specialized tools that can deal with different sources and data formats, as well have a failover mechanism to distribute the logs to the destination or aggregators in a reliable manner.
The following presentation will introduce the concepts of Logging in Kubernetes focusing on: collection, forwarding and aggregation patterns that will benefit the attendees with critial concepts that are a key to have a sacalable logging solution in cloud native environments. The following is a list of topics that will be covered:
- Common Logging standards
- The Container Era
Introducing containers and Docker
Deploying at Scale
Orchestration: Kubernetes (k8s)
- Log Forwarding
- Fluent Bit: Log Forwarding
- Fluentd: Log Aggregation
- Logging at Scale
Choosing the right aggregation patterns
Reliability: in-memory or file system buffering ?
Integration with different backends