The art of log forwarding and log collection (at scale)

Cloud native orchestration systems such as Kubernetes, are smart enought smart to apply scale-on-demand logic for containerized applications. Internally the scheduling process is not so simple, and other areas also bring their own complexity such as Logging.

Since applications requires a logging interface , either for troubleshooting, monitoring or gather statistics, the process of data collection and forwarding requires specialized tools that can deal with different sources and data formats, as well have a failover mechanism to distribute the logs to the destination or aggregators in a reliable manner.

The following presentation will introduce the concepts of Logging in Kubernetes focusing on: collection, forwarding and aggregation patterns that will benefit the attendees with critial concepts that are a key to have a sacalable logging solution in cloud native environments. The following is a list of topics that will be covered:

• Logging
  • Introduction
  • Common Logging standards
  • Challenges
  • Scaling
• The Container Era

  • Introducing containers and Docker

  • Deploying at Scale

  • Orchestration: Kubernetes (k8s)

  • Logging Patterns

  • Log Forwarding

  • Log Aggregation

  • Performance

• Fluent Bit: Log Forwarding
• Fluentd: Log Aggregation
• Logging at Scale

  • Choosing the right aggregation patterns

  • Reliability: in-memory or file system buffering ?

  • Integration with different backends

  • Other configurations

• Demos