The art of log forwarding and log collection (at scale)


Cloud native orchestration systems such as Kubernetes, are smart enought smart to apply scale-on-demand logic for containerized applications. Internally the scheduling process is not so simple, and other areas also bring their own complexity such as Logging.

Since applications requires a logging interface , either for troubleshooting, monitoring or gather statistics, the process of data collection and forwarding requires specialized tools that can deal with different sources and data formats, as well have a failover mechanism to distribute the logs to the destination or aggregators in a reliable manner.

The following presentation will introduce the concepts of Logging in Kubernetes focusing on: collection, forwarding and aggregation patterns that will benefit the attendees with critial concepts that are a key to have a sacalable logging solution in cloud native environments. The following is a list of topics that will be covered:

  • Logging
    • Introduction
    • Common Logging standards
    • Challenges
    • Scaling
  • The Container Era
    • Introducing containers and Docker

    • Deploying at Scale

    • Orchestration: Kubernetes (k8s)

    • Logging Patterns

    • Log Forwarding
    • Log Aggregation

    • Performance

  • Fluent Bit: Log Forwarding
  • Fluentd: Log Aggregation
  • Logging at Scale
    • Choosing the right aggregation patterns

    • Reliability: in-memory or file system buffering ?

    • Integration with different backends

    • Other configurations

  • Demos




Ballroom A
Saturday, March 4, 2017 - 18:00 to 19:00