Long-time Linux user Eduardo Silva will be giving a presentation at SCALE 14X on "Fluentd, the Open Source Data Collection Tool" at 4:30 on SCALE 14X Saturday. The SCALE Team caught up with Eduardo and asked him about his talk.
Q: Could you please introduce yourself and tell us a little about your background?
A: My name is Eduardo Silva, I have been using Linux since 1999 and doing open source development since 2000. I've always been intrigued about performance and networking services in general, for hence I got some fun developing Monkey Server which is a full Web Server (ARM, MIPS, x86 and x86_64) and Duda I/O web services framework mostly. I was a Google Summer of Code student and contributor for OLPC in the past, further administrator of Monkey for GSoC too, a wonderful journey!. Nowadays I am very proud to be part of the Open Source Engineering team at Treasure Data (we simplify data management), where I focus on developing logging solutions for Embedded/IoT and Cloud mostly. I like to spread the word about our experience solving data collection with Fluent Bit and Fluentd.
Q: You're giving a talk on "Fluentd, the Open Source Data Collection tool." Without tipping your hand on the actual talk, can you give us an idea of what we might expect?
A: Attendees will expect a real technical presentation about logging, as you can realize logs are not longer a simple file, they are a stream of data and it management it's hard. Due to the nature of different environments such as Servers, Mobile, IoT/Embedded and Containers, there is more complexity in the equation, consider that each environment may have multiple sources of logs with different formats, handle that is a consistent pain. I will share our experience dealing with these problems covering from data collection to having an unified logging layer in the cloud, general performance involved and reliability: what do we do to don't lose data. At the end of the day we collect over 1 million events per second, that's 100x the volume of Twitter and Fluentd is the core part of our success key and other few thousands around the globe.
Q: Is this your first visit to SCALE? If so, what are your expectations? If not, can you give us your impressions of the event?
A: Yes!, this is my first time at SCALE and I am very excited and happy to get this opportunity. One of the most interesting things to me is to get involved in the community, interact with people face-to-face, learn from them and share our humble experience. We all face similar problems and I strongly believe that cooperation delivers brilliant solutions.
Q: What do you think are the most challenging problems when handling data collection ?
A: If I have to give a short answer I'd focus on two words: performance and reliability. I have seen many implementations struggle when trying to scale a logging solution or simply lose data, from a business perspective low performance increase costs, even worse if we lose valuable information that affects: end user, devops and business. At Fluentd we care a lot about that and I think the community have been very graceful with their feedback, we are doing good and right. Now for a special case such as IoT and Embedded devices, we have been working in a custom open source solution called Fluent Bit which I will share more details in the presentation, join us!
Q: Is there anything else you'd like to add?
A: I'd be happy to reach and discover more Fluentd users, if you are one or want to know more about Fluentd please come to our booth or just drop me a message on twitter to @edsiper (just use the #scale14x to make things easier to filter). You can start getting in touch from now if you join our Slack channel at http://slack.fluentd.org, free talk!. If you have something in mind to share before Scale this is the right moment :). I will travel with some stickers and t-shirts, if you want some, lets met!
SCALE Team interview by Sean McCabe