More data, more data

CloudFlare's Data Team reports on their edge network logs processing, detailing the numbers from an ordinary day in July 2016. They handle nearly 360TB of raw, Cap’n Proto event logs daily, using two Kafka clusters and a significant amount of hardware for data consolidation. The team highlights what has worked well in their system, such as the log forwarder, Kafka, persistence for log sharing, CitusDB, and the platform and SRE. They also discuss areas they want to improve or add in the near future, including making services more reliable, new analytics, new data pipelines, and better support for complex analysis. The Data Team invites feedback and comments on their work.