Scaling data pipelines on Kubernetes

Airbyte, an open-source data replication tool, initially used Docker for its containerization needs. However, as the user base grew and data volumes increased, scaling became a challenge. Kubernetes emerged as a potential solution due to its capability of horizontally scaling workloads. The main challenge in adapting Airbyte from Docker to Kubernetes was passing data between different containers, especially since there are no guarantees that pods will be scheduled on the same Kubernetes nodes. To address this issue, the team used Linux's minimalist and modular approach to software development by leveraging various networking tools like socat and named pipes. They also employed the sidecar pattern to create a sidecar socat container alongside the main container within the same Kube pod. This allowed them to switch between different networking tools without affecting users, ensuring isolation and encapsulation. The team used the Kubernetes API to dynamically create job containers as Kubernetes pods whenever Airbyte receives a job. They also utilized named pipes for piping data between the sidecar and main containers within the same Kube pod. The worker pod in an Airbyte Kubernetes deployment orchestrates all necessary pods to complete a job, ensuring smooth communication between source and destination pods. Despite some challenges like managing multiple STDIO streams and inefficient networking, this architecture has enabled Airbyte to scale its operations effectively. The team is currently working on V2 of the architecture to further improve efficiency and performance.