Building a Streaming Platform in Go for Postgres

PeerDB has transitioned from a pull-and-push model to a more efficient streaming approach using Go channels for its Postgres-first data-movement platform. The new streaming approach involves buffering and concurrently pushing data to the target in batches, as it is pulled from PostgreSQL. This pipelining of data transfer offers significant advantages such as improved efficiency, reduced latency, efficient resource management, and better error handling. PeerDB's performance improvement is remarkable with throughput reaching 10-12k Transactions Per Second (TPS) and minimal lag of 1-5 seconds compared to roughly 30 seconds in the previous approach. Future enhancements include spilling the record stream to disk to prevent Out-of-Memory (OOM) issues for handling larger datasets without sacrificing performance or reliability.