Company
Date Published
Jan. 16, 2025
Author
Eric O'Connor
Word count
688
Language
English
Hacker News points
None

Summary

Fivetran has introduced a new benchmarking system to measure the performance of its data pipelines, enabling users to optimize their usage and observe the results of pipeline sync performance improvements. The benchmark focuses on replication from operational and transactional databases to data warehouses and data lakes, with an emphasis on loading from common OLTP relational databases like Oracle, PostgreSQL, SQL Server, and MySQL. A bulk historical load benchmark has been set up to evaluate throughput, while a TPROC-C workload will be used to assess day-to-day latency of high-volume database connections. The benchmark results show that Fivetran can import large datasets at speeds exceeding 500 GB/hour, demonstrating significant performance improvements over the past year, including expanded parallelism and optimized data load patterns. These advancements enable faster time to value for customers, allowing them to access their data when and where they need it.