Company
Date Published
Author
Lambert Le Manh
Word count
927
Language
English
Hacker News points
None

Summary

This company, incident.io, was facing a bottleneck due to its growing data ecosystem and the inability of its current data platform to scale. The Data Engineer role I joined brought about significant improvements to the development platform, including enhanced dbt workflow and CI, as well as reduced costs through optimized Docker image management and cost-effective Fivetran BigQuery usage. Additionally, exploring alternative data ingestion tools was conducted, with Google Datastream being ruled out due to high upsert costs, and Fivetran remaining the most cost-effective option. The company's platform costs were also analyzed and reduced by nearly 75% through clustering of destination tables in BigQuery, resulting in a more efficient data management workflow.