Real-Time Aggregation Features for Machine Learning (Part 2)

Tecton's solution for real-time aggregation features for machine learning uses a tiled time window approach, where the aggregation is broken down into compacted tiles of smaller time windows that store aggregations over the tile interval and a set of projected raw events at the head and tail of the aggregation time window. The configuration requires selecting and projecting raw events using SQL, defining the aggregation information using a simple DSL, and streaming ingestion to the online store. Batch ingestion is used for backfilling and forward-filling the offline store. Compacted data is produced by periodic batch Spark jobs that read from the streaming source's offline mirror, reducing the worst-case number of rows that have to be fetched from the store. The solution provides benefits such as ultra-fresh features, compute and memory-cost efficient processing, fast feature retrieval, and scalable storage requirements. Airbnb and Uber use Tecton's implementation for several years. The approach can be generalized beyond just time-window aggregations and extend to completely user-defined transformation steps.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.