Reproducible Machine Learning with Buildkite

Culture Amp uses Buildkite to test hundreds of potential changes to their machine learning models in a reproducible, traceable, and scalable manner. The company's Natural Language Processing (NLP) team specializes in extracting insights from data and feeding them back to customers. They automate the model training process using Buildkite, AWS Batch, Metaflow, and Weights & Biases. This enables tracking of the ID of the model and parameters used for a given training run. The deployment flow involves pulling in the model artifacts from S3 into Buildkite, pushing them to where they’re needed in the production accounts, creating a SageMaker endpoint, and serving it behind a Fargate web application hosted in production.