Export your model inference metrics to your favorite observability tool

Baseten has introduced an export metrics integration that allows users to export model inference metrics like response time, replica count, and hardware utilization to observability platforms such as Grafana, New Relic, Datadog, and Prometheus. This feature enhances production model management workflows by providing a single source of truth, fine-grained metrics and control, and custom alerts. The supported metrics include inference request count, end-to-end response time, replica count, and hardware usage for CPU and GPU resources. Metrics are exported using the vendor-neutral OpenTelemetry standard and can be scraped by observability tools at a set interval. Baseten's integration supports over 800 tools in the OpenTelemetry registry, with specific documentation available for popular platforms like Grafana, New Relic, Datadog, and Prometheus.