Zero to a Million: Instrumenting LLMs with OTEL

Arize's OpenInference instrumentation has reached one million monthly downloads, marking a significant milestone in observability for AI using OpenTelemetry (OTEL). The journey has been challenging but rewarding as the team, along with other key players in the industry, paves the way for OTEL LLM instrumentation. They faced several challenges such as dealing with latent data, navigating lists in OTEL, avoiding attribute loss, handling futures and promises, managing streaming responses, and defining the scope of instrumentation. Despite these hurdles, Arize remains committed to shaping the future of standardization in LLM observability.