What is OpenTelemetry?

Bringing a unified, open-source framework for collecting, processing and exporting all types of telemetry data (logs, metrics and traces)

In recent years, the rise of unified, open data formats has been a major trend in the data infrastructure space with Apache Iceberg and Delta for data lakehouses (read “What is Apache Iceberg?”). Instead of silo-ed data systems with unique formats, data infrastructure has pushed towards interoperability and performance optimizations across various platforms, reducing vendor lock-in and simplifying their data architecture. Similarly, in the realm of observability, OpenTelemetry is playing a comparable role by standardizing telemetry data—logs, metrics, and traces—into a unified, vendor-neutral framework. Just as Iceberg and Delta Lake have revolutionized data lakehouses by providing a consistent foundation for data analytics, OpenTelemetry is doing the same for telemetry data, enabling more integrated and flexible observability solutions across diverse environments.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework under CNCF (its 2nd most popular library) that provides a unified approach to collecting, processing, and exporting telemetry data, including logs, metrics, and traces. Inspired by the unification seen in data formats like Apache Iceberg and Delta Lake, which streamline data management in data lakehouses, OpenTelemetry standardizes telemetry data formats to unify observability across different systems.

The key innovations of OpenTelemetry are:

1. Unified Data Collection: OpenTelemetry consolidates logs, metrics, and traces into a single framework, simplifying observability and reducing the need for multiple tools.

2. Vendor-Neutral Instrumentation: It offers a vendor-agnostic approach, enabling organizations to instrument their applications once and direct data to any observability backend of their choice, avoiding vendor lock-in.

3. Automatic Instrumentation: OpenTelemetry supports automatic instrumentation, minimizing manual coding and reducing errors, making it easier for teams to deploy observability across their services.

4. Standardized Protocols: It uses standardized protocols like the OpenTelemetry Protocol (OTLP), ensuring consistent data formats across various tools, enhancing data correlation, and troubleshooting efficiency.

5. Extensibility and Modularity: OpenTelemetry’s modular design allows for easy customization, enabling companies to adapt telemetry data collection to their specific needs and environments.

Key players adopting OpenTelemetry

OpenTelemetry’s growth is driven by widespread adoption and integration across major observability and cloud platforms:

• Splunk: A major contributor to OpenTelemetry, Splunk has integrated it into its observability suite, enhancing data collection and tracing capabilities.

• Azure Monitor: OpenTelemetry is supported within Azure Monitor and Application Insights, providing enhanced visibility across Microsoft’s cloud and observability tools.

• Grafana: Supports OpenTelemetry to integrate telemetry data directly into its dashboards, improving visualization and analysis.

• Snowflake: Snowflake has adopted OpenTelemetry to provide better observability for data workflows and performance monitoring.

• Databricks: Integrates OpenTelemetry for enhanced tracing and monitoring within data engineering and machine learning pipelines.

• AWS, Google Cloud, Microsoft Azure: These cloud providers ensure OpenTelemetry compatibility, enabling consistent monitoring across their cloud services.

• Datadog: Datadog integrates OpenTelemetry into its platform, enhancing its monitoring capabilities with unified trace, metrics, and log data, allowing users to adopt a vendor-neutral approach to observability.

• Dynatrace: Fully integrates OpenTelemetry into its platform, providing end-to-end observability and data ingestion.

• New Relic: Offers strong support for OpenTelemetry, allowing for comprehensive telemetry data collection and analysis without vendor restrictions.

• Elastic: Integrates OpenTelemetry into its observability stack, supporting seamless ingestion and analysis of telemetry data.

• Cribl: Cribl supports OpenTelemetry by allowing users to route, shape, and enrich telemetry data before sending it to various observability backends, making data management more efficient and cost-effective.

This broad compatibility establishes OpenTelemetry as the standard for observability, reducing integration overhead and enhancing performance monitoring across diverse technologies. 1,106 companies are contributing code, with the top 3 being Splunk (27%), Microsoft (17%) and Lighstep / Servicenow (8%).

Key trends in observability

Grafana has emerged as a pioneer and champion of “open observability”, crossing $300M ARR and raising recently at a $6B valuation. Grafana’s observability survey of 300+ industry leaders provides key trends in the future of observability.

1. Open source dominates the observability landscape: Open source solutions like Prometheus (89%) and OpenTelemetry (85%) are widely adopted, with nearly 40% of organizations using both. More than half of respondents increased their usage of these tools over the past year, underscoring the shift toward open standards.

   

   

2. Cost concerns are increasingly prominent: Cost is the top concern for over half of respondents, with related issues such as cardinality, unpredictable billing, and vendor lock-in also frequently mentioned. It’s widely reported OpenAI’s Datadog bill is over $100M, while Coinbase spends more than $65M on Datadog in 2023.

   

3. Observability approaches are still evolving: A little over half of organizations have adopted a proactive observability approach, indicating growth in the field. However, many are still reactive rather than systematic, often discovering problems through customer feedback rather than preemptive monitoring.

4. Tool and data sprawl remain significant challenges: Over two-thirds of teams use at least four observability tools, with more than 60 technologies in use overall. Half of Grafana users have six or more data sources connected, leading to complexity and overhead. Centralizing observability efforts has led to time or cost savings for 79% of teams. Large companies have the most complexity.

   

   

   

5. AI’s potential in observability is promising but unrealized: AI’s role in observability is still emerging, with practitioners optimistic about its future impact on incident response and adoption. Anomaly detection tops the list of desired AI features, with over 75% of respondents expressing interest.

The report provides insights on the observability tools that Grafana customers also use:

Conclusion

OpenTelemetry is rapidly transforming the observability space, giving rise an increasing breadth in tooling due to standardization of telemetry data collection. All the key industry players from Splunk, Datadog, Dynatrace, to the major cloud vendors are embracing the open data architecture. Open architectures are the area that customers are most excited about, followed by AI and its potential for use cases like anomaly detection in observability systems.

If you are a founder building in observability, I’d love to hear from you: chris@wing.vc