How to use OpenTelemetry for Kafka Monitoring

Apache Kafka is a high-throughput, low-latency platform for handling real-time data feeds. Its storage layer is in essence a massively scalable pub/sub message queue designed as a distributed transaction log. It can be used to process streams of data in real-time, building up a commit log of changes.

Kafka has strong ordering guarantees that enable it to handle all sorts of dataflow patterns including very low latency messaging and efficient multicast publish / subscribe.

Apache Kafka derivatives including Confluent Kafka, IBM Event Streams, and Cloudera Kafka add in additional support and functionality over and above the default functions that are included as part of a normal installation.

Why use Kafka Monitoring Tools

There are many reasons why you would want to use Kafka monitoring tools. For example, if you are running a Kafka cluster in production, it is vitally important that you are able to monitor the health of the cluster and ensure that it is running without any problems. In addition, Kafka monitoring tools can help to identify performance bottlenecks and optimize your system’s configuration.

Generally speaking, Kafka monitoring tools provide you with visibility into the inner workings of your Kafka cluster, which can be extremely valuable in terms of both troubleshooting and performance tuning. It is important to ensure that you use one or more Kafka monitoring tools to protect the valuable data and transactions that are important to your business.

What is OpenTelemetry?

OpenTelemetry is an open-source vendor-agnostic set of tools, APIs and SDKs that have been assembled in order to act as a solution for instrumenting code and tracing messaging within distributed systems. OpenTelemetry is a collaboration between many developers to attempt to standardize ways to provide and collect telemetry data and to increase the observability of data while it is inside the system.

Transaction and message tracing can be undertaken by instrumenting your code with an OpenTelemetry “Collector”. This is done by utilizing some of the libraries that are part of the OpenTelemetry suite. Data can then be sent to a backend platform such as Nastel XRay to process and analyze the data that has been collected.

How Does OpenTelemetry Help with Kafka Monitoring?

It is possible to use OpenTelemetry to introduce relevant tracking metadata into the headers that are generated by Kafka. This makes it possible to include a unique key,value and timestamp which will then make it possible to identify and track the progress of each transaction as it makes its way through the Kafka system. However this isn’t always necessary. Nastel XRay can use existing common fields, such as payment id or invoice number, to achieve the same objective.

OpenTelemetry can help you to understand and resolve any performance bottlenecks within your Apache Kafka installation.

As a real-world business example, it may cost more to run your Apache Kafka clusters if they are not performing at the optimum level. If it is taking too long and using too much processing power to complete what should be a relatively simple transaction or set of transactions, this can have real-life financial consequences so it makes sense to use OpenTelemetry to ensure that you are able to optimize your system fully.

You can read more details about Nastel’s complete integration infrastructure management platform for Kafka here as well as even more depth on its transaction tracking solution here.