Monitoring Kafka Performance: What Metrics Matter? 

Running Apache Kafka in production? You know monitoring is a must. But with all those metrics coming at you, it’s easy to get lost in the weeds. After a while, you start to figure out that monitoring everything isn’t really worth it. It’s about focusing on a few key metrics that give you the biggest bang for your buck. Here’s a breakdown of the most important Kafka performance metrics to keep your eye on. 

1. Broker Health Metrics 

Brokers are the backbone of your Kafka cluster. If they’re not healthy, your cluster’s in trouble. Metrics like CPU usage, memory usage, and disk I/O are crucial to understanding broker performance. If your CPU usage is through the roof, it might mean your brokers can’t keep up with the workload. Memory issues? Could be garbage collection making things worse. 

Let’s imagine a scenario where brokers suddenly slow to a crawl. After some investigation, it turns out that disk I/O is the culprit, causing the bottleneck. By tracking those disk read/write times early on, you catch the problem and upgrade the disks before things get worse. Keeping tabs on these broker health metrics can save you from a world of hurt. 

2. Topic and Partition Metrics 

Topics and partitions are where the magic happens in Kafka. But if things aren’t smooth, the whole system grinds down. Two metrics you need to watch: UnderReplicatedPartitions and OfflinePartitionsCount. 

UnderReplicatedPartitions tells you if your partitions aren’t fully replicated, which happens when a broker lags or goes down. OfflinePartitionsCount is even more serious—it means partitions are completely offline, potentially leading to data loss. 

These metrics help you catch replication issues before they blow up in your face. 

3. Consumer Lag 

Ah, consumer lag—the bane of real-time data processing. This metric shows how far behind your consumers are from the latest message in a partition. You don’t want this number to climb, trust me. 

Imagine a scenario where a consumer group is constantly lagging behind. No one notices until it’s too late, and by then, downstream systems are a mess. Monitoring consumer lag helps ensure your consumers don’t fall behind the data flow. 

4. Request Latency 

Request latency measures how long it takes brokers to process requests. This can give you a heads-up when something’s off—maybe it’s network trouble, resource strain, or a misconfiguration somewhere. 

The earlier you catch it, the quicker you can diagnose and fix the issue before it spreads. 

5. Network Throughput 

Kafka’s job is moving data, so network throughput metrics like BytesInPerSec and BytesOutPerSec are essential. They tell you how much data your brokers are moving at any given time. 

If throughput dips unexpectedly, it’s a sign something’s wrong—whether it’s a consumer failure or another system slowing down. 

6. Disk Usage and I/O 

Kafka leans hard on disk storage. Metrics like LogDir show how much space each topic is eating up, while disk I/O metrics reveal how well data is being read or written to disk. 

Disk usage can sneak up on you, causing write errors or even data loss. Keep an eye on this and clean up or add space before it becomes a problem. 

7. Garbage Collection 

Kafka runs on the JVM, and garbage collection can be a killer if not handled properly. Metrics like GCCount and GCTimeMillis help you track how often and how long garbage collection is happening. 

Imagine a situation where a Kafka cluster tanks because garbage collection happens way too often. By keeping an eye on GC metrics, you can avoid performance hits before they escalate. 

8. Leader Election Metrics 

Leader elections are normal, but if they’re happening all the time, it could mean trouble. Frequent leader elections can indicate instability in the cluster, often caused by network problems. Metrics like LeaderElectionRateAndTimeMs track how often and how long leader elections are taking. 

In one scenario, frequent leader elections were caused by network instability. Tracking this metric allowed the team to stabilize the cluster before it worsened. 

9. ISR (In-Sync Replicas) Shrink and Expansion 

ISR shrinkages are another sign of trouble. IsrShrinksPerSec and IsrExpandsPerSec metrics show how the ISR is changing. Frequent shrinkages mean replicas are lagging behind, which affects data durability. 

If you’re seeing lots of shrinkages, it’s time to dig deeper to avoid potential data loss. 

10. Zookeeper Metrics 

Even though Kafka is moving away from Zookeeper, many setups still rely on it. Monitoring Zookeeper metrics like ZookeeperRequestLatencyMs and ZookeeperAliveConnections helps ensure it’s not causing Kafka coordination issues. 

Make sure Zookeeper stays healthy, or you’ll have bigger problems on your hands. 

Conclusion 

Monitoring Kafka performance isn’t just about collecting data—it’s about knowing what to do with it. By focusing on key metrics like broker health, topic and partition status, consumer lag, request latency, network throughput, disk usage, garbage collection, leader elections, ISR changes, and Zookeeper health, you can spot issues early and keep your Kafka cluster running smoothly. 

When you monitor what matters, you can take proactive steps to keep everything humming along. Don’t get bogged down in all the noise—focus on what counts and take action when needed.