Top 10 Changes and Key Improvements in Apache Kafka 4.0.0

In this post, we summarize the major changes in the recently officially released Apache Kafka 4.0.0 version. We will look at the most notable features compared to the previous versions and explain what these changes mean in real production environments and what improvements they can bring to your streaming infrastructure.

1. Goodbye to ZooKeeper

The biggest change in Kafka 4.0.0 is the removal of ZooKeeper and the Kraft becoming the default metadata management.

Previously, Kafka relied on ZooKeeper for metadata management, requiring an additional ZooKeeper ensemble, increasing complexity. The Kafka community proposed KIP-500 in 2020 to replace ZooKeeper with KRaft. With Kafka 3.x, KRaft was tested in various production stages, and in version 4.0.0, ZooKeeper is no longer supported.

Upgrade Steps:

1. Upgrade Kafka cluster to version 3.9.x
2. Migrate to KRaft cluster
3. Upgrade Kafka cluster to version 4.0.0

2. New Consumer Group Protocol

Kafka’s new consumer rebalance protocol, introduced in KIP-848, enhances stability and scalability by eliminating global synchronization barriers. Previously, when a consumer group rebalanced, all consumers had to stop temporarily, causing delays and performance issues.

Key Improvements:

• Lightweight Clients: The rebalance logic has shifted to the group coordinator, simplifying client operations.
• Incremental & Cooperative Rebalancing: Rebalancing now happens incrementally, reducing the impact of changes on the entire group.
• Simplified Protocol: The protocol is now more maintainable and easier to understand.

These enhancements reduce downtime and improve overall responsiveness, making Kafka more efficient in large-scale environments.

3. Queues for Kafka (Shared Group)

Traditional Kafka consumer groups lacked support for queue-based messaging patterns. KIP-932 introduces the concept of shared groups, allowing cooperative consumption within a topic.

Benefits of Shared Groups:

• Multiple consumers per partition: Unlike standard consumer groups, shared groups allow multiple consumers to read from the same partition.
• Flexible Scaling: More consumers can be added beyond the number of partitions, improving parallel processing.
• Individual Acknowledgment & Retry: Messages can now be individually acknowledged and retried, making work queue patterns easier to implement.

Note: KIP-932 is in an early access phase in Kafka 4.0.0 and is recommended for testing rather than production use.

4. Kafka Docker Image

Previously, developers had to rely on unofficial Kafka Docker images. With KIP-975, Apache Kafka now provides an official Docker image, distributed via the Apache Software Foundation’s Docker profile.

Key Features:

• Smaller Size: The Java 21-based image is approximately 457.86 MB when unzipped.
• Faster Startup: Kafka servers can now start in a container in about 0.816 seconds.
• Latest Java Support: The image is built with Java 21, ensuring performance and security improvements.

This official image enhances deployment reliability and maintains consistency across development, testing, and production environments.

5. Centralized Monitoring

Kafka’s monitoring capabilities have been significantly enhanced with KIP-714 and KIP-1076. Previously, system monitoring was limited due to the lack of centralized metrics collection.

Enhancements:

• KIP-714: Brokers can now collect client metrics (managers, producers, consumers) through plugins, improving observability.
• KIP-1076: Applications can register their own metrics with Kafka using the new registerMetricForSubscription and unregisterMetricFromSubscription methods.

These improvements make system performance analysis easier and provide better insights into Kafka cluster behavior.

6. Replication of User Internal Topics

In the past, Kafka’s MirrorMaker2 did not replicate topics with names ending in -internal or .internal, as they were considered system topics. This led to unintentional exclusions of custom topics that followed similar naming conventions.

ReplicationPolicy interface fixes:

Previously, topics were considered internal if their name matched a specific pattern, but now only topics that start with ‘mm2’ and end with ‘.internal’ are recognized as internal topics of MirrorMaker2. This prevents incorrect identification of user-defined topics.

Updated exclusion list of DefaultTopicFilter:

Previously, topics were excluded using the pattern .*[\-\.]internal, but now the more specific pattern ^mm2-.*\.internal$ is used to exclude only actual internal topics used by MirrorMaker2. This prevents incorrect exclusion of user-created topics.

This improvement removes constraints on topic naming for users and provides greater flexibility and accuracy when replicating data using MirrorMaker2.

7. Utilizing Log4J2

Previously, log4j 1.x version was used for log management of Apache Kafka. However, log4j 1.x has been discontinued since 2012, which was at risk of being exposed to security vulnerabilities. In particular, the Log4Shell vulnerability (CVE-2021–44228) that occurred in late 2021 was a serious remote code execution (RCE) vulnerability discovered in log4j 2.x version, causing a huge ripple effect worldwide.

In addition to these security issues, KIP-653 was proposed to improve the latest features and performance. This KIP aims to upgrade Kafka’s server-side log dependency from log4j 1.x to log4j2. KIP-653 upgraded Kafka’s logging to log4j2, improving:

• Security (eliminates risks like Log4Shell vulnerability)
• Logging features (supports modern log management tools)
• Compatibility (maintains existing configurations while upgrading backend dependencies)

8. Duration-Based Offset Reset Option

Previously, the auto.offset.reset option in Apache Kafka’s consumer settings could only be set to earliest or latest. This limitation caused difficulties when the consumer had a large amount of data to process or when only data for a specific period was to be reprocessed.

KIP-1106 introduced a new value, by_duration:<duration>, to the auto.offset.reset option to address this issue. This allows consumers to consume data from an offset that is 30 days prior to the current time.

Key changes:

auto.offset.reset option extension:

In addition to the existing earliest, latest, and none values, by_duration:<duration> format is now supported. Here, <duration> specifies a period in ISO8601 format, for example, P30D means 30 days.

How it works:

When the consumer cannot find the initial offset or the current offset no longer exists on the server, it automatically resets the offset to the previous timestamp for the specified period. For example, setting auto.offset.reset=by_duration:P30D will consume data from an offset that is 30 days prior to the current time.

This addition allows consumer applications to easily reprocess or analyze data from a specific period, increasing data processing flexibility and efficiency, and providing one more flexible option when resetting offsets in the event of a failure.

9. Goodbye to Legacy Client Protocols

Previously, Apache Kafka supported all protocol API versions since version 0.8.0 (released in December 2013). However, over time, supporting these legacy versions increased code complexity and testing burden, and in fact, their value decreased as the transition to modern clients took place. Accordingly, KIP-896 was proposed to remove legacy client protocol API support in Kafka 4.0.0.

Key changes:

Deprecated protocol API versions:

• Kafka 4.0 deprecates support for several legacy protocol APIs, including Produce (V0-V2), Fetch (V0-V3), and ListOffset (V0). This means that these APIs can no longer be used for communication between clients and brokers.

Minimum client version requirement:

• Due to these changes, you must use at least Kafka 2.1.0 (released in November 2018) or higher to be compatible with Kafka 4.0.0. This means that if you are using an older client version, you must upgrade.
• Importance of client upgrade:
• These protocol API deprecations highlight the need for client upgrades. Failure to upgrade producer and consumer clients to the latest version may result in compatibility issues with Kafka 4.0 brokers, which may cause data transmission and reception failures. Therefore, client upgrades cannot be delayed any longer, and migrating to the latest clients is essential for stable system operation.

10. Goodbye to Old Java Versions

It is time to move away from Java 8 and Java 11 and move to the latest versions such as Java 17. Apache Kafka 4.0.0 finally makes this change official.​ Java 8 support deprecation (KIP-750):

Java 8 support deprecation (KIP-750):

Java 8 was released in 2014 and has been used for a long time. However, its due for replacement and support for this version will be completely deprecated starting from Kafka 4.0.

Java 11 support deprecation (KIP-1013):

Java 11 is a LTS (Long-Term Support) version released in 2018 and has been widely. However, as Java 17 emerged as a new LTS version, support for Java 11 will be completely deprecated starting from Kafka 4.0. The need to move to the latest Java version:

Kafka 4.0.0 officially deprecates Java 8 and Java 11:

• Java 8 (KIP-750): Released in 2014, now outdated.
• Java 11 (KIP-1013): Replaced by Java 17, the latest LTS version.

Why Upgrade?

• Performance improvements
• Security enhancements
• Access to new Kafka features

Conclusion

Apache Kafka 4.0.0 introduces significant improvements in cluster management, consumer efficiency, messaging flexibility, deployment, monitoring, and replication. These updates simplify Kafka operations, reduce downtime, and enhance scalability, making Kafka more powerful than ever before.

For those upgrading from previous versions, careful planning is necessary, especially for the transition from ZooKeeper to KRaft. Stay updated with the latest KIPs and community discussions to make the most of these new features.