Kafka Core Concepts: From Consumer Groups to High Throughput

Kafka is often introduced as a distributed event-streaming platform, but that description does not explain how it behaves under load—or why seemingly small configuration changes can alter its delivery guarantees.

To use Kafka confidently, you need a mental model that connects five areas:

1. How consumer groups divide work
2. How offsets track progress
3. How schemas evolve without breaking applications
4. What happens inside the producer after send()
5. Why a disk-backed platform can still achieve high throughput

This article connects those ideas and highlights the production details that simplified explanations often miss.

Version note: Configuration defaults in this article are based on Apache Kafka 4.x. In particular, linger.ms defaults to 5 ms from Kafka 4.0 onward; older references commonly show 0.

1. Consumer Groups and Partition Assignment

A consumer group is a set of consumers that cooperate to read one or more topics. Within a group, every subscribed partition has exactly one active consumer at a time.

Consider an orders topic with four partitions:

Topic: orders

P0 ──► Consumer A
P1 ──► Consumer B       Consumer group: order-svc
P2 ──► Consumer C
P3 ──► Consumer D

If another consumer joins the same group, Kafka rebalances the assignments. With four partitions and five consumers, one consumer remains idle because Kafka cannot assign the same partition to two active consumers in one group.

The scaling rule is therefore straightforward:

Useful consumer parallelism within a group is bounded by the number of assigned partitions.

Adding partitions can increase potential parallelism, but it is not a free operation. More partitions mean more metadata, files, replication work and rebalance complexity. They can also change key distribution for newly produced records.

Partition ownership does not eliminate duplicates

The one-consumer-per-partition rule prevents two current members of the same group from intentionally processing the partition simultaneously. It does not guarantee exactly-once business processing.

Suppose a consumer:

1. Reads record 42
2. Updates a database
3. Crashes before committing its new offset

After reassignment, another consumer resumes from the previous committed position and processes record 42 again. The application must therefore use idempotent processing, deduplication or an appropriate transactional pattern.

Java consumer example

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-svc");
props.put(
    ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
    StringDeserializer.class
);
props.put(
    ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
    StringDeserializer.class
);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("orders"));

    while (true) {
        ConsumerRecords<String, String> records =
            consumer.poll(Duration.ofMillis(100));

        for (ConsumerRecord<String, String> record : records) {
            process(record);
        }
    }
}

2. Offsets and Commit Management

An offset is a monotonically increasing position within a partition. It is unique only within that partition.

For example:

Partition 0

Record offset:     0   1   2   3   4   5   6
                              ▲
                         processed 4

Committed position: 5
Restart position:   5

Kafka commits the offset of the next record to consume. If offset 4 has been processed successfully, the committed value should normally be 5.

For consumer groups, committed positions are stored in Kafka’s internal __consumer_offsets topic.

Position and committed position are different

The consumer’s current position advances as poll() returns data. The committed position changes only when an offset commit succeeds.

That distinction creates three common processing models:

Commit timing	Possible failure outcome	Typical semantic
Before processing	Work can be skipped after a crash	At-most-once
After processing	Completed work can be repeated	At-least-once
Kafka transaction covering output and offsets	Kafka output and consumed positions commit atomically	Exactly-once within the supported Kafka scope

Manual offset commit

props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

ConsumerRecords<String, String> records =
    consumer.poll(Duration.ofMillis(100));

Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();

for (ConsumerRecord<String, String> record : records) {
    process(record);

    offsets.put(
        new TopicPartition(record.topic(), record.partition()),
        new OffsetAndMetadata(record.offset() + 1)
    );
}

consumer.commitSync(offsets);

In real applications, be careful when processing records concurrently. Committing the largest offset while an earlier record is still running can cause Kafka to skip that unfinished work after a restart.

3. Schema Registry and Safe Schema Evolution

A Schema Registry stores and versions event schemas such as Avro, Protobuf and JSON Schema. It creates an explicit contract between producers and consumers.

The typical flow is:

Producer ──register schema──► Schema Registry
Producer ◄────schema ID────── Schema Registry

Producer ──serialized record──► Kafka

Consumer ──resolve schema ID──► Schema Registry
Consumer ◄────writer schema──── Schema Registry

Schema lookups are cached by serializers and deserializers. A consumer does not normally make a registry request for every record.

Confluent wire format

The traditional Confluent payload-prefix format is:

[ magic byte: 0x00 ][ schema ID: 4 bytes ][ encoded payload ]

Current Confluent versions can also carry the schema identifier in a Kafka record header. The payload-prefix representation remains the default for the standard serializers.

Compatibility modes

Mode	Guarantee	Usual deployment direction
BACKWARD	New readers can read data written with the previous schema	Consumers first
FORWARD	Previous readers can read data written with the new schema	Producers first
FULL	Both backward and forward compatible	Either direction, subject to checks
NONE	No compatibility validation	Avoid in production

Compatibility is evaluated according to the schema format. Avro, Protobuf and JSON Schema do not have identical evolution rules.

Avro evolution example

Adding a field with a default allows a new Avro reader to read older data that does not contain that field:

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.investment.orders.v1",
  "fields": [
    { "name": "orderId", "type": "string" },
    { "name": "portfolioId", "type": "string" },
    { "name": "amount", "type": "double" },
    { "name": "currency", "type": "string", "default": "EUR" },
    { "name": "email", "type": ["null", "string"], "default": null }
  ]
}

The email field is genuinely nullable because its type is a union containing null, and its default matches the first union member.

A textual field rename is normally breaking. Avro aliases can support controlled migrations, but they should be tested against the actual compatibility mode, serializer and consumer implementations rather than treated as universally safe.

Producer configuration

props.put(
    ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
    KafkaAvroSerializer.class
);
props.put("schema.registry.url", "http://schema-registry:8081");

Compatibility check

curl -X POST \
  http://schema-registry:8081/compatibility/subjects/orders.placed-value/versions/latest \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  -d '{"schema":"{...escaped candidate schema...}"}'

Set compatibility for a subject:

curl -X PUT \
  http://schema-registry:8081/config/orders.placed-value \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  -d '{"compatibility":"FULL"}'

Run this compatibility check in CI before deploying a producer or consumer that introduces a new schema version.

4. What Happens Inside the Kafka Producer?

KafkaProducer.send() starts a pipeline rather than performing a synchronous network write:

send()
  │
  ▼
Serialize key and value
  │
  ▼
Choose partition
  │
  ▼
RecordAccumulator: per-partition batches
  │
  ▼
Background Sender and NetworkClient
  │
  ▼
Kafka broker

The main stages are:

1. Serialization: Key and value serializers convert application objects to bytes.
2. Partition selection: An explicit partition takes precedence. Otherwise, the producer applies key-based or sticky partitioning behaviour.
3. Accumulation: Records enter batches associated with their target partitions.
4. Drain: The background sender creates requests when batches become ready.
5. Completion: Broker responses complete futures and invoke callbacks.

Important producer settings

Setting	Kafka 4.x default	Purpose
batch.size	16,384 bytes	Upper bound for a normal record batch per partition
linger.ms	5 ms	Maximum batching delay under normal conditions
buffer.memory	33,554,432 bytes	Approximate total producer buffer budget
compression.type	none	Compression applied to complete batches
max.in.flight.requests.per.connection	5	Unacknowledged requests allowed per broker connection

send() is asynchronous, but it is not guaranteed to return immediately. It can block while waiting for metadata or buffer space, up to max.block.ms. Serialization also runs on the calling thread.

5. Tuning for Throughput and Latency

There is no universal “fastest” configuration. Measure end-to-end p95 and p99 latency, throughput, batch-size average, compression rate, retries, buffer exhaustion and broker request latency under a realistic workload.

Durable high-throughput baseline

Properties props = new Properties();

props.put(ProducerConfig.BATCH_SIZE_CONFIG, 65_536);
props.put(ProducerConfig.LINGER_MS_CONFIG, 10);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67_108_864L);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(
    ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION,
    5
);

These values are starting points, not prescriptions. Benchmark lz4 and zstd with your payload and CPU budget.

Latency-sensitive baseline

props.put(ProducerConfig.LINGER_MS_CONFIG, 0);
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16_384);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "none");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);

For financially important events, changing to acks=1 merely to reduce latency is usually an unsafe default. Benchmark acks=all with idempotence first. Any weaker durability guarantee should be an explicit business-risk decision.

Safe asynchronous callback

producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        // metadata may be null when the send fails
        log.error("Failed to publish record", exception);
        alertOrRouteForRecovery(record, exception);
        return;
    }

    log.debug(
        "Published to {}-{} at offset {}",
        metadata.topic(),
        metadata.partition(),
        metadata.offset()
    );
});

Keep callbacks lightweight because they execute on the producer’s I/O thread.

6. Why Is Kafka Fast Even Though It Uses Disk?

Kafka’s throughput comes from several mechanisms working together.

Sequential log access

Kafka writes partition logs as append-oriented files. Sequential access minimizes random seeks and works well with filesystem prefetching and modern storage devices.

This is efficient, but “disk is as fast as RAM” is too absolute. Real performance depends on the storage medium, filesystem, page-cache hit rate, flush policy and workload.

Operating-system page cache

Kafka relies heavily on the operating system’s page cache rather than maintaining a separate application-level cache of log data. Recently written or read pages can be served from memory, while the OS manages eviction and writeback.

Zero-copy transfer

Where the operating system and connection path allow it, Kafka can use sendfile-style transfer to avoid copying record bytes through the JVM heap.

Conventional user-space path:

Storage → kernel page cache → application buffer → socket → NIC

Zero-copy path:

Storage → kernel page cache ──────────────────────────→ NIC
                         transfer stays in kernel path

The exact number of physical copies depends on the operating system, TLS configuration and hardware capabilities. Zero-copy is best understood as reducing application-level copying and CPU overhead.

Batching

Producers send multiple records in a single batch, and consumers fetch records in batches. This amortizes network round trips, checksums, compression and protocol overhead across many records.

Partition-level parallelism

Partitions distribute storage, replication, production and consumption across brokers and consumers. Effective parallelism is still limited by key distribution: a hot key can create a hot partition even when the topic has many partitions.

Final Mental Model

The major Kafka concepts fit together as one continuous lifecycle:

Producer
  └─ serializes, partitions and batches records
         │
         ▼
Kafka partition log
  └─ stores ordered records efficiently
         │
         ▼
Consumer group
  └─ assigns each partition to one active member
         │
         ▼
Offset commit
  └─ records the next restart position

Schema Registry governs the contract across the full flow.

The most useful production lessons are these:

• Partition assignment enables parallelism but does not eliminate duplicate processing.
• An offset is a restart position, not proof that every external side effect succeeded.
• Compatibility checks should be automated before deployment.
• Producer tuning always trades memory, batching, latency and durability.
• Kafka is fast because sequential access, page cache, efficient transfer and batching reinforce one another.

References

• Apache Kafka 4.1 Producer Configurations
• Apache Kafka 4.1 Design
• Apache Kafka KafkaProducer API
• Confluent Schema Registry SerDes
• Confluent Avro Serializer and Deserializer

---

If you found this useful, follow Systems in Practice for more articles on Kafka, Java, distributed systems and reliable backend architecture.