Kafka
Most important commands to remember
kcat -L— inspect topic and partition metadata.kcat -C— consume records from the selected partition.
Commands and flags
| Command or option | Meaning |
|---|---|
timeout 10 |
Stop an invocation after ten seconds if it has not finished. |
-b localhost:9092 |
Use this broker as a bootstrap address. |
-L -t lab-events |
Request metadata for the existing topic. |
-C -p 0 |
Consume directly from partition 0. |
-o beginning |
Start at the earliest currently available offset. |
-c 2 -e |
Stop after two records or on reaching the partition end. |
-f '…' |
Format records: %p partition, %o offset, %s value, \n newline. |
The quoted format belongs to kcat. This direct partition example does not join a consumer group or advance a group’s committed offsets.
The concepts that matter
1. A topic is divided into ordered partitions
Kafka stores records in topic partitions. Each partition is an ordered log, and each record has an offset identifying its position within that partition.
Offsets are not global message numbers across a topic. Records in different partitions have no single built-in total order. Partitioning enables parallel work, but choosing keys and partition strategy also determines which related records can be processed in order.
2. Consumption does not delete the record
A consumer reads the log at a chosen position. The broker retains records according to policies such as time, size, or compaction rather than deleting each record when one consumer reads it.
This lets another consumer read the same data and lets a consumer replay available history. Retention is still finite, and compaction can remove older values for a key. The earliest available offset need not be zero, and offsets can have gaps.
3. Consumer groups divide processing responsibility
Within a consumer group, partitions are assigned among its active consumers. A partition is assigned to one group member at a time under the usual group model. More consumers than partitions therefore do not automatically increase useful parallelism.
Committed offsets record the group’s progress for later recovery. They are different from a record’s permanent position. Commit timing matters: committing before work completes risks skipped work, while completing work before committing can cause work to repeat after failure.
4. Replication and processing correctness are separate
Partition replicas help Kafka tolerate broker failures under the configured replication and acknowledgment policies. They do not make every external database write or API call exactly once.
Idempotent production and transactions provide specific guarantees within their defined boundaries. A consumer that calls an external service still needs a plan for retries, duplicates, and coordination. A successful read demonstrates availability of that record, not the durability or correctness of the complete pipeline.
One small example
Optional: use the existing test topic. Run each command separately. The ten-second bound prevents an unavailable broker from leaving the inspection running indefinitely.
timeout 10 kcat -b localhost:9092 -L -t lab-events
timeout 10 kcat -b localhost:9092 -C -t lab-events -p 0 -o beginning -c 2 -e -f 'partition=%p offset=%o value=%s\n'
timeout 10 kcat -b localhost:9092 -C -t lab-events -p 0 -o beginning -c 2 -e -f 'partition=%p offset=%o value=%s\n'The metadata should identify partition 0, its leader, and replica information. The two consumption commands deliberately use the same starting position. If retention and compaction do not change the data between runs, compare the repeated partition/offset pairs and values.
Repeated records show that reading did not consume them destructively. Fewer than two records can mean the partition end was reached; timeout exit status 124 means the wrapper stopped the command. This test neither changes group progress nor proves group rebalancing, replication recovery, or end-to-end exactly-once effects.
Keep this idea: Kafka keeps a partitioned log; consumers track their position in it instead of removing records as they read.