Partitioning

Kafka is a distributed event streaming platform designed for building high-throughput, fault-tolerant, and scalable data streaming applications.

This article covers key designs in kafka such as how messages for a topic are shared into partitions assigned to brokers. Then we see some guarantees about producers, consumers and consumer groups.

Data partitioning refers to the process of dividing a system’s data into smaller, more manageable subsets, which are distributed across multiple storage locations or nodes.

This article covers some strategies for partitioning including random partitioning, by hash key, by range and a hybrid approach for skewed workloads. We also see strategies to rebalance partitions if there's a static or dynamic number of partitions.