How to Learn Kafka (Without Mixing Up Topics, Partitions, and Consumer Groups)

You watch a twenty-minute Kafka explainer over lunch. Topics, partitions, consumer groups, offsets. It all clicks. Two weeks later a teammate asks why your consumer isn't picking up messages, and you can feel the words sliding around. You know you understood this. You just can't get it back in order. I've done that more than once. Kafka isn't hard because the words are fancy. It's hard because the pieces only make sense together, and most tutorials walk you through them one at a time, then never ask you again. This is the version that finally stuck for me, and how I keep it from leaking out between projects. A topic is a named stream, not a queue A topic is just a name for a stream of records. Producers write to it. Consumers read from it. If you come from a job-queue background (SQS, or a RabbitMQ work queue), the surprise is that reading a message does not remove it. Other consumers can read the same record. The data hangs around based on retention, not based on who already saw it. That one fact explains a lot of "why did two services both get this event?" conversations. In Kafka, that's often the point. Partitions are how Kafka scales a topic A topic is split into partitions. Each partition is an ordered log. Records in a partition have offsets: 0, 1, 2, and so on. Across partitions there is no single global order. If you produce with a key, say user_id=42, Kafka hashes the key and always sends that user's records to the same partition. That's why order is guaranteed per key, not for the whole topic. If you produce with no key, records spread across partitions and you should not expect them back in the order you sent them. Three partitions is not three topics. It's one stream cut into three ordered lanes so more brokers and more consumers can share the work. Consumer groups decide who does the work This is the part I used to mix up with partitions. A consumer group is a team of consumers that share a group id. Kafka assigns each partition to only one member of that group. If you have six partitions and three consumers in the same group, each consumer typically gets two partitions. If you add a fourth consumer, partitions get reassigned. Two different group ids reading the same topic each get their own copy of the stream. That's how you have a billing service and an analytics service both following the same orders topic without stealing work from each other. Inside one group, members split partitions. Across groups, everyone gets the full stream. The group also tracks the last offset it committed. After a restart, members pick up from that offset instead of from the beginning (unless you configured them to). When a consumer "falls behind," it means its committed offset is lagging the end of the partition. The picture I keep in my head One topic named orders. Three partitions. Two services: billing (group billing) and analytics (group analytics). Billing runs two instances, so those two instances split the three partitions. Analytics runs one instance, so that one instance reads all three. A new order shows up in partition 2. Both groups see it. Only one billing instance sees it. If I can redraw that on a napkin without looking it up, I actually know it. Why the explainer fades by next sprint You learned Kafka the usual way: a blog post, a Confluent talk, maybe a local Docker Compose file you tore down the same afternoon. The model was clear for a day. Then you went back to application code, and nothing asked you about consumer groups until production did. Re-watching the same video feels like studying and isn't. You recognize the diagrams. Recognition is not the same as being able to explain why two consumers in one group won't both process the same partition. What helps is writing the napkin picture in your own words, then being asked about it a few days later, before you need it in an incident. I started saving the talks and docs that actually explained a piece into Recl. It takes a video or an article, gives you a short summary and key takeaways, and files the save under a topic. I add my own note on top: "same group = split partitions. different groups = each gets the whole topic. order is per partition, and per key if you set one." That note is what I want next time someone says "we're seeing duplicate processing." Often two instances accidentally used different group ids, or someone republished after a reset. Let a question come back later A few days later Recl nudges you: "Still remember this one?" The question is usually something small, like what a consumer group actually shares. If I answer it, it comes back in about a week. If I blank, I read my note and see it sooner. Index cards work too. I just got tired of maintaining a separate deck for every backend topic I keep relearning. It's easier when the prompt comes from the talk I already saved. A loop you can run this week Draw one topic, three partitions, and two consumer groups. Put two consumers in the first group and one in the second. Label who reads which partition. Add a keyed record (user_id=42) and say which partition it can land on next time (the same one). Write, in one sentence, what a committed offset is for. In a few days, before you open a diagram, try to redraw the picture. Check yourself. You don't need to memorize the entire Kafka protocol. You need the relationship between topics, partitions, and groups to still be there the next time a consumer "isn't getting messages," and a way to pull that picture back up on purpose. If you want to see what summarized takeaways look like on real material, the public Recl library has a few hundred of them. I'm Asutosh, and I build Recl, an app for saving videos, articles and notes, getting the key ideas, and recalling them on a schedule so they actually stick.