91 Blog Posts To Learn About Kafka

16 Aug 2026

Kafka is a distributed streaming platform enabling applications to publish, subscribe to, store, and process stream records in real-time. It matters as a highly scalable and fault-tolerant solution for building real-time data pipelines and streaming applications.

1. Setting up Kafka on Docker for Local Development

In a world where data is king, Kafka is a valuable tool for developers and data engineers to learn.

2. Kafka vs RabbitMQ: Find the Best Fit for Your Project

Explore the key differences, features, and use cases of Kafka and RabbitMQ in modern distributed systems and microservices.

3. Handling Custom Type Mapping in Kafka Listeners for Messages with "TypeId" Header

This article has provided a step-by-step guide to configuring and implementing the custom type mapper.

4. How to Move End-to-end Encrypted Data Through Kafka

The Confluent add-on for Ockam Orchestrator enables tamper-proof and end-to-end encrypted message streams through Confluent Cloud, with zero-code changes.

5. How to Consume Kafka Messages With NestJS

In this article I will tell you how to implement your own kafka message consumer using the NestJs framework. I’ll also tell you how to write tests for it.

6. How to Stream From a REST API Using Kafka Connect

Learn how to stream data efficiently from a REST API into a Kafka topic using Kafka Connect.

7. Real Time Data Processing: Easily Processing 10 Million Messages With Golang, Kafka and MongoDB

How fast Golang can be for processing a high number of messages coming from a Kafka topic?

8. Load Testing for High-Load Distributed Systems

Explore load testing strategies for high-load services, discussing staging, isolation, and emulation.

9. How Stream Processing Makes Your Event-Driven Architecture Better

If you’re an architect or developer looking at event-driven architectures, stream processing might be just what you need to make your app faster, more scalable, and more decoupled.

10. What The Heck is WarpStream?

Discover WarpStream, a powerful and user-friendly Kafka API-compatible data streaming platform designed to simplify your data infrastructure.

11. The System Design Cheat Sheet: Message Queues - ActiveMQ, RabbitMQ, Kafka, ZeroMQ

Message queues are a form of asynchronous service-to-service communication, including ActiveMQ, RabbitMQ, Kafka, and ZeroMQ.

12. Using Kafka & Zookeeper Offsets

Kafka version 0.9v and above provide the capability to store the topic offsets on the broker directly instead of relying on the Zookeeper.

13. How to Authenticate Kafka Using Kerberos (SASL), Spark, and Jupyter Notebook

Kafka & Spark integration may be tricky when Kafka is protected by Kerberos. Here is the guide on how to access Kafka with Spark and Spark Streaming.

14. How to Execute a Scheduled Task in Keycloak on Startup

In this article, we will look at how to execute a scheduled task in Keycloak on startup using a Kafka consumer as an example.

15. Best Practices For Apache Kafka Configuration

Having worked with Kafka for more than two years now, there are two configs whose interaction I've seen be ubiquitously confused.

16. Kafka Basics and Core Concepts: Explained

In this article we will cover the core concepts of Kafka and also will touch upon a few of the advanced topics.

17. Auto-generation of Documentation for Event-driven Architecture

Auto-generation of documentation for Event-driven architecture

18. Build your Own Live Chart With Deno, WebSockets, Chart.js and Materialize

We will build a simple dashboard app that displays data from a Deno Web Socket server.

19. Real-time Analytics and Data Processing with Kafka & Spark

Real-time analytic systems use data processing frameworks, including Apache Kafka and Apache Spark. Learn more here!

20. Apache Kafka: the Complicated Simplicity

This post is about issues, misunderstandings and sometimes heroic solutions from our experience of using Kafka as the main data exchange platform.

21. Stream and Display Data in Realtime with Materialize and Adonis

In this tutorial, we are going to build a web application using AdonisJS and integrate it with Materialize to create a real-time dashboard

22. I Just Want to See It Work: Kafka Stream, C# Producer and Consumer

A quick start for running Kafka in docker container, and a working example of a producer and a consumer in C# .NET

23. Deploying Apache Kafka With Kubernetes

Deploying Kafka on Kubernetes is a low-effort approach to setting up an event-driven architecture to support your API ecosystem in the cloud.

24. The Streaming Bridges - A Kafka, RabbitMQ, MQTT and CoAP Example to Learn More

How can we use data stream and what does that mean? Does Kotlin really offers an alternative?

25. 10 Days of .Net Aspire: Day 5 — Integrating Apache Kafka

Learn how to integrate the Apache Kafka component into your .Net Aspire project using Visual Studio 2022.

26. Kafka Connect Framework: Creating a Real-Time Data Pipeline Using CDC

Microservices, Machine Learning & Big Data are making waves among organizations. Curiously they all share the same biggest concern: data.

27. Jaeger Persistent Storage with Elasticsearch, Cassandra and Kafka

Running systems in production involve requirements for high availability, resilience and recovery from failure. When running cloud-native applications this becomes even more critical, as the base assumption in such environments is that compute nodes will suffer outages, Kubernetes nodes will go down and microservices instances are likely to fail, yet the service is expected to remain up and running.

28. The Only Decoupling Checklist You Need To Know About

My team has recently successfully decoupled one of the critical business domains of the company. The initial integration had such a tough deadline that the only way to meet it was to add code to the monolith. And… The feature that went from conception to production in three weeks ended up taking almost one year to decouple.

29. A Modern Stack to Build a Real-time Event-driven App

The rise of real-time event-driven applications has led to the development of modern technology stacks that can handle large volumes of events in real time.

30. Apache Kafka Architecture 101: Internal Components and How They Work Together

In this article, you'll better understand all the components associated with Apache Kafka and how they work together.

31. Kafka Storage Design - Making File Systems Cool Again!

What makes Kafka so Fast? A Deep Dive into Kafka Storage Internals.

32. Lambda Architecture Speed Layer: Real-Time Visualization For Taxi [Part 1]

Lambda architecture has 3 components, a) Speed layer, which is the streaming data layer or real time data layer, b) serving layer, which is the database layer, which is derived by aggregating data from speed layer, and c) batch layer, which is the set of computations which are perfomed on large sets of data, typically stored in a distributed file system. In this post i will be talking about how to implement the speed layer, by visualizing real time taxi data. Post that, the visualization will allow us to make some real time business decisions. Code for this article can be found here.

33. Build a Live Dashboard with Materialize, Airbyte, MySQL and Redpanda/Kafka

34. Record-Replay Strategy for Testing Event-Driven Architecture

Testing event-driven architectures like kafka based systems using record-replay strategy.

35. How to Use Materialize and Redpanda to Analyze Raspberry Pi Temperature Data

This is a self-contained demo using Materialize to process data IoT devices data directly from a PostgreSQL server.

36. Top Apache Kafka® Interview Questions for Juniors In 2023

Prep for an Apache Kafka interview by reading this questions! Aimed at juniors.

37. 5 Problems and Their Solutions With Creating a High-Load Service Using .NET and Kafka

Specifics and complications of creating a high-load service using .NET and Kafka.

38. Introduction to Event Streaming with Kafka and Kafdrop

Event sourcing, eventual consistency, microservices, CQRS... These are quickly becoming household names in mainstream application development. But do you know what makes them tick? What are the basic building blocks required to assemble complex, business-centric applications from fine-grained services without turning the lot into a big ball of mud?

39. Kafka Message Testing: How to Write Integration Tests

This article offers an approach to writing integration tests for Kafka-based applications that focuses on interaction specification, making tests more readable

40. Lambda Architecture Serving Layer: Real-Time Visualization For Taxi [Part 2]

In this part i would be talking about the serving layer of the Lambda Architecture. Serving layer is derived either by performing computation on batch data to arrive at a view that is mid way from speed layer and batch layer

41. Take your Materialized Views to the Next Level by Joining MySQL and Postgres

42. Node.js Tutorial: How to Build a Simple Event-Driven Application With Kafka

Build a real-time event-driven app with Node.js and Kafka on Heroku. Follow this step-by-step guide to set up, deploy, and manage your application efficiently.

43. Boosting Integration Test Performance with EmbeddedKafka and GraalVM

In this article, I want to share my experience with creating a native image for EmbeddedKafka using GraalVM.

44. Key Apache KafkaⓇ Concepts Every Dev Needs to Know

In my first few months learning Apache Kafka, I drew up a blog post on the fundamental concepts behind implementing it.

45. I Built 10 Pokemon Agents That Learn From Each Other Using Kafka and Flink

The agent runs headless inside a stereOS VM with no display server. It reads game memory directly from known RAM addresses.

46. Comparing Apache Kafka with Oracle Transactional Event Queues (TEQ) as Microservices Event Mesh

This blog contrasts and compares transactional and message delivery behavior of Kafka with the converged Oracle DB and Oracle Transactional Event Queues/AQ

47. Resolving Complex JSON in Kafka Source Using Apache SeaTunnel

Use the extension of UDF functions to parse the nested JSON data from Kafka sources.

48. Choosing Between Enterprise Messaging and Event Streaming For Your Architecture

Comparing Enterprise messaging and event streaming across different dimensions to see how they excel at solving different but related messaging problems

49. RocksDB Is Steadily Eating the Database World

Technical design. Because one of the most common use cases of the new databases is storing data that is generated by high-throughput sources, it is important that the store engine is able to handle write-intensive workloads, all while offering acceptable read performance. RocksDB implements what is known in the database literature as a log-structured merge tree aka LSM tree.

50. Kafka Schema Evolution: A Guide to the Confluent Schema Registry

Learn Kafka Schema Evolution: Understand, Manage & Scale Data Streams with Confluent Schema Registry. Essential for Data Engineers & Architects.

51. Building Production-Grade RAG Systems for Document AI: What It Actually Takes

Learn to build production-grade RAG for enterprise PDFs. Focus on robust ingestion, hybrid retrieval, security, and traceability to move beyond demos.

52. How to Easily Stream Data From a Headless BI and SQL Engine

Now, you can use Cube to build data modeling, caching, and access control layers on top of streaming SQL, just as with cloud data warehouses.

53. How the ZooKeeper Solves the Dining Philosophers Problem

In this blog by Paul Brebner, Instaclustr's tech evangelist explains the Apache ZooKeeper using the famous dining philosophers problem.

54. Transitive Compatibility Types in Schema Registry for Apache Kafka

Apache Kafka provides Schema Registry capabilities. There are three types of transitive compatibility.

55. Data Pipelines and Expiring Dictionaries

Designing a data pipeline comes with its own set of problems. Take lambda architecture for example. In the batch layer, if data somewhere in the past is incorrect, you’d have to run the computation function on the whole (possibly terabytes large) dataset, the result of which would be absorbed in serving layer and are reflected.

56. Minimizing Rebalance Downtime: Optimizing Stateless Kafka Streams Apps (x450)

With a single config change, reduce rebalance downtime for your stateless Kafka Streams Apps from 45s to <100ms!!

57. Apache Kafka’s New Tiered Storage: What Developers Need to Know

Learn how open source Apache Kafka's Tiered Storage transforms data retention economics, enabling affordable long-term storage while maintaining performance.

58. Top 5 Cloud Skills That Will Matter The Most in 2026

The cloud skills of the future are not about knowing one cloud provider better than another.

59. Apache Kafka: How Does It Work?

This article was originally posted to the Confluent blog.

60. What are Pub/Sub Systems

Pub/sub, short for publish/subscribe, are systems that allow communication between different components or services in a distributed architecture.

61. Reliable Testing with Kafka: Isolation Techniques

Kafka integration testing with Testcontainers and Spring Boot 3.1, ensuring test isolation and full message access for reliable and efficient tests.

62. Kafka Administration and Monitoring UI Tools

Kafka itself comes with command line tools that can do all the administration tasks, but those tools aren’t very convenient because they are not integrated into one tool and you need to run a different tool for different tasks. Moreover, it is getting difficult to work with them when your clusters grow large or when you have several clusters.

63. 5 Things Every Apache Kafka Dev Needs To Know: A Performance and Architectural Deep Dive

Here are five tips on how Kafka works and how you can get started with Apache Kafka.

64. Lambda Architecture Batch Layer: Visualizing All Time Taxi Data [Part 3]

In this part i would be talking about the batch layer of the Lambda Architecture. Batch layer is computed by applying a function to the whole historical dataset, to answer some high level questions which cannot be answered by either speed layer or serving layer. The computations typically take hours or days to run, and the results are stored usually in a distributed file system (although this is not a requirement). For example, the queries that might need to be answered would range from the beginning of the dataset to now, in our case, till date how many cabs have served how many passengers, or what is the total distance driven by all the cabs. In this article i would try to answer questions like these based on the dataset that i have. The code for the article can be found here.

65. Building a Distributed Timer Service at Scale: Handling 100K Timers Per Second

This article details a horizontally scalable, distributed timer service achieving 100 K timer creations per second with millisecond precision.

66. Debezium Introduction: Another Change Data Capture Tool

Building an enterprise data warehouse can be either relatively straightforward or very sophisticated. It depends on many factors, such as the conceptual data model complexity and the variety of source systems. In many cases, applying the Change Data Capture (CDC) approach can make the data integration simpler. Fortunately, there are plenty of CDC tools available in the market, many of which are easy-to-use and affordable, while others are cumbersome and expensive (for what it is).

67. Deploy Your Application with Kafka and Docker in 20 Minutes

In this tutorial, we'll walk you through how to use Docker, Kafka, and Kubernetes to deploy a simple application.

68. Understanding Kafka Partitions: How to Get the Most out of Your Kafka Cluster

This blog provides an overview around the two fundamental concepts in Apache Kafka: Topics and Partitions. While developing and scaling our Anomalia Machina application we have discovered that distributed applications using Kafka and Cassandra clusters require careful tuning to achieve close to linear scalability, and critical variables included the number of Kafka topics and partitions. In this blog, we test that theory and answer questions like “What impact does increasing partitions have on throughput?” and “Is there an optimal number of partitions for a cluster to maximize write throughput?” And more!

69. Why Regulated Enterprises Can’t Just “Use Kafka” — and What to Run Instead

How to modernize messaging in regulated industries: Kubernetes-native platforms for air-gapped, zero-egress environments with full security control.

70. Understanding Kafka with Factorio

[https://hackernoon.com/photos/zXE6n93W9qXDNsP8hiip4vQhbqg2-pi1dk2f60] This is a 1.0 story that I edited

Thanks to Tom de Ruijter, [https://medium.co

71. Kafka Gotchas

I’ve assisted several large clients in building a microservices-style architecture using Kafka as a messaging backbone, having a reasonably good understanding of its abilities and the use cases that really bring them out. But I’m not a Kafka apologist by any stretch; any technology that has gone through such a rapid adoption curve is bound to polarise its audience and rub certain developers up a wrong way, and Kafka is no exception. Like anything else, you need to invest a significant amount of time in getting across Kafka and event streaming in general, before you become fully proficient and can harness its might. And be prepared to face one or two frustrations, to put it mildly, along the way.

72. Turbocharging AI Sentiment Analysis: How We Hit 50K RPS with GPU Micro-services

The sentiment analysis stack was one big codebase for data ingestion, model inference, logging, and storage. It worked great, until traffic shot up.

73. A Brief Introduction to Commit Logs

Logs are everywhere in software development. Without them there’d be no relational databases, git version control, or most analytics platforms.

74. For Entry-Level Data Engineers: How to Build a Simple but Solid Data Architecture

This article aims to provide a reference for non-tech companies who are seeking to empower their business with data analytics.

75. Using KSQL Stream Processing & Real-Time Databases to Analyze Kafka Streaming Data [A How-To Guide]

Intro

76. Architecting Resilient and Scalable Systems with Java, Kafka and AWS: A Case Study Approach

Architect resilient, scalable systems using Java, Kafka and AWS - microservices, event-driven design and cloud scaling.

77. When ML Meets Microservices: Engineering for Scalability and Performance

Discover how combining machine learning with microservices architecture enables scalable, high-performance systems by leveraging modular design, efficient data

78. Data Speedways: How Kafka Races Ahead in System Design

Unlock the Power of Real-Time Data with Kafka: A Deep Dive into the Fast and Scalable System Design Championed by Kafka. Learn More!

79. Run a Flink Kafka Job from DolphinScheduler (Standalone VM Setup)

Configure Kafka + Flink networking, package a Kafka consumer JAR, upload it, and run it as a scheduled Flink task node end-to-end.

80. The Noonification: Everything Missing in Cosmos DeFi (12/28/2023)

12/28/2023: Top 5 stories on the HackerNoon homepage!

81. Real-Time Data Processing with Kafka Streams: Simplifying Stream Processing Applications

Explore Kafka Streams: a Java library for building scalable, fault-tolerant stream processing apps. Learn how to simplify real-time data processing.

82. Why Your Debezium CDC Stream Is Full of __debezium_unavailable_value (and How to Fix It)

If you see __debezium_unavailable_value in your Debezium CDC stream, it’s not a bug; it’s PostgreSQL and Debezium being honest about what they don’t know.

83. Using dbt with Materialize and Redpanda

84. How to Stream XML messages from IBM MQ into Kafka into MongoDB

Let’s imagine we have XML data on a queue in IBM MQ, and we want to ingest it into Kafka to then use downstream, perhaps in an application or maybe to stream to a NoSQL store like MongoDB.

85. Unveiling Kafka-Streams-Cassandra-State-Store

'Drop-in' Kafka Streams State Store implementation that persists data to Apache Cassandra / ScyllaDB

86. Tips About Kafka Connect On Heroku You Can't Afford To Miss

Introduction

87. Kuma 1.0 GA Released With 70+ New Features & Improvements

Today is a big day for Kuma! Kuma 1.0 is now generally available with over 70 features and improvements ready to use and deploy in production to create modern distributed service meshes for every application running on multiple clusters, clouds, including Kubernetes and VM-based workloads.

← Previous

121 Blog Posts To Learn About K8s

91 Blog Posts To Learn About Kafka

Let's learn about Kafka via these 91 free blog posts. They are ordered by HackerNoon reader engagement data. Visit the Learn Repo or LearnRepo.com to find the most read blog posts about any technology.

Thank you for checking out the 91 most read blog posts about Kafka on HackerNoon.