How to Set Up Apache Kafka with Docker Compose: A Practical Example

X (Twitter) Facebook Pinterest LinkedIn

Apache Kafka is a distributed streaming platform known for its high throughput, reliability, scalability, and fault tolerance. It’s widely used for building real-time streaming data pipelines and applications. Deploying Kafka in a Dockerized environment can simplify the setup and management process, especially when leveraging Docker Compose to orchestrate multi-container applications. This article provides a step-by-step guide on deploying a basic Apache Kafka setup using Docker Compose, covering everything from configuration to execution.

Table of Contents

Understanding Apache Kafka and Docker Compose

Apache Kafka operates on a publish-subscribe model, allowing it to process and store large streams of data efficiently. It’s commonly used in scenarios requiring real-time analytics, event sourcing, and log aggregation. Docker Compose, on the other hand, is a tool for defining and running multi-container Docker applications. By using Docker Compose to deploy Kafka, you benefit from simplified configuration, deployment, and scalability.

Prerequisites

Before proceeding, ensure Docker and Docker Compose are installed on your system. You should also have a basic understanding of Docker concepts and familiarity with YAML syntax.

Step 1: Define Your Docker Compose File

Create a Project Directory: Start by creating a directory for your Kafka setup. This directory will contain your docker-compose.yml file and any additional configurations you might need.

mkdir kafka-docker-compose && cd kafka-docker-compose

Create the Docker Compose File: Inside your project directory, create a docker-compose.yml file.

touch docker-compose.yml

Configure Kafka and Zookeeper Services: Edit the docker-compose.yml file in a text editor and add the following configuration:

version: '3.8'
services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "test:1:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper

In this configuration:

Zookeeper: A centralized service for maintaining configuration information, naming, and providing distributed synchronization. Kafka requires Zookeeper for cluster management.
Kafka: The Kafka broker service, configured to connect to the Zookeeper instance. The KAFKA_ADVERTISED_HOST_NAME should match the service name defined in Docker Compose for internal communication. KAFKA_CREATE_TOPICS is used to create a test topic upon startup.
Ports: Ports 2181 for Zookeeper and 9092 for Kafka are exposed for external access.

Step 2: Launch Your Kafka Cluster

With your docker-compose.yml file ready, start your Kafka cluster by running:

docker-compose up -d

This command pulls the necessary Docker images and starts the Zookeeper and Kafka containers in detached mode.

Step 3: Verify the Kafka Setup

To ensure your Kafka cluster is up and running, you can use Kafka’s command-line tools from within the Kafka container. First, execute a shell session on the Kafka container:

docker-compose exec kafka bash

Then, within the container, you can list topics to verify the automatic creation of the “test” topic:

kafka-topics.sh --list --zookeeper zookeeper:2181

You should see the “test” topic listed, indicating that your Kafka cluster is operational.

Best Practices

Persistent Storage: For production environments, consider defining volumes for Kafka and Zookeeper data to ensure data persistence across container restarts.
Networking: Use Docker Compose networks to enhance communication security between Kafka and Zookeeper containers.
Configuration Management: For complex setups, externalize Kafka and Zookeeper configurations into separate files and mount them into your containers for better manageability.

Conclusion

Deploying Apache Kafka with Docker Compose offers a streamlined approach to setting up a Kafka cluster, making it accessible for development, testing, and production use. By following the steps outlined in this guide, you can quickly get a basic Kafka setup running, allowing you to focus on developing real-time data processing applications. Whether you’re a seasoned Kafka user or new to streaming data platforms, Docker Compose can simplify and accelerate your Kafka deployment workflow.

Author
Recent Posts

Follow me

Anastasios Antoniadis

Anastasios Antoniadis is the founder and editor-in-chief of BORDERPOLAR... He is a software engineer, blogger, and avid gamer covering tech, gaming, and coding guides for over 4 years. He is a 2014 graduate of the Department of Informatics and Telecommunications of the University of Athens, an M.Sc. holder in Computer Science, and a Ph.D. student in Program Analysis.