Kafka on local machine

This page describes the steps to run a Kafka setup on your local machine using docker

1. Copy the following docker-compose.yml .

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.5.0
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    container_name: sn-zookeeper

  kafka:
    # "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
    # An important note about accessing Kafka from clients on other machines: 
    # -----------------------------------------------------------------------
    #
    # The config used here exposes port 9092 for _external_ connections to the broker
    # i.e. those from _outside_ the docker network. This could be from the host machine
    # running docker, or maybe further afield if you've got a more complicated setup. 
    # If the latter is true, you will need to change the value 'localhost' in 
    # KAFKA_ADVERTISED_LISTENERS to one that is resolvable to the docker host from those 
    # remote clients
    #
    # For connections _internal_ to the docker network, such as from other services
    # and components, use kafka:29092.
    #
    # See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
    # "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
    #
    image: confluentinc/cp-kafka:5.5.0
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    container_name: sn-kafka
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

2. Bring up the docker containers for zookeeper and kafka

docker-compose up -d

3. Check status of the containers

docker-compose ps

4. Login to the kafka container

docker exec -it sn-kafka /bin/bash

5. Create a topic

kafka-topics --bootstrap-server localhost:9092 \
--create --topic first_topic \
--partitions 1 \
--replication-factor 1

6. List all topics

kafka-topics --bootstrap-server localhost:9092 --list

7. Produce messages

kafka-console-producer --broker-list localhost:9092 --topic first_topic

8. Consume messages

kafka-console-consumer --bootstrap-server localhost:9092 \
--topic first_topic \
--from-beginning

Reference

• https://dev.to/de_maric/how-to-get-started-with-apache-kafka-in-5-minutes-18k5
• https://github.com/codingharbour/kafka-docker-compose/tree/master/single-node-avro-kafka