Install Apache Kafka on CentOS

Let’s start with updating all the repositories and installing java;

yum update -y

wget –no-cookies –no-check-certificate –header “Cookie:oraclelicense=accept-securebackup-cookie” “http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.rpm”

yum -y localinstall jdk-8u131-linux-x64.rpm

Now, we can download Kafka tar file into local server;

wget https://downloads.apache.org/kafka/2.4.1/kafka_2.12-2.4.1.tgz

We need to extract Kafka tar file;

tar -xzf kafka_2.12-2.4.1.tgz

Change its directory;

mv kafka_2.12-2.4.1.tgz /usr/local/kafka

Now, let’s configure ZooKeeper (Broker Manager) service;

vim /etc/systemd/system/zookeeper.service

We need to add following code into zookeeper.service;

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

It’s time to configure Kafka service;

vim /etc/systemd/system/kafka.service

We need to add following code into kafka.service;

Note that YOU NEED TO SET YOUR OWN “JAVA HOME“

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment=”JAVA_HOME=/usr/java/jdk1.8.0_131/”
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

We completed the setup and configuration,

Let’s start ZooKeeper and Kafka server;

I will use “nohup” to work the process on the background

cd /usr/local/kafka

nohup bin/zookeeper-server-start.sh config/zookeeper.properties &

nohup bin/kafka-server-start.sh config/server.properties &

After starting the servers, It is time to create TOPIC;

bin/kafka-topics.sh –create –bootstrap-server localhost:9092 –replication-factor 1 –partitions 1 –topic sarper-test

Let’s list the TOPICS;

bin/kafka-topics.sh –list –bootstrap-server localhost:9092

Now, we will create TWO NEW SSH TERMINAL,

First terminal will be Producer,

Second terminal will be Consumer,

In First terminal, we will execute following command;

bin/kafka-console-producer.sh –broker-list localhost:9092 –topic sarper-test

In Second terminal, we will execute following command;

bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic sarper-test –from-beginning

Display the Messages at Producer-Consumer site

Manually Delete Apache Kafka TOPICS

Firstly, list the topics

bin/kafka-topics.sh –list –bootstrap-server localhost:9092

Now, we will add following code into /usr/local/kafka/config/server.properties;

delete.topic.enable=true

Now we can delete topic with the following command;

bin/kafka-topics.sh -zookeeper localhost:2181 -delete -topic sarper-test