Smoke testing Kafka in HDP

Assuming that you have a vanilla HDP, or the HDP sandbox, or have installed a cluster with Ambari and added Kafka, then the following may help you to smoke test the behaviour of Kafka. Obviously if you’ve configured Kafka or Zookeeper to be running on different ports, this isn’t going to help you much, and it also assumes that you are testing on one of the cluster boxes, and a ton of other assumptions.

The following assumes that you have found and changed to the Kafka installation directory – for default Ambari or HDP installations, this is probably under /usr/hdp, but your mileage may vary. To begin with, you might need to pre-create a testing topic:

bin/kafka-topics.sh
    --zookeeper localhost:2181 \
    --create --replication-factor 1 \
    --partitions 1 \
     --topic test

then in one terminal window, run a simple consumer:

bin/kafka-console-consumer.sh \
     --zookeeper localhost:2181 \
     --topic test \
     --from-beginning

Note that this is reading from the beginning of the topic, if you want to just tail the recent entries, omit the --from-beginning instruction. Finally, in another terminal window, open a dummy producer:

bin/kafka-console-producer.sh \
    --broker-list localhost:6667 \
    --topic test

There is an annoying asymmetry here – the consumer and most other utilities look to ZooKeeper to find the brokers, but the dummy producer requires an explicit pointer to one or more of the brokers. On this consumer window, type stuff, and you should see it echoed realtime in the consumer window. When finished, ^C out of the producer and consumer, and consider your work done.