C Kafka Examples

Mixing the old with the new. Kafka examples with C using RDKafka.

MAC OS Installation

1. Install the dependent library for Kafka:

brew install librdkafka

This will install into the /usr/local/include and /usr/local/lib which we will include in CMake to find the header files.

Check the headers are installed

pkg-config --cflags rdkafka

Check the libs are intalled

pkg-config --libs rdkafka

Option 1 CMAKE Build

mkdir build && cd build   
cmake ..
make

 ./MainProject/src/MainProject localhost:9091 my-lowercase-consumer lowercase-topic

Option 2 Command Build

Lib File Mac Ref Header File Library

# Create object files for the Library files
gcc -Wall -g -c src/consumer.c -o consumer.o -I include/
gcc -Wall -g -c src/UppercaseTopology.c -o UppercaseTopology.o -I include/
# Todo UNIX?
ar ruv mylib.a consumer.o UppercaseTopology.o
# Create Mac Shared Lib
gcc -dynamiclib -undefined suppress -flat_namespace consumer.o UppercaseTopology.o -o libmyProject.dylib

# Need to be in the directory where the .dylib file is 
gcc -Wall -std=c11 -L/Users/Stuart/Documents/Programming/C_Programming/c-kafka-examples/LibProject/ -lmyProject -lrdkafka MainProject/src/main.c -o myconsumer -I LibProject/include

Note we found the rdkafka library and linked as its on the usual path we can search for it using

pkg-config --libs rdkafka

3. Start up Kafka.

docker exec kafka-1 kafka-topics --create --zookeeper zookeeper-1:22181 --replication-factor 1 --partitions 1 --topic lowercase-topic

docker exec kafka-1 kafka-topics --create --zookeeper zookeeper-1:22181 --replication-factor 1 --partitions 1 --topic uppercase-topic

docker exec kafka-1 kafka-topics --zookeeper zookeeper-1:22181 --list

./myconsumer localhost:9091 my-lowercase-consumer lowercase-topic

docker exec -it kafka-1 kafka-console-producer --broker-list kafka-1:29091 --topic lowercase-topic --property "parse.key=true" --property "key.separator=:"

docker exec kafka-1 kafka-console-consumer --bootstrap-server kafka-1:29091 --topic uppercase-topic --property print.key=true --property key.separator="-" --from-beginning

Useful Docker

Stop and remove all running containers.

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

TODO

• Avro example