Kafka example - a simple producer and consumer for kafka using spring boot + java

Overview

Kafka example Kafka Profile last updated

Functionalities to be added

  • MongoDB integration
  • Unit tests
  • Integration Tests
  • CI/CD with Git actions
  • Postman Documentation

To run this project you will need:

  • Java11
  • ApacheMaven
  • Docker
  • Docker-Compose

Project structure

Project
├── kafka-producer-example
|   ├── Dockerfile
|   ├── src
|      ├── main
|         ├── java/com/project/kafkaexample
|            ├── service: contains all of service implementation and business rules
|            ├── dto: contains data transfer objects used in the integrations
|            ├── api: contains controller and apis interfaces
|            ├── config: contains all spring bean configuration
|            ├── domain: contains all domain entity objects/classes
             ├── exception: contains all exception handlers
|         ├── resource: contains property files and static content
├── kafka-consumer-example
|   ├── Dockerfile
|   ├── src
|      ├── main
|         ├── java/com/project/kafkaexample
|            ├── service: contains all of service implementation and business rules
|            ├── dto: contains data transfer objects used in the integrations
|            ├── api: contains controller and apis interfaces
|            ├── respository: contains all interfaces with data-bases
|            ├── config: contains all spring bean configuration
|            ├── domain: contains all domain entity objects/classes
|            ├── exception: contains all exception handlers
|         ├── resource: contains property files and static content
├── docker-compose.yml

How to build and run:

After downloading the repository, to run from scratch you must following the steps bellow:

If you want to run it from scratch: 🔰

  • ▶️ go to ./kafka-consumer-example and run mvn clean package -DskipTests;
  • Come back to root dir;
  • ▶️ go to ./kafka-producer-example and run mvn clean package -DskipTests;
  • Come back to root dir (again);
  • ▶️ run docker-compose up build -d or simply docker-compose up -d
  • wait until all containers are up and runing

Testing:

Now it's time to use these examples by calling an api using curl or any other tools you prefer: 👽

curl -X POST 'http://localhost:8080/api/send' -d '{"name":"Full Name", "email": "[email protected]"}' -H 'content-type: application/json'

Validating:

To check if everything was ok, go to terminal and type following commands:

  • docker logs kafka-producer to show kafka producer logs (here you should see a log saying you've sent a message) 🟢
  • docker logs kafka-consumer to show kafka consumer logs (here you should see a log saying you've receved a new message) 🟢
  • Now it's possible to query through API the sample message sent from producer to customer:
    • query all people in data-base:
    curl 'http://localhost:8081/api/people?number=0&size=10'
    • query specific person using id: (remember replacing {id} for a valid id returned in the first query)
    curl 'http://localhost:8081/api/people/{id}'

Contact

Gmail Badge Linkedin Badge Twitter Badge

You might also like...

Mirror of Apache Kafka

Apache Kafka See our web site for details on the project. You need to have Java installed. We build and test Apache Kafka with Java 8, 11 and 15. We s

Jan 5, 2023

Kryptonite is a turn-key ready transformation (SMT) for Apache Kafka® Connect to do field-level 🔒 encryption/decryption 🔓 of records. It's an UNOFFICIAL community project.

Kryptonite - An SMT for Kafka Connect Kryptonite is a turn-key ready transformation (SMT) for Apache Kafka® to do field-level encryption/decryption of

Jan 3, 2023

A command line client for Kafka Connect

kcctl -- A CLI for Apache Kafka Connect This project is a command-line client for Kafka Connect. Relying on the idioms and semantics of kubectl, it al

Dec 19, 2022

A command line client for Kafka Connect

A command line client for Kafka Connect

🧸 kcctl – Your Cuddly CLI for Apache Kafka Connect This project is a command-line client for Kafka Connect. Relying on the idioms and semantics of ku

Dec 19, 2022

Publish Kafka messages from HTTP

Kafka Bridge Publish Kafka messages from HTTP Configuration Example configuration for commonly used user + password authentication: kafka-bridge: ka

Nov 9, 2021

Implementação de teste com Kafka

Implementação de teste com Kafka

TesteKafka01 Implementação de teste com Kafka Projeto criado para estudo e testes com Kafka Recursos que estarão disponiveis: -Envio de msg -Recebe Ms

Sep 17, 2021

MemQ is a new PubSub system that augments Kafka

MemQ is a new PubSub system that augments Kafka

MemQ: An efficient, scalable cloud native PubSub system MemQ is a new PubSub system that augments Kafka at Pinterest. It uses a decoupled storage and

Dec 30, 2022

Aula sobre segurança no kafka usando SSL

Aula sobre segurança no kafka usando SSL

Kafka4Devs - Segurança no Kafka com SSL Você sabe o que acontece por debaixo dos panos de uma aplicação segura? Sabe como empresas grandes que utiliza

Feb 28, 2022

Microservice-based online payment system for customers and merchants using RESTful APIs and message queues

Microservice-based online payment system for customers and merchants using RESTful APIs and message queues

Mar 23, 2022
Comments
  • Feature/integrating mongodb

    Feature/integrating mongodb

    • Added mongoDB integration
    • Created 2 api resource in kafka-consumer-example
    • Changed default serializer from kafka-consumer-example
    • Removed default header type from kafka-producer-example
    • Updated README.md
    opened by arturcampos 0
Owner
arturcampos
Let's code a better world together!
arturcampos
A template and introduction for the first kafka stream application. The readme file contains all the required commands to run the Kafka cluster from Scrach

Kafka Streams Template Maven Project This project will be used to create the followings: A Kafka Producer Application that will start producing random

null 2 Jan 10, 2022
Demo project for Kafka Ignite streamer, Kafka as source and Ignite cache as sink

ignite-kafka-streamer **Description : Demo project for Kafka Ignite streamer, Kafka as source and Ignite cache as sink Step-1) Run both Zookeeper and

null 1 Feb 1, 2022
KC4Streams - a simple Java library that provides utility classes and standard implementations for most of the Kafka Streams pluggable interfaces

KC4Streams (which stands for Kafka Commons for Streams) is a simple Java library that provides utility classes and standard implementations for most of the Kafka Streams pluggable interfaces.

StreamThoughts 2 Mar 2, 2022
Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

Firehose - Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

Open DataOps Foundation 279 Dec 22, 2022
This repository contains a functional example of an order delivery service similar to UberEats, DoorDash, and Instacart.

Order Delivery Microservice Example In an event-driven microservices architecture, the concept of a domain event is central to the behavior of each se

Kenny Bastani 198 Dec 7, 2022
An example Twitch.tv bot that allows you to manage channel rewards (without requiring a message), and chat messages.

Twitch Bot Example shit code that can be used as a template for a twitch bot that takes advantage of channel rewards (that dont require text input) an

Evan 3 Nov 3, 2022
Fast and reliable message broker built on top of Kafka.

Hermes Hermes is an asynchronous message broker built on top of Kafka. We provide reliable, fault tolerant REST interface for message publishing and a

Allegro Tech 742 Jan 3, 2023
Dataflow template which read data from Kafka (Support SSL), transform, and outputs the resulting records to BigQuery

Kafka to BigQuery Dataflow Template The pipeline template read data from Kafka (Support SSL), transform the data and outputs the resulting records to

DoiT International 12 Jun 1, 2021
Output Keycloak Events and Admin Events to a Kafka topic.

keycloak-kafka-eventlistener Output Keycloak Events and Admin Events to a Kafka topic. Based on Keycloak 15.0.2+ / RH-SSO 7.5.0+ How to use the plugin

Dwayne Du 4 Oct 10, 2022
A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues

Nakadi Event Broker Nakadi is a distributed event bus broker that implements a RESTful API abstraction on top of Kafka-like queues, which can be used

Zalando SE 866 Dec 21, 2022