Publish Kafka messages from HTTP

Related tags

Overview

Kafka Bridge

Publish Kafka messages from HTTP

Configuration

Example configuration for commonly used user + password authentication:

kafka-bridge:
  kafka:
    "[bootstrap.servers]": <kafka-host>:9092
    "[security.protocol]": SASL_SSL
    "[sasl.mechanism]": PLAIN
    "[sasl.jaas.config]": >-
      org.apache.kafka.common.security.plain.PlainLoginModule
      required
      username="<kafka-username>"
      password="<kafka-password>";
    "[ssl.endpoint.identification.algorithm]": https
  schema-registry:
    url: https://<schema-registry-host>/
    "[basic.auth.credentials.source]": USER_INFO
    "[basic.auth.user.info]": <schema-registry-username>:<schema-registry-password>

Example configuration for setting a specific template directory:

kafka-bridge:
  template-directory: /kafka-bridge/templates

If unset, the template paths will be resolved against the working directory. This setting is ignored for absolute paths.

Run

export SPRING_CONFIG_LOCATION=file:///.../application.yaml
java -jar kafka-bridge.jar

Send Kafka message from HTTP

Example: String key + Avro value

POST /topics/Products/send
Key: Product-0001
Content-Type: application/avro+json
Schema-Subject: de.neuland.kafkabridge.Product

{
  "type": "REGULAR",
  "name": "Kafka Bridge Product",
  "available_since": 1629559054000
}

Example: Avro key + Avro value

POST /topics/Products/send
Key: { "code": "Product-0001" }
Key-Content-Type: application/avro+json
Key-Schema-Subject: de.neuland.kafkabridge.ProductKey
Content-Type: application/avro+json
Schema-Subject: de.neuland.kafkabridge.Product

{
  "type": "REGULAR",
  "name": "Kafka Bridge Product",
  "available_since": 1629559054000
}

Templating

The templating is done via Thymeleaf. In addition to what's available per default, the Java 8 Time Thymeleaf extras and two custom variables have been added. The variables are utcZoneId and systemDefaultZoneId pointing to java.time.ZoneOffset#UTC and java.time.ZoneId#systemDefault respectively.

Example: Avro key + Avro value with templating

Contents of key.json:

{
  "code": "default"
}

Contents of value.json:

{
  "type": "REGULAR",
  "available_since": [(
    ${ #temporals.createNowForTimeZone(systemDefaultZoneId).minusSeconds(5).toInstant().toEpochMilli() }
  )]
}

Request against Kafka Bridge:

POST /topics/Products/send
Key: {}
Key-Content-Type: application/avro+json
Key-Schema-Subject: de.neuland.kafkabridge.ProductKey
Key-Template-Path: key.json
Content-Type: application/avro+json
Schema-Subject: de.neuland.kafkabridge.Product
Template-Path: value.json

{
  "type": "SPECIAL",
  "name": "Kafka Bridge Product"
}

Actual JSON key that gets converted to Avro:

{
  "code": "default"
}

Actual JSON value that gets converted to Avro:

{
  "type": "SPECIAL",
  "name": "Kafka Bridge Product",
  "available_since": 1630566414000
}

Comments

Add conversion of ISO 8601 strings to epoch milliseconds

Timestamps are usually encoded as epoch milliseconds in Avro. It should be possible to input 2021-09-23T07:02:56Z and have it converted to epoch milliseconds automatically.
enhancement

opened by michel-zimmer 0
Add testing of the Helm Chart

A draft for a GitHub Workflow is already there at /.github/workflows/helm-charts-lint-test.yaml, but it failed - probably because of helm/chart-testing-action#69.
enhancement

opened by michel-zimmer 0

A command line client for Kafka Connect

kcctl -- A CLI for Apache Kafka Connect This project is a command-line client for Kafka Connect. Relying on the idioms and semantics of kubectl, it al

Dec 19, 2022

🧸 kcctl – Your Cuddly CLI for Apache Kafka Connect This project is a command-line client for Kafka Connect. Relying on the idioms and semantics of ku

Implementação de teste com Kafka

TesteKafka01 Implementação de teste com Kafka Projeto criado para estudo e testes com Kafka Recursos que estarão disponiveis: -Envio de msg -Recebe Ms

Sep 17, 2021

Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

Firehose - Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

Dec 22, 2022

MemQ is a new PubSub system that augments Kafka

MemQ: An efficient, scalable cloud native PubSub system MemQ is a new PubSub system that augments Kafka at Pinterest. It uses a decoupled storage and

Dec 30, 2022

KC4Streams - a simple Java library that provides utility classes and standard implementations for most of the Kafka Streams pluggable interfaces

KC4Streams (which stands for Kafka Commons for Streams) is a simple Java library that provides utility classes and standard implementations for most of the Kafka Streams pluggable interfaces.

Mar 2, 2022

Output Keycloak Events and Admin Events to a Kafka topic.

keycloak-kafka-eventlistener Output Keycloak Events and Admin Events to a Kafka topic. Based on Keycloak 15.0.2+ / RH-SSO 7.5.0+ How to use the plugin

Oct 10, 2022

Aula sobre segurança no kafka usando SSL

Kafka4Devs - Segurança no Kafka com SSL Você sabe o que acontece por debaixo dos panos de uma aplicação segura? Sabe como empresas grandes que utiliza

Feb 28, 2022

Kafka is not for event sourcing, isn't it? Kafka alone is not an event store, but Kafka and ksqlDB together allow building full-featured event stores. This repository provides a sample of event sourced system that uses Kafka and ksqlDB as event store.

Event Sourcing with Kafka and ksqlDB Introduction Example Domain Event Sourcing and CQRS 101 State-Oriented Persistence Event Sourcing CQRS Advantages

Dec 28, 2022

httpx - CLI to test HTTP/gRPC/RSocket/Kafka... services by HTTP DSL

httpx: CLI for run http file httpx is a CLI to execute requests from JetBrains Http File. Request types supported by httpx HTTP REST PUB/SUB - Apache

Dec 15, 2022

Eclipse Jetty® - Web Container & Clients - supports HTTP/2, HTTP/1.1, HTTP/1.0, websocket, servlets, and more

Eclipse Jetty Canonical Repository This is the canonical repository for the Jetty project, feel free to fork and contribute now! Submitting a patch or

Introduction to CYS4-SensitiveDiscoverer, a Burp extension that discovers sensitive information inside HTTP messages.

CYS4-SensitiveDiscoverer Introduction Burp Suite is a useful tool used to do web application security testing. While Burp Suite provides a lot of func

Nov 16, 2022

Share the chat messages across Minecraft Servers via HTTP backend powered by Spring Boot, this is the backend part of the project.

InterconnectedChat-Backend Share the chat messages across Minecraft Servers via HTTP backend powered by Spring Boot, this is the backend part of the p

Oct 6, 2021

This is a simple realization of custom messages pre/post processing in spring-boot HTTP/Stream requests & responses

spring-boot-custom-message-converting-instances This is a simple realization of custom messages converting in spring-boot HTTP requests and responses.

Jul 22, 2022

Redisson - Redis Java client with features of In-Memory Data Grid. Over 50 Redis based Java objects and services: Set, Multimap, SortedSet, Map, List, Queue, Deque, Semaphore, Lock, AtomicLong, Map Reduce, Publish / Subscribe, Bloom filter, Spring Cache, Tomcat, Scheduler, JCache API, Hibernate, MyBatis, RPC, local cache ...

Jan 5, 2023

Jan 3, 2023

Publish Jenkins performances metrics to an OpenTelemetry endpoint, including distributed traces of job executions and health metrics of the controller.

OpenTelemetry Introduction Architecture Features Getting Started Examples Configuration as Code Contributing Introduction Collect Jenkins monitoring d

Dec 26, 2022

Publish commands to shopping cart Google pub/sub topic.

shopping-cart-pub-java-akka-serverless This project is an Akka Serverless service that is used to publish shopping cart commands to a Google pub/sub t

Nov 17, 2021

End to End project for Kafka Streams using Spring Cloud Kafka streams

Spring Kafka Streams using Spring Cloud Streams End to End example Endpoint http://localhost:8080/domain/lookup/facebook - to pull all facebook relate

Dec 20, 2022