Kafka helps you to build fast, high throughput, fault tolerance, scalable microservices, and applications. Kafka Streams stores data in Kafka Clusters (Kafka State Stores) and gets data wicket fast.

Technologies Used

1. Kafka Streams (Confluent)
2. Materialized views and Kafka State Stores
3. REST API using Jetty Server
4. Confluent Schema Registry
5. Avro Serializer/Deserializer
6. Spring Boot
7. Java 8/11

Overview

This repository demonstrates CQRS Event Sourcing (Materialized views) with Kafka Streaming (Version: 2.1.0)

In a typical production environment, we have multiple microservices and we want to perform multiphase commit to each microservice databases. Let’s take an example of order checkout and fulfillment process. The user wants to place an order in eShopping application, we have different microservices do the following operations (mentioned high-level tasks)

1. Check the inventory for the requested product (Inventory MicroService takes care of this)
2. Check the if any Payment method available and process payment(Payments MicroService takes care of this)
3. Get Shipping Address and Billing Address (Customer Management MicroService takes care of this)

If anyone of the above microservice fails, we want to roll back the transaction and roll back updates made to microservices

In this repository all of the above operations (except rolling back) done through Event Sourcing or Event Streaming. In simple words, we split each transaction into small operations and then process through multi-phase commit.

For the sake of simplicity, in this repository, you can send customers, orders and greetings events through URL and processed in listener/receiver then stored in Kafka State Stores, which then accessed through REST API implemented through Jetty Server (Not MicroServices REST API).

Though the workflow engine is required to automate the whole process, it is out of the scope for this article and not implemented. You can automate and implement the roll back mechanism with the help of a workflow engine. There is a nice article from Bernd explaining about workflow engines and automation process. Please read it.

How to Run the demo?

Please download the repository KafkaStream-CQRS-EventSourcing from GitHub before proceeding

1. Download and install Kafka either from Confluent or follow instructions from here first. I recommend Confluent as it combines all the servers into one package with additional tools. start Kafka with the following command

$ <path-to-confluent>/bin/confluent start

2. Clone this repository and open in IntelliJ or Eclipse as maven project and run KafkaStreamApplication class. This will bring up producer class.

3. Now open Kafka user interface to create topics. Go to http://localhost:9021 => Topics=> create topics customer, order and order-to-ktable (if they do not exist)

4. Go to EventsListener class and execute the main method to start REST Proxy (Jetty) which accesses Kafka Materialized view data through REST API

5. You have 2 options to send events. One, Use the following web pages to send customer, order and greetingsevents or follow steps 6 and 7

• Go to the page http://localhost:8090/create-customer to send customer event
• Go to the page http://localhost:8090/create-order to send order event
• Go to the page http://localhost:8090/create-greeting to send greeting event

6. Skip this step if you followed step. Go to http://localhost:8090/ to send events and retrieve and see data from Kafka Materialized views.

• Since EventsSender application and Jetty Server run on different ports( you can run them as different applications), I used RestTemplate to get data from Kafka Materialized views and show it in HTML pages

7. Skip this step if you followed step 5, otherwise sent events can be accessed through REST API created through Jetty Server which gets it from Kafka State Store. See StateStore Rest API for possible methods, you can customize it further

• Go to http://localhost:8095/store/customer-order/all to see all customer orders
• Go to http://localhost:8095/store/customers to get all customers
• Go to http://localhost:8095/store/orders to get all orders

---

Modify code, if you want to send events with different information. Let me know in comments if you have any questions