How to start an event driven microservice backend application using Spring Kafka

By -

Have you ever started a project from the start and it became a very large Project? 

As your application grows, the complexities to your code also grows. Even the once perfect, orderly project can turn into chaos because you constantly put more and more features, services and assets into the project, making maintenance a nightmare. 

One of the solutions to prevent a growing application from becoming a mess is by separating your application into smaller microservices. Microservice is a contrast to monolithic architecture, rather than putting your feature into one giant codebase, in microservice you divide your project into smaller projects(microservice), and then each microservice has only one responsibility with a specific task or functionality.

Here’s some feature what most application has that can be divided into microservices:

1. User authentication and user Management

Make your authentication and authorization of your APIs into different services, usually this type of service called Single sign on(SSO),  using some techniques like producing JWT tokens that can be used for authentication located on other services. Other than signing in, you can also put your user registration, user details and signing out combined in this service.

2. Website and app information

You can separate your company, web or app  information management into specific services, so that it makes it easier to maintain the code related to this feature. You can put features like managing pages, links, blogs in this type of microservice.

3. Product management Microservice

Whatever is related to the product must be handled by this microservice. This service provides APIs for listing, sorting, and product searching. Also handle the creation and updating of products.

4. Order Management

Processing and tracking orders can be isolated into a microservice. You can handle product checkout, order listing, order status updates, and related operations in this service.

5. Payment Gateway

Integration with external payment providers, managing transactions, and handling payment-related operations can be dedicated to this microservice.

6. Analytics and Reporting

Collecting and analyzing application usage data, generating reports, and providing insights can be performed by a separate microservice. This service aggregates and processes data, enabling businesses to gain valuable insights.

7. File Management

Handling image or file uploads, storage, retrieval, and serving can be abstracted into a dedicated microservice. You can utilize some cloud storage solutions or have your own storage infrastructure.

Microservice Spring and Kafka

When it comes to building scalable, maintainable backend systems, the Spring framework is one of the most popular choices not only in the Java ecosystem, but considered to be the perfect solution out of many frameworks in many programming environments.

In this article, we will develop a starting point on the journey of event-driven microservices using Kafka.

With Spring, you have the flexibility to choose the modules that best suit your needs. It provides seamless integration with various messaging systems, and one standout option is Kafka. Kafka is the right distributed streaming platform because Kafka is very excels at handling high-throughput, fault-tolerant, and scalable event streams.

This is a guide process step by step to start your Microservice in spring:

1. Setting Up Kafka Broker

First things first, we need to set up our development environment. Course you must have a kafka broker for your message streaming. Kafka broker is  a single instance of the Kafka server. It is just a running Kafka process that handles the storage of your message. Your kafka message is distributed into topics and then your message is consumed by a kafka group.

Best practice to set up a kafka for development is just by using Docker. Use this simple docker compose:

To run or start your Kafka broker for development, use the following docker command on the same folder where you put the docker-compose.yml:

docker-compose up -d

Open your docker, if you are seeing any Kafka broker in your docker container, it means that you have successfully run Kafka. Now that your kafka is running on your machine, let’s continue to configure it into your Spring project. 

So let's dive into the exciting part of coding our backend application development, with event driven microservice architecture.

2. Implementing Microservices with Spring Boot

With a kafka server running and a solid foundation about micro service provided above, it's time to get our hands dirty by starting to write our example microservice. We will be implementing the Product Microservice based on the chart above, a simple project consuming  product ID and producing and producing the ID to another topic.

To use Kafka in Spring-boot project, add Kafka in your maven dependency list.

And then set your kafka configuration inside application.yml, here’s an example of Kafka configuration in Spring boot, in the local machine, you can just use localhost. If you run Kafka on the server, change the bootstrap server with your server IP address.

Now comes to the Java part, integrating Spring Kafka into our microservices by writing  consumers and producers. In this example you will learn how to consume events within your microservices and how to publish events to Kafka topics, enabling seamless communication and event-driven behavior.

3. Testing

A robust backend application requires rigorous testing. In this tutorial I will provide you with a simple testing of our application. It will test the process of testing your event-driven microservices, ensuring that they perform as expected. Here’s how a simple  test can be written.

The above code is to test the processing of the kafka event method inside the MyKafkaService. When our application is running, processMessage will be triggered each time there’s an event coming into the productTopic.

In the next chapter we'll also discuss deployment strategies and best practices for scaling your system to handle increasing loads, so stay tuned.

This is just your first journey of event-driven microservices using Spring Kafka. Microservice means breaking down the application into smaller, independent services, you now maintain a project that has good levels of scalability and maintainability. With this new knowledge, you're well-equipped to the next level, I hope you can do more implementation of building scalable backend systems. Go fort and start building more amazing applications that you think will have a positive impact, but always remember that you must always love the process of building it.

I am an experiences web developer, have a good problem-solving and deep understanding of software development principles. With comprehensive skill set in Java, PHP, Python, C and JavaScript. Able to do database thing like MySQL and Oracle. Along with expertise in frameworks like Spring, React, and Firebase. Have a solid understanding of software development methodologies, like Agile and Scrum. I applies best practices to ensure the development process is efficient and of the highest quality output.

Popular posts from this blog

Spring Kafka - how to use ReplyingKafkaTemplate send and reply synchronously

By -
Image
Microservice Spring with Apache Kafka Kafka is an asynchronous messaging queuing system, but when you have a case in your project that you need to have the result immediately from another service you can do that quite easily. Kafka from the client side produces a message to the server side, then waiting for the server to return back the message exactly what the client wants. It’s identified in the kafka header request named KafkaHeaders.CORRELATION_ID and a topic where the server should send. If the server side produces the same correlation id to the same topic in the request, then the client side will catch that message as its reply. Although you can work with kafka from a different framework and language other than Java and Spring. Here’s in this example i just specifically giving the example of sending and replying kafka messages using Spring Boot. The Project Overview The client-project will send a string message to the server-project over kafka, then the server-project wil
Read more

How To Connect SSH Using PEM Certificate On Windows

By -
Image
Usually you will need a pair of SSH username and password to connect to a remote server. But there's also another way to connect to the server via SSH but without password, just using a dot pem (example.pem) file. Here's how we usually do on Windows 1. Let’s SSH login using pem to confirm if it’s throwing error or the pem is fine, if error then we need to move to step 2 to fix it: ssh adam@10.0.0.123  -i server-cert.pem And here's the usual error that will throw if your pem file is not ready for SSH login. You need to follow the next step to make your pem certificate ready to login to your server via SSH. 2. Open Windows Explorer, then right-click on the pem file, then select "Properties". Navigate to the "Security" tab, and click "Advanced" 3. Disable inheritance 4. Delete all permissions, so it doesn’t have any permission 5. Now add you as the only only owner. By click button Add and then Select a principal, then Advanced. 6. On the showing p
Read more

ERROR 1348 Column Password Is Not Updatable When Updating MySQL Root Password

By -
Image
MySQL and MariaDB password As already mentioned in the title of this blog, this error message sometimes occurred when we are trying to update our root password database, either MySQL or MariaDB, The error says: ERROR 1348 (HY000): Column 'Password' is not updatable This is because if we use this SQL command to update our database root password, and it turn out to be Restricted by MySQL to use update on mysql database. UPDATE mysql.user SET Password=PASSWORD('1234') WHERE User='root'; The solutions for MySQL Error1348 So you can not update a user password using Update commands, to update the password, use ALTER commands instead. like the following. ALTER USER 'root'@'localhost' IDENTIFIED BY '1234'; And then you need to do the flush privileges to commit previous queries (that alter command above) into the system, simply do like this. flush privileges; So now you have your root with password 1234. Although it is recommended to
Read more

How To Create Spring Boot Project Using Netbeans

By -
Image
Spring Boot is program that you can use to develops Spring Framework easier, without the need of using so many hard to handle configuration. This tutorial, we are gonna create Spring Boot Application by using Netbeans IDE. You can of course create Spring Boot Appilcation on Netbeans, but it will be easier if just using plugin for Spring development in Netbeans, 1. Open Netbeans, click tools menu and click plugin. After plugin window opened, click tab Available Plugins and find the "Nb SpringBoot" plugin then install and activate the plugin. 2. After restarting the Netbeans IDE. Now you gonna be able to create a new project with Spring boot application. File > New Project > Under categories, choose Maven > then choose  Spring Boot Basic. Now you can start the development of spring boot application using your Netbeans IDE. Open pom.xml to add spring boot dependencies you want.
Read more

Upload and Download Rest API using Spring Boot Reactive WebFlux

By -
Image
Reactive Spring Logo This is maybe a very simple task, but for those of you that may not be familiar with WebFlux found it difficult to create a Rest API that handles upload and download in Reactive spring application. Create the skeleton Spring project Create a new project, go to start.spring.io , on the dependencies section, add Spring Reactive Web. Generate and extract it into your desired project location. Instead of using spring-boot-starter-web, use spring-boot-starter-webflux instead in your pom.xml. Here’s the pom.xml for our Reactive Rest API project. Create our Rest controller Now that you have set to use WebFlux instead of the usual web, we have to change a little bit in the Controller class that handles the API endpoints. It doesn’t change much but you have to wrap the Response in Mono and the request parameter to FilePart. So our upload and download file controller will look like this. Create utility to handle file upload We need to structure our project better, so we cr
Read more