System Design-Distributed Messaging Queue- RabbitMQ and Kafka

 


UseCase: Asynchronous
Advantage: Less latency
                

 

UseCase: Retry/Circuit breaker

UseCase: Pace matching


Point2Point Vs PubSub
P2P - one message can be consume by one consumer at a time



PubSub- one message can be consumed by many consumers


Kafka Architecture

Partition 0 in topic A can communicate with only one consumer in CG1 only at a time but it can communicate with other CG2


Cluster Vs ZooKeeper- 
list of topics is cluster
communication between topics take care by zoo keeper


Message:-
    Key- 
    Value- Actual msg -mandatory
    Partition-
    Topic- mandatory

Consumer have below information

Consumer No
Consumer grp
Topic no
Partion no
Offset no- stating that this mush consumer have read.

Use of offset: If the consumer goes down then the next consumer will read the message and check the offset that from where i need to read the message. and hence consumer group is required.




Leader (w/r)and Replica Partition
If leader is down then the Replica become a leader



what happen if the message read failes.

the consumer retry till max limit after that it pull the message in the dead letter queue.



Kafka vs Rabbit MQ
-> Kafka consumer pulls the messsage from Q
->Rabbit MQ push the message to consumer



RABBIT MQ

FANOUT->Broadcast the message to all the Queue
DIRECTEXCHANGE->Exactly matching(message key=routing key)
TOPIC Exchange -> Wild card matching

If consumer is not able to process the message and it failed or not able to process and it will reque it again. If thresold then still not processed then it will put it in the dead letter Q.




















 

Comments

Post a Comment

Popular posts from this blog

Microservice Pattern: SAGA

Image
  Context You are developing a server-side enterprise application. It must support a variety of different clients including desktop browsers, mobile browsers and native mobile applications. The application might also expose an API for 3rd parties to consume. It might also integrate with other applications via either web services or a message broker. The application handles requests (HTTP requests and messages) by executing business logic; accessing a database; exchanging messages with other systems; and returning a HTML/JSON/XML response. There are logical components corresponding to different functional areas of the application. Problem What’s the application’s deployment architecture? Forces There is a team of developers working on the application New team members must quickly become productive The application must be easy to understand and modify You want to practice continuous deployment of the application You must run multiple instances of the application on multiple machines ...
Read more

Microservice Pattern: Database per service Context

Image
Let’s imagine you are developing an online store application using the   Microservice architecture pattern . Most services need to persist data in some kind of database. For example, the   Order Service   stores information about orders and the   Customer Service   stores information about customers. Problem What’s the database architecture in a microservices application? Forces Services must be loosely coupled so that they can be developed, deployed and scaled independently Some business transactions must enforce invariants that span multiple services. For example, the  Place Order  use case must verify that a new Order will not exceed the customer’s credit limit. Other business transactions, must update data owned by multiple services. Some business transactions need to query data that is owned by multiple services. For example, the  View Available Credit  use must query the Customer to find the  creditLimit  and Orders to calcula...
Read more

SQL vs NoSQL | Difference between SQL & NoSQL | SQL Vs NoSQL Tutorial | SQL, NoSQL system design

  SQL  1. Structure          Table,row,column          PreDetermined Schema                    Create table with all the defined datatypes and size.                    Then you can use the table          Relations               1:1,1:M,M:M 2. Nature            Centralized data-all tables in the same database server 3. Scalability          Vertical                     Increase the RAM,Storage         Horizontal -> NOT good for horizontal scaling                    Sharding                      ...
Read more