Message Queue

Overview

Message Queue (MQ) is a mechanism for asynchronous communication in distributed systems. It allows different system components to exchange data by sending and receiving messages without direct connections. Message queues act as middleware between systems, providing reliable message delivery, storage, and processing functions.

Main Features

1. Asynchronous Communication: The message sender and receiver do not need to be online simultaneously. After a message is sent, it is stored in the queue, and the receiver can read it at their convenience.
2. Decoupling: Senders and receivers do not communicate directly but exchange data through message queues, reducing coupling between systems.
3. Persistence: Message queues typically provide persistent message storage to ensure messages are not lost during system failures.
4. Load Balancing: Through message queues, loads can be distributed to multiple processing nodes, improving system throughput and reliability.
5. Order Guarantee: Some message queue implementations can guarantee message order, ensuring messages are processed in the order they are sent.

Working Modes

Two working modes of the exchange:

• Routing by Identifier: When a message arrives at the exchange, it is delivered to the queue where the queue identifier matches the delivery identifier.
• Broadcast Mode: When a message arrives at the exchange, it is delivered to all queues.

Common Message Queue Systems

• RabbitMQ: An open-source message queue system based on the AMQP protocol, supporting multiple messaging patterns and persistence mechanisms.
• Apache Kafka: A high-throughput, distributed message queue system suitable for real-time data streaming and log collection.
• ActiveMQ: An open-source message queue system under the Apache Foundation, supporting multiple protocols and persistence methods.
• Amazon SQS: A fully managed message queue service provided by AWS, with high availability and scalability.

INFO

RabbitMQ is used to implement message queues in Informat.

Usage Scenarios

• Asynchronous Processing: In web applications, user requests can immediately return responses, while background asynchronous processing is handled through message queues.
• Task Scheduling: Scheduled tasks or batch processing tasks can be scheduled and executed through message queues.
• Log Collection: System logs and events can be collected and processed through message queues, facilitating centralized management and analysis.
• Microservice Communication: In microservice architectures, services communicate through message queues, reducing direct dependencies between services.

Message Reading

After consuming messages, consumers need to send acknowledgments to the system based on consumption status:

• After sending a confirm acknowledgment, the system will delete the message from the queue.
• After sending a not confirm acknowledgment, if the re-delivery parameter is specified, the message will be re-delivered to the queue; if not set, the message will be discarded. This method is generally used when exceptions occur during message consumption.
• After sending a reject acknowledgment, if the re-delivery parameter is specified, the message will be re-delivered to the queue; if not set, the message will be discarded.

Message Acknowledgment Mechanisms

Manual Acknowledgment

In this mode, after consuming a message, consumers need to return an acknowledgment to the Broker based on consumption status: whether to confirm (ack) to make the Broker delete the consumed message, confirm failure (nack), or reject the message. After enabling manual acknowledgment, if a consumer crashes before returning an ack after receiving a message, the message will not be lost. The message will only be deleted from the queue after receiving the returned ack.

Ignore Exception Automatic Acknowledgment

By default, consumers automatically acknowledge while ignoring exceptions and do not care whether the message was successfully consumed/processed.

Automatic Acknowledgment without Exception

1. If no exception is thrown during consumption, automatic acknowledgment is performed.
2. If an exception is thrown, the message will return to the queue. If only one consumer is listening to the queue at this time, the message will be pushed to this consumer again after returning to the queue, causing an infinite loop.

Re-delivery Mechanism

When the acknowledgment mode is manual acknowledgment or acknowledgment based on conditions, if an exception occurs, a retry delivery mechanism is needed.

Default retry mechanism is as follows:

• Maximum number of retries: 5 times
• Maximum retry interval time: 30 minutes
• Initial retry interval time: 5 seconds
• Interval time multiplier: 4

The first retry will be triggered after 5 seconds, the second after 45=20 seconds, the third after 204=80 seconds, the fourth after 804=320 seconds, and the fifth after 3204=1280 seconds.