Introduction
The VIDIZMO platform utilizes an event-driven architecture for efficient communication between various components. VIDIZMO offers users various broker service options, among which Rabbit MQ is leveraged as one of the choices. RabbitMQ is a well-known open-source message broker that plays a crucial role in the VIDIZMO architecture. It is used as a broker service to facilitate the reliable exchange of messages between microservices.
For a deeper understanding of VIDIZMO's architecture, we recommend exploring our article titled "Design and Architecture Overview."
Understanding RabbitMQ
Rabbit MQ is a message broker that accepts and forwards messages between different applications. It operates using the Advanced Message Queuing Protocol (AMQP), which defines the behavior of message publishers and consumers and ensures reliable message transportation.
Functioning as a message queue, RabbitMQ efficiently manages the routing, queuing, and delivery of messages from producers (senders) to consumers (receivers). It implements a versatile publish-subscribe model, where messages are sent to exchanges. These exchanges then route the messages to the appropriate queues based on predefined rules. This architecture decouples system components, enhancing scalability, fault tolerance, and flexibility in system design.
For a better understanding of RabbitMQ, please refer to its documentation.
VIDIZMO and RabbitMQ: How They Work Together
In VIDIZMO's architecture, events such as video uploads trigger the publication of messages containing event information to RabbitMQ. These messages are then routed to the appropriate services based on configurable rules. RabbitMQ uses exchanges and queues to facilitate this message routing.
Exchanges in RabbitMQ act as message routers. They determine how messages from publishers are handled and to which routes they are directed. VIDIZMO uses the fanout exchange type, which directs messages to all bound queues regardless of specific criteria. Queues serve as temporary storage for these messages until they are consumed by a subscribed service.
The key aspects of message handling in VIDIZMO depend on the following factors:
Exchanges and Queues
Let's explain how exchanges and queues operate within VIDIZMO's architecture. When an event occurs, a message containing information about the event is published to RabbitMQ. RabbitMQ then routes this message to the appropriate services based on configurable rules using exchanges and queues.
There are several types of exchanges, including event exchanges, telemetry exchanges, and tracking exchanges. For instance, if a workflow-related event occurs, the message is published to the event exchange. Exchanges route messages based on predefined rules. The fanout exchange type is used. This exchange type sends the message to all queues bound to the exchange. This approach ensures that all instances of a particular service receive the message, providing redundancy if one service instance fails.
When a message containing event information is published, it is directed to an exchange. The exchange then routes the message to the appropriate queue. For example, the event exchange directs messages to the workflow service queue. There are several basic queues, such as event service, workflow service, notification service, telemetry service, and tracking service.
Queues act as buffers, temporarily holding messages until a subscribed service consumes them. Messages are enqueued and dequeued in a first-in, first-out (FIFO) manner. For example, messages in the workflow service queue are consumed by the workflow service. The consumer services include event logger services, workflow services, telemetry services, and tracking services.
VIDIZMO operates multiple instances of the same service, such as the Workflow service. If one instance fails due to an error, the Workflow service remains operational through the other instances.
To ensure that each event is processed by only one instance of the Workflow service, VIDIZMO utilizes queues in the RabbitMQ. When a service instance starts, it creates a queue named after the service, for example, "Workflow service." If two instances of the Workflow service are running, they share the same queue name and are added as consumers. This approach enhances VIDIZMO's high availability.
RabbitMQ manages these queues and ensures that if there are multiple consumers for the same queue, only one instance of the Workflow service processes each message. This prevents duplicate processing and maintains efficient service operations.
The Queues behave according to the defined parameters, so there is a set of some mandatory and optional parameters that are defined so that Queues behave accordingly when they store messages.
- Name: VIDIZMO assigns names to queues for reference purposes. These names, selected by VIDIZMO, can be up to 255 bytes and consist of UTF-8 characters.
- Durable: This property is turned on, which means the messages in the queue will remain even if the broker service restarts. The delivery mode controls how the broker treats the message from a durability standpoint. The delivery mode for a message is set to persistent. The value of the delivery mode tells RabbitMQ it must store the message on disk first. Messages marked as persistent messages that are delivered to durable queues will be stored to the disk.
- Exclusive: Only one connection uses the queue at a time, and it disappears when that connection ends. In VIDIZMO, this property is kept off, meaning multiple connections can access the queue, and it won't disappear when a single connection closes.
- Auto-delete: In VIDIZMO, this property is kept off to ensure the queues are not automatically removed even if no more consumers are subscribed to them.
- Arguments (optional): In VIDIZMO, this optional property is left as null, meaning no extra configurations are specified.
Binding
A binding is a link set up to bind a queue to an exchange. It defines the routing rules for deciding which messages go to which queues. Since VIDIZMO leverages Fanout Exchange, the routing logic is straightforward in this type of exchange. It doesn't look at the message's routing key or any patterns. It just sends the message to all the queues that are connected to it.
Consumers
Another critical element that constitutes RabbitMQ is Consumers, which are services that take messages from queues. In the context of VIDIZMO, consumers mainly include Event loggers, workflow services, notification services, telemetry services, and logger services. RabbitMQ, by default, ensures that all the messages consumed from a queue are distributed evenly among consumers.
Message Delivery and Acknowledgement
When RabbitMQ delivers a message to a VIDIZMO service (consumer), it needs to know when the message is considered successfully received. For example, a workflow service receives a message about a new video upload. Similar to when you sign for a package, the service sends a basic acknowledgment back to RabbitMQ. This basic acknowledgment tells RabbitMQ that the notification has been received and processed successfully and that the service is ready for the next message. With this confirmation, RabbitMQ can then safely remove the message from the queue.
If the service is unable to process the event due to an error, it will send a nack to RabbitMQ instead of ack so that messages don’t get lost. This is like a red flag that informs RabbitMQ that the message cannot be processed. Often, nacks come with an opportunity to retry. RabbitMQ might put the nack (acknowledgment) message back in the queue so the consumer can attempt to re-process it. This is helpful for temporary errors that might get resolved on a second attempt.
Considerations
Please consider the following key factors and recommendations for utilizing RabbitMQ, the VIDIZMO application:
Deployment Consideration
RabbitMQ is a well-suited message broker option in VIDIZMO for cloud or on-premises deployments. It offers horizontal scaling to accommodate increasing message volumes. Additionally, RabbitMQ provides features like acknowledgments and redeliveries to ensure reliable message delivery. Security features like authentication and authorization help control access and secure communication between VIDIZMO and the message broker.
Message Volume and Characteristics
RabbitMQ can handle moderate message volumes, such as thousands of messages per second, making it ideal for scenarios with moderate throughput needs. While its centralized architecture might add some performance overhead, it ensures robustness and maintains message integrity.
Virtual Host
RabbitMQ supports the creation of virtual hosts within a single server. These virtual hosts provide isolated environments where exchanges, queues, and users can be defined independently. The virtual host property defines the logical grouping of the RabbitMQ channel. A virtual host has a name. When a client connects to RabbitMQ, it specifies a virtual host name to which to connect. If authentication succeeds and the provided username has been granted permission to the virtual host, a connection is established. Connections to a virtual host can only operate on exchanges, queues, bindings, and so on in that virtual host. By default, in VIDIZMO, the virtual host path is set. If a host is not explicitly defined in RabbitMQ, it will automatically connect to the virtual host.
Key Operational Features of RabbitMQ
Security
The authentication and authorization mechanism in RabbitMQ makes it a secure choice as a broker service. Before performing operations, the application must authenticate, meaning it needs to present and prove its identity. With that identity, RabbitMQ nodes can look up their permissions and grant access to resources such as virtual hosts, queues, exchanges, and so on. A client uses a username/password pair to authenticate. These pairs can be used with a variety of authentication backends that verify the credentials. Connections that fail to authenticate are closed, and an error message is recorded in the server log.
High Availability
RabbitMQ provides features to support high availability and disaster recovery. Implementing a high availability (HA) cluster for RabbitMQ ensures continuous message delivery even if a broker node fails.
To configure HA in Rabbit MQ, refer to our documentation “RabbitMQ HA Implementation.”
Scalability
The core principles for building scalable applications include decoupling software components and performing background and asynchronous operations. RabbitMQ facilitates all three, enabling VIDIZMO to scale efficiently. VIDIZMO can increase its message-handling capacity to accommodate growth without compromising performance. RabbitMQ clustering allows the setup of multiple RabbitMQ nodes that act as a single logical broker, providing redundancy and scalability. If VIDIZMO experiences a surge in activity, adding more nodes to the cluster distributes the load, ensuring smooth operation.
Monitoring
RabbitMQ includes a management panel where you can manage all queues, exchanges, bindings, and messages. This panel also provides detailed statistics, such as messages per second, messages per queue, and messages per exchange. Various management tabs display comprehensive statistics about your RabbitMQ instance, allowing you to monitor and manage all queues and the message activity within each queue.
Troubleshooting
This section provides troubleshooting tips and recommendations for a few issues that you may see when using RabbitMQ as the broker service.
Port Accessibility Verification
To troubleshoot connectivity issues, ensure that the necessary ports are accessible. The ports RabbitMQ is usually configured on are 5672 and port 15672 for RabbitMQ management.
Use Microsoft TELNET to check the status of these ports and identify any potential blocking issues within the security system.
Connection Errors
The following are the possible connection errors that you might encounter.
- If VIDIZMO cannot connect to RabbitMQ, it may be because it is not correctly configured. Verify that VIDIZMO's RabbitMQ connection settings (username, password, host) are correct.
- Check if RabbitMQ is running and accessible using the configured hostname and port.
- Ensure firewall rules on your premises allow communication between VIDIZMO and RabbitMQ.
Message Delivery Errors
- If messages are getting stuck or are not being consumed by RabbitMQ queues, try reducing the chunk size of the messages to resolve the issue.
- If VIDIZMO, as a publisher, is not confirming message delivery, it could be due to memory limitations. Consider reducing the memory size to address this issue.
Access-control
If authentication and authorization errors occur, server logs may show entries of failed authentication attempts. Refer to the Troubleshooting Authentication guide for further details on resolving these issues.
The issues mentioned above are among the most commonly encountered. For detailed troubleshooting instructions, refer to Troubleshooting Guidance.
Kindly refer to our comprehensive guide for setting up RabbitMQ in the VIDIZMO portal. "Configuring RabbitMQ as a Broker Service in VIDIZMO."