Introduction

VIDIZMO relies on a central broker service for inter-communication among its services. This broker service is a crucial component of the Event-Driven system used by VIDIZMO. VIDIZMO has different brokers available, with Azure Service Bus being one of them. The broker service is responsible for facilitating smooth communication among different programs and services within the system. This article focuses on Azure Service Bus and its role as a smart intermediary in this process.   

For a deeper understanding of VIDIZMO's architecture, we recommend exploring our article titled "Design and Architecture Overview."


Understanding Azure Service Bus

Azure Service Bus (ASB) plays a pivotal role in facilitating seamless communication within the VIDIZMO architecture. As a cloud-based broker service provided by Microsoft, Azure Service Bus serves as a central hub for inter-communication between various components and services in the VIDIZMO ecosystem. Acting as a smart intermediary broker service, ASB ensures that different programs and services communicate smoothly, fostering a reliable and organized exchange of information.


To delve deeper into this concept, explore the "Concepts of Azure Bus Service."

VIDIZMO's Integration with ASB 

VIDIZMO uses Azure Service Bus to create multiple topics for each of its services, serving as a communication channel between components. In VIDIZMO, the Azure Service Bus (ASB) uses the Pub/Sub (Publish/Subscribe) messaging pattern. When various components need to be informed about a particular event, the publish-subscribe pattern of Azure Service Bus ensures effective communication.


Azure Service Bus employs message queues to store and manage messages, ensuring efficient communication between different parts of VIDIZMO. This queuing mechanism helps in organizing and regulating the flow of data.


Utilizing Advanced Message Queuing Protocol (AMQP) Azure Service Bus ensures the secure and efficient transmission of messages. This protocol enhances the reliability and interoperability of VIDIZMO's communication infrastructure.


When a new video is uploaded or requires processing, such as transcoding, an event is triggered and sent to an ASB topic. Each topic, resembling a virtual queue, receives a copy of these messages independently. In this context, subscriptions establish links between services and events. This enables services like Event Logger, Workflow, Notification, Telemetry, Tracking, Analytics, and Scheduler within VIDIZMO to subscribe to relevant topics via subscriptions, ensuring they only receive messages pertinent to their interests. Subscribed services are the recipients of event messages from these topics and process the events in accordance with the designated functionality. Overall, this architecture fosters modular and scalable communication between services within the VIDIZMO platform.


The key aspects of message handling in VIDIZMO:

  • Ordering of messages

Messages are delivered in the exact order they were published in a FIFO manner. In specific workflows, such as video encoding, the precise order of messages within a topic is essential.

  • No partitions

A queue or topic in VIDIZMO is managed by a single message broker and stored in one messaging store. For VIDIZMO, it's not necessary to partition messages within Azure Service Bus (ASB) topics, as message order is crucial for delivering messages, and partitioning doesn't ensure order.

  • Max message size handling

The maximum message size supported by ASB depends on the chosen messaging tier (Basic, Standard, or Premium). It's important to note that topics and subscriptions are only available in the Standard or Premium tiers; they are not supported in the Basic tier. ASB enforces limitations on the size of messages transmitted.


These limits vary depending on the tier you're using. Standard Tier includes 256 KB (including headers). Premium Tier includes 1 MB for messages using HTTP (including headers) and 100 MB for messages using AMQP (including headers).


If the size of an incoming message exceeds the set quota, the service bus will not receive it and will reject it, and the calling code will receive an exception.


To handle incoming messages that exceed the set quota, VIDIZMO employs a compression technique that is implemented within our application logic. There is no direct configuration option in Azure Service Bus to handle messages larger than the quota. However, we provide the option for users to configure "Compress Messages After (bytes)" for the compression of messages to occur after reaching the specified size. VIDIZMO checks this message size before sending messages and compresses them if required.


Setting a threshold for compressing messages in Azure Service Bus can be a proper optimization technique, especially for scenarios involving large message payloads. However, it's important to note that implementing message compression adds complexity and might introduce overhead due to the additional logic required. There's a potential risk of message loss if compression and decompression operations fail due to various reasons, such as network issues and system failures. Nonetheless, the Azure Service Bus is designed to handle message processing robustly, and such failures are rare.

  • Receives and  Deletes message

Once a consumer successfully processes a message, it sends an acknowledgment to ASB. This acknowledgment triggers the deletion of the message from the queue or topic, preventing unnecessary redelivery. In some configurations like TTL (Time-to-Live), where value is defined as a specific retention period, messages might be retained for a particular period irrespective of acknowledgment. ASB automatically deletes these messages upon expiration.TLL effectively manages undelivered messages and ensures they are not retained indefinitely.


When a specific topic lacks active subscriptions, and an event listener is engaged, incoming events like 'start workflow' may arrive during this period, each with a TTL (time-to-live) of 10 seconds. If no subscriber is actively listening within this timeframe, these events will be discarded according to their TTL. Consequently, when VIDIZMO's service reactivates, it will not process these discarded events. This behavior ensures that user-initiated actions, such as processing workflows, do not occur during periods of inactivity and are not queued for later execution.

  • Batch Processing

VIDIZMO efficiently manages and processes large volumes of data by grouping messages into batches, improving performance and scalability within its architecture. 


VIDIZMO  primarily uses batching to receive messages in a batch. This approach involves aggregating multiple messages in a batch, thereby reducing the number of requests and optimizing network traffic. The batch size is ten. VIDIZMO receives the batch and notifies each message one by one to every consumer service at the same time through an event. If any of the messages fail for any reason, the entire batch won't be rolled back, but only the message that failed will be. The order of messages may vary, but the process is unaffected.

  • Retries

VIDIZMO has already implemented a pre-defined retry strategy within its codebase. By managing the retries internally, VIDIZMO provides a smoother deployment experience for its customers. This means that customers do not need to delve into complex ASB configurations, making it easier for them to get started and use VIDIZMO effectively.


For instance, consider an event sent by a web application, and for some reason, the VIDIZMO web application fails to publish a message to ASB, then it will retry three times till the message is sent and published to ASB.VIDIZMO's retry mechanism ensures that only failed messages are retried, preventing duplicate processing. The default value is three retries. Messages exceeding the maximum retry threshold are sent to ASB. Regardless of the pre-configured retry settings, ASB provides a dead-letter queue (DLQ) where messages exceeding the maximum delivery attempts are stored. This allows customers to monitor the DLQ and identify persistent failures, guaranteeing successful message delivery.

  • Support for Premium Messaging

The Premium tier is designed to meet the expected scale, performance, and availability requirements of critical applications. It is highly recommended for production scenarios. VIDIZMO offers support for Premium messaging, which provides several advantages over the Standard tier. 


The Premium tier allows for much larger message sizes and handles larger volumes of messages with lower latency compared to the Standard tier. The HTTP/SBMP messages have a limit of 1 MB, while the AMQP messages have an impressive limit of 100 MB.


Read more: Service Bus premium messaging tier.


Considerations


Please consider the following key factors and recommendations for utilizing Azure Service Bus with the VIDIZMO application:  

  • Deployment Consideration

When deploying VIDIZMO on an Azure platform, utilizing a message broker like ASB (Azure Service Bus) can be a strong option. ASB offers scalability, reliability, and managed services that are ideally suited for cloud-based deployments. However, ASB is technically usable on-premise, but latency and throughput limitations make it less desirable.

  • Manage Services

For users seeking a fully managed message broker solution, Azure Service Bus (ASB) stands out as a compelling choice.ASB offloads infrastructure management, maintenance, and updates entirely to Azure. This translates to a streamlined experience for clients who prioritize a hands-off approach, allowing them to focus on core application development and business logic.

  • Throughput

Choose the appropriate pricing tier (Standard or Premium) based on your throughput needs. The Standard tier is for low volumes and test environments, while the Premium tier offers higher capacities, predictable performance, and auto-scaling for production use.

  • Monitor usage 

Keep track of message throughput and latency to ensure your chosen tier meets your application's demands. For further details, visit the official documentation ”Monitor Azure Service Bus.”

  • Message Security

To enhance data security during message exchange, VIDIZMO integrates seamlessly with Azure Service Bus (ASB). It provides encryption during transmission, which means that data is encrypted while it is being sent. This includes encryption when data is moving from the client to the Service Bus, within the Service Bus, and from the Service Bus to the consumer. Apart from encryption during transmission, Service Bus also performs encryption at rest, which means that messages are encrypted while they are stored. This process is automatic, and users do not have to do anything to enable it. To fully understand, refer to Azure Service Bus security documentation.

  • ASB Access Key

ASB utilizes Shared Access Signatures (SAS) for authentication. This involves creating cryptographic keys that grant specific access rights to ASB resources within a namespace. VIDIZMO leverages connection strings to connect to ASB securely. These strings include the endpoint URL, SAS key name, and the specific entity path. This method provides a secure and easy way for VIDIZMO to interact with ASB while adhering to best practices for data protection and communication.

Benefits of Azure Service Bus 

  • Scalability    

VIDIZMO is a video management platform that may experience varying workloads based on user demand and video upload/download patterns. Azure Service Bus enables VIDIZMO to scale its messaging infrastructure dynamically based on changing workloads, ensuring smooth operation during peak usage periods. 


  • Reliability and High Availability  

VIDIZMO's workflows often involve multiple stages, such as video upload, processing, and distribution. Azure Service Bus ensures reliable message delivery, reducing the risk of data loss and ensuring that VIDIZMO's video-related messages are consistently processed. 

  • Cost-effectiveness 

Offers flexible pricing models based on message volume and throughput, providing cost efficiency. Azure Service Bus offers Standard and Premium tiers to cater to your specific requirements to ensure smooth and reliable delivery. To get more details, refer to Service Bus Premium and Standard messaging tiers. 

  • Security and Compliance 

It adheres to strict security standards, including FIPS 140-2 and ISO 27001, offering peace of mind for security-sensitive VIDIZMO data. 

  • Message Filtering and Routing 

Azure Service Bus enables the filtering and routing of messages based on their content. VIDIZMO leverages Service Bus's adaptability to achieve precise message routing and workflow execution. 

Monitoring

When using Azure Service Bus as a broker service, it is necessary to monitor key metrics to ensure optimal performance and identify potential issues. To meet this requirement, Azure Service Bus offers comprehensive monitoring tools and diagnostic capabilities that enable proactive identification and troubleshooting of issues within your message flow.


By using the monitoring services, a user can keep track of various aspects. These aspects may include but are not limited to:

Messages Requests

Monitoring request rates helps identify potential bottlenecks in message processing, ensuring timely video delivery or encoding tasks triggered by messages.

  • Incoming Messages are an essential metric for measuring bus service performance. It tracks the number of requests made within a specific period, helping to identify message volume trends and potential resource limitations. Additionally, it's crucial to monitor the Average Size of Incoming Messages" to avoid processing delays caused by large message sizes. Tracking these metrics ensures efficient and effective service operation.
  • Outgoing Messages: This metric tracks the number of events or messages that the Service Bus has received within a specific timeframe. Note that the metric excludes any outgoing messages that have been auto-forwarded. It also measures the total number of messages sent by VIDIZMO services and the time it takes for a message to arrive at its intended destination. A high latency could indicate potential issues with message delivery or processing delays on the receiving end.

Dead-lettered messages

Track the number of messages that failed to deliver due to errors and analyze them to resolve delivery issues.

Queue/Topic Length

For smooth and efficient message processing in each queue or topic, it's essential to keep track of the number of messages currently waiting to be processed. High values in the queue or topic might indicate processing delays or bottlenecks. Therefore, it's recommended to monitor the size (number of messages) of your queues and topics regularly. Growing queues indicate slow processing, while empty queues suggest an underutilization of resources.

Latency

The time taken by the Service Bus to fulfill a request. High latency can indicate network congestion or overloaded services.

Connection Metrics

This refers to the count of active, open, and closed connections on a namespace and an entity within it. It keeps track of the number of applications currently connected to the Service Bus and helps identify potential scaling requirements. Monitoring the health and stability of the connection between VIDIZMO and ASB is crucial to identify and address any connection drops that could disrupt message flow.

Monitor Tools and Charts

The Azure portal comes equipped with monitoring tools for ASB namespaces, queues, and topics. You can analyze metrics for Azure Service Bus, as well as metrics from other Azure services, by selecting Metrics from the Azure Monitor section on the home page for your Service Bus namespace. These tools offer visualizations such as queue/topic length charts and message delivery latency graphs. You can use Azure Monitor to keep track of message throughput, latency, and errors, and identify bottlenecks or resource limitations.


Troubleshooting   

This section provides troubleshooting tips and recommendations for a few issues that you may see when using Azure Service Bus as the broker service.

  • Troubleshooting Message Pileup

When using the Azure Service Bus as a broker service in VIDIZMO, if you experience a message piling up in the queue, it may lead to an expectation of QuotaExceededException.


The most common causes of this exception are messages failing to process multiple times and being sent to the dead-letter queue. These messages are kept separate and not sent to the subscribing application. Or the application consuming messages from the queue might have stopped functioning, causing messages to accumulate. 


To troubleshoot the cause, investigate the dead-letter queue associated with the main queue. If messages reside in the DLQ, process them manually or address the root cause of the processing failures to prevent future accumulation. Furthermore, investigate Receiver Activity by reviewing receiver application logs for any errors or exceptions that might explain the halt in message consumption. You can also monitor queue metrics to identify trends and potential bottlenecks.


To resolve this issue, you can begin by examining the messages that were sent to the Dead Letter Queue. Try to re-process these messages to determine the root cause of the failure. Once you have identified the cause, update the system to handle that use case. Some common causes of failure are database locks, network errors, and configuration errors.

  • Troubleshooting Azure Service Bus Error Creating Topic

Another typical error users face is the Azure Service Bus error when creating a topic. To troubleshoot, first verify that when creating a topic in Azure Service Bus, your namespace utilizes the Basic pricing tier. Topics are functionalities exclusive to the Standard and Premium tiers, as the Basic tier caters to simpler queueing needs.


To resolve this, verify your namespace tier within the Azure portal and consider upgrading to Standard or Premium if necessary. Upgrading is a straightforward process within the portal itself. 


To find out more, refer to detailed comparisons of pricing tiers and their features in Microsoft's documentation for informed decision-making.


If you encounter an error while creating a topic, please ensure that the SAS policy associated with your service bus namespace has "Manage" permission on the service bus. These permissions are essential for successful topic creation, regardless of the subscription tier.


Too many topics or queues per namespace can exceed the quota limit, which can cause new requests to be declined. To troubleshoot, check and adjust quotas in the Azure portal or via management APIs. Identify existing resources to optimize usage and delete unused ones to free up space within quota limits.

  • Troubleshooting Azure Service Bus Error Creating Subscription

To ensure the successful creation of a subscription, please make sure that the SAS policy associated with your service bus namespace must have "Manage," "Listen," and "Send" permissions on the service bus. These permissions are essential for the proper functioning of a subscription.

  • Troubleshooting Error Processing Event

If an event processing error occurs, it may be caused by messages that are too large. Depending on the allowed pricing tier type, exceeding the allowed message size for your tier could indeed cause processing errors. Ensure the application consuming messages from the queue is functioning correctly. Monitor CPU and memory utilization on the machines running VIDIZMO applications. More resources can contribute to processing errors.


To get more detailed information, refer to “Troubleshooting guide for Azure Service Bus”


Additional Troubleshooting 

  • Verify network connectivity

Ensure the VIDIZMO app server can access the Service Bus namespace by pinging or browsing its URL. Check for firewall or network security restrictions.


If encountering intermittent connectivity challenges, to diagnose and resolve these issues, consider the following steps:


1. Service Bus Accessibility Check

Ensure the Service Bus namespace is reachable from the VIDIZMO application server. A preliminary test involves browsing or using WGET on 'https://<yournamespace>.servicebus.windows.net/'. This aids in identifying IP filtering, virtual network, or firewall issues commonly encountered in secure network environments. 


2. Port Accessibility Verification 

Confirm that the necessary ports are accessible. VIDIZMO's connectivity relies on ports 443 (HTTPS), 5671 (AMQP), and 9354 (Net Messaging/SBMP). Employ MS TELNET to inspect port status and detect any potential blocking issues on the security system. 


3. Network Security Assessment 

Examine whether organizational network policies might restrict access to Azure Service Bus. Verify that essential ports are not blocked, potentially hindering communication. This ensures VIDIZMO can establish seamless connections with the Service Bus.


Issues that may occur with service upgrades/restarts 

You may experience the following issues with your VIDIZMO app when there are any Service Bus-related upgrades or service restarts. 

  • Requests may be momentarily throttled. 

  • There may be a drop in incoming messages/requests  

  • The log file may contain error messages  

  • The application may be disconnected for a few seconds 

 

Backend service upgrades and restarts may cause these issues in your VIDIZMO application.  


For resolution, the retry policy is already built in Azure Service Bus, and it is always active. The VIDIZMO application will reconnect without a significant impact on the application/workflow.