Introduction
VIDIZMO's architecture is framed around an event-driven design. So, what exactly is Event-Driven Architecture (EDA)?
Overview of Event-Driven Architecture (EDA)
Event-driven architecture is a design framework where events are the catalysts for data flow within a system. In Event-Driven Architecture, there are event producers responsible for generating a continuous stream of events, and event consumers that actively listen for these events. The delivery of events occurs in near real-time, allowing immediate responses from consumers as events unfold.
In the context of VIDIZMO's architecture, the adoption of an Event-Driven approach holds paramount importance. This design choice enhances the system's flexibility and openness.
For a deeper understanding of VIDIZMO's architecture, we recommend exploring our article titled "A Comprehensive Design and Architecture Overview."
VIDIZMO, with its complex event processing requirements, benefits from the smooth integration and communication facilitated by Event-Driven Architecture. The architecture ensures that multiple subsystems efficiently process the same events in real time, minimizing time lag. This real-time processing capability is crucial for VIDIZMO, making event-driven architecture a significant contributor to its successful execution. The inherent scalability and distribution of the architecture further enhance its capabilities.
VIDIZMO's Event-Driven system relies on Azure Service Bus (ASB) as a central component for inter-communication among its services. ASB, a cloud-based broker service from Microsoft, uses the Advanced Message Queuing Protocol (AMQP) to transmit messages. Serving as a messenger, ASB ensures seamless communication between various applications and services within the system.
While VIDIZMO supports various options for the broker service, this article focuses on the Azure Service Bus, showcasing its role as a smart intermediary that fosters smooth communication among different programs and services within the system.
VIDIZMO's Implementation of ASB
Azure Service Bus (ASB) plays a pivotal role in facilitating seamless communication within the VIDIZMO architecture. As a cloud-based broker service provided by Microsoft, Azure Service Bus serves as a central hub for inter-communication between various components and services in the VIDIZMO ecosystem. Acting as a broker service, ASB ensures that different programs and services communicate smoothly, fostering a reliable and organized exchange of information.
To delve deeper into this concept, explore the "Concepts of Azure Bus Service."
How VIDIZMO leverages ASB as a broker service
VIDIZMO operates as an Event-Driven system, employing Azure Service Bus as the intermediary for communication among its component services. Data exchange across various applications and services transpires through messages. Within this framework, the VIDIZMO application initiates an event, broadcasting it to the Azure Service Bus topic, essentially functioning as a publish-subscribe messaging entity. VIDIZMO is responsible for publishing events to this topic, while services subscribe to specific events through configurations known as subscriptions.
Message Queues: Azure Service Bus employs message queues to store and manage messages, ensuring efficient communication between different parts of VIDIZMO. This queuing mechanism helps in organizing and regulating the flow of data.
Azure Service Bus relies on messaging entities such as queues, topics, subscriptions, and rules/actions to form the foundation of its messaging capabilities. Within VIDIZMO, Azure Service Bus is employed to establish Topics, serving as a communication channel between distinct component services.
In this context, subscriptions establish associations between services and particular events. Each service subscribes to the relevant topic, ensuring it only receives messages pertinent to its interest services, representing applications or microservices within VIDIZMO. Subscribed services are recipients of event messages from the topic, and each service processes the event in accordance with its designated functionality.
Advanced Message Queuing Protocol (AMQP): Utilizing AMQP, Azure Service Bus ensures the secure and efficient transmission of messages. This protocol enhances the reliability and interoperability of VIDIZMO's communication infrastructure.
Messaging Pattern Supported by ASB in VIDIZMO
In VIDIZMO, the Azure Service Bus (ASB) supports the Pub/Sub (Publish/Subscribe) messaging pattern. This messaging infrastructure maintains a record of subscriptions, and upon publishing an event, it transfers the event to each subscriber. This mechanism enables VIDIZMO to broadcast messages to multiple consumers concurrently. For instance, when there is a requirement to inform various components about a particular event, the publish-subscribe pattern of Azure Service Bus ensures effective and widespread communication.
Benefits of Azure Service Bus
- Scalability
VIDIZMO is a video management platform that may experience varying workloads based on user demand and video upload/download patterns. Azure Service Bus enables VIDIZMO to scale its messaging infrastructure dynamically based on changing workloads, ensuring smooth operation during peak usage periods.
Azure Service Bus facilitates parallel processing of messages by allowing partitioning of queues (Point-to-point messaging where one message reaches one receiver, ensuring order and reliability) and topics (Publish-subscribe messaging where one message reaches multiple interested subscribers simultaneously). This feature enables VIDIZMO to distribute message processing across various instances, ensuring efficient utilization of resources during peak loads and making it dynamically scalable.
Reliability and High Availability
VIDIZMO's workflows often involve multiple stages, such as video upload, processing, and distribution. Azure Service Bus ensures reliable message delivery, reducing the risk of data loss and ensuring that VIDIZMO's video-related messages are consistently processed.
- Cost-effectiveness
Offers flexible pricing models based on message volume and throughput, providing cost efficiency. Azure Service Bus offers Standard and Premium tiers to cater to your specific requirements to ensure smooth and reliable delivery. To get more details, refer to Service Bus Premium and Standard messaging tiers.
- Integration with Azure ecosystem
Seamlessly integrate with other Azure services, simplifying workflow automation and event-driven architecture within VIDIZMO. Azure Service Bus can easily integrate with other Azure components that VIDIZMO utilizes, such as Azure Blob Storage for video storage and Azure Functions for serverless computing.
Security and Compliance
It adheres to strict security standards, including FIPS 140-2 and ISO 27001, offering peace of mind for security-sensitive VIDIZMO data.
Message Filtering and Routing
Azure Service Bus enables the filtering and routing of messages based on their content. VIDIZMO leverages Service Bus's adaptability to achieve precise message routing and workflow execution.
Shortcomings of Azure Service Bus
- Cost Considerations
Depending on the volume of messages exchanged within VIDIZMO, the cost of using Azure Service Bus may become a consideration. Exploring alternative pricing options or considering self-hosted solutions might be necessary.
Configuration in VIDIZMO
Deploying Azure Service Bus
Before diving into VIDIZMO's broker service setup, let's ensure you have the necessary foundation in place.
Prerequisites:
Azure Subscription: If you don't have one, head over to sign up for a free trial.
Basic Azure Portal knowledge: You'll be creating resources within the Azure portal, so familiarity is helpful.
Steps:
Create a Service Bus Namespace:
Access the Azure portal and search for Service Bus.
Click on "Service Bus" and then "+ Create" to begin creating a new namespace.
Fill in the required details like resource group name, location, and namespace name. Choose a pricing tier that suits your needs.
Click "Create" to bring your namespace to life.
Create a Queue:
Within your newly created namespace, navigate to Queues and click + Queue.
Give your queue a name and leave the other settings at their default values.
Click "Create" to add the queue to your namespace.
Retrieve the Connection String:
Open your namespace and go to Shared Access Policies.
Select the policy named "RootManageSharedAccessKey" and copy the Primary Connection String. This string is crucial for configuring Azure Service Bus in VIDIZMO later as the RootManageSharedAccessKey provides administrative access to your namespace, which is essential for VIDIZMO configuration.
For detailed instructions, refer to the official Microsoft documentation:" Use Azure portal to create a Service Bus namespace and a queue."
Setting up ASB as a broker service in VIDIZMO
The 'Application Configuration' portion in your VIDIZMO app includes the 'Distributed Run Time Config' section, where you would configure the Endpoints needed to establish a connection with the broker service. The application may run into unexpected issues if the Endpoints are not specified correctly. Make sure the correct Connection String is added with all VIDIZMO services in the 'Distributed Run Time Config' section.
Note: This is specific to on-premises VIDIZMO deployments and requires Administrator privileges.
To configure broker service in the VIDIZMO application, please follow the instructions given below:
Steps:
Login to the VIDIZMO application with Administrator privileges.
- Navigate to the Navigation Menu
- Click on the Control Panel
3. Select the Application Configuration
4. Click on the VIDIZMO Runtime Configuration
5. Click 'Edit' to change/update existing configuration.
6. Under Event System, select Azure Service Bus from the drop-down menu.
In the Connection String field, paste the Primary Connection String associated with the RootManageSharedAccessKey policy rule in your Azure Service Bus namespace. Please set the maximum message size (KB) to 0 and configure the compression of messages to occur after reaching a size of 262,144 bytes. This recommendation is advised for optimal service performance.
Considerations
Please consider the following key factors and recommendations for utilizing Azure Service Bus with the VIDIZMO application:
- Throughput
Choose the appropriate pricing tier (Standard or Premium) based on your throughput needs. The Standard tier is for low volumes and test environments, while the Premium tier offers higher capacities, predictable performance, and auto-scaling for production use.
Monitor usage
Keep track of message throughput and latency to ensure your chosen tier meets your application's demands. For further details, you can visit the official documentation" Monitor Azure Service Bus."
Troubleshooting
This section provides troubleshooting tips and recommendations for a few issues that you may see when using Azure Service Bus as the broker service in the VIDIZMO app. If encountering intermittent connectivity challenges, to diagnose and resolve these issues, consider the following steps:
- Verify network connectivity
Ensure the VIDIZMO app server can access the Service Bus namespace by pinging or browsing its URL. Check for firewall or network security restrictions.
- Service Bus Accessibility Check
Ensure the Service Bus namespace is reachable from the VIDIZMO application server. A preliminary test involves browsing or using WGET on 'https://<yournamespace>.servicebus.windows.net/'. This aids in identifying IP filtering, virtual network, or firewall issues commonly encountered in secure network environments.
- Port Accessibility Verification
Confirm that the necessary ports are accessible. VIDIZMO's connectivity relies on ports 443 (HTTPS), 5671 (AMQP), and 9354 (Net Messaging/SBMP). Employ MS TELNET to inspect port status and detect any potential blocking issues on the security system.
- Network Security Assessment
Examine whether organizational network policies might restrict access to Azure Service Bus. Verify that essential ports are not blocked, potentially hindering communication. This ensures VIDIZMO can establish seamless connections with the Service Bus.
- Diagnostic Tool Utilization
In instances of intermittent connectivity, leverage diagnostic tools to assess network health. Employ the PSPING tool to initiate 25 TCP connections per second with the service, gauging connection success/failure rates and TCP latency. Download the PSPING tool for thorough network diagnostics.
Monitor performance metrics
Utilize Azure Monitor to track message throughput, latency, and errors. Identify bottlenecks or resource limitations.
Optimize message size
Minimize message size for faster processing and lower bandwidth usage.
Adjust Service Bus settings
Scale up the Service Bus tier or increase message delivery concurrency if experiencing throttling or high latency.
Issues that may occur with service upgrades/restarts
You may experience the following issues with your VIDIZMO app when there are any Service Bus-related upgrades or service restarts.
Requests may be momentarily throttled.
There may be a drop in incoming messages/requests
The log file may contain error messages
The application may be disconnected for a few seconds
Backend service upgrades and restarts may cause these issues in your VIDIZMO application.
For resolution, the retry policy is already built in Azure Service Bus, and it is always active. The VIDIZMO application will reconnect without a significant impact on the application/workflow.
VIDIZMO Component Services
VIDIZMO uses different component services that run in the background to allow the application to function. The VIDIZMO application may run into issues when one or more services are in a stopped state.
When one or more VIDIZMO services are in a stopped state, it can cause the broker service to stop relaying messages between component services. At this point, you will see the Service Bus message queue being piled up (see the count). To see the message queue under your Azure Service Bus resource, log in to your Azure portal account > go to your Service Bus resource > select the Overview section, and you can view the message queue.
Please make sure the VIDIZMO services are running on the application server. See the below example:
