Pentaho Data Integration
InstallationBusiness AnalyticsCToolsData CatalogData QualityLLMs
  • Overview
    • Pentaho Data Integration ..
  • Data Integration
    • Getting Started
      • Configuring PDI UI
      • KETTLE Variables
    • Concepts & Terminolgy
      • Hello World
      • Logging
      • Error Handling
    • Data Sources
      • Flat Files
        • Text
          • Text File Input
          • Text File Output
        • Excel
          • Excel Writer
        • XML
          • Read XML
        • JSON
          • Read JSON
      • Databases
        • CRUID
          • Database Connections
          • Create DB
          • Read DB
          • Update DB
          • Insert / Update DB
          • Delete DB
        • SCDs
          • SCDs
      • Object Stores
        • MinIO
      • SMB
      • Big Data
        • Hadoop
          • Apache Hadoop
    • Enrich Data
      • Merge
        • Merge Streams
        • Merge Rows (diff)
      • Joins
        • Cross Join
        • Merge Join
        • Database Join
        • XML Join
      • Lookups
        • Database Lookups
      • Scripting
        • Formula
        • Modified JavaScript Value
        • User Defined Java Class
    • Enterprise Solution
      • Jobs
        • Job - Hello World
        • Backward Chaining
        • Parallel
      • Parameters & Variables
        • Parameters
        • Variables
      • Scalability
        • Run Configurations
        • Partition
      • Monitoring & Scheduling
        • Monitoring & Scheduling
      • Logging
        • Logging
      • Dockmaker
        • BA & DI Servers
      • Metadata Injection
        • MDI
    • Plugins
      • Hierarchical Data Type
  • Use Cases
    • Streaming Data
      • MQTT
        • Mosquitto
        • HiveMQ
      • AMQP
        • RabbitMQ
      • Kafka
        • Kafka
    • Machine Learning
      • Prerequiste Tasks
      • AutoML
      • Credit Card
    • RESTful API
    • Jenkins
    • GenAI
  • Reference
    • Page 1
Powered by GitBook
On this page
  1. Use Cases
  2. Streaming Data

AMQP

Advanced Message Queuing Protocol (AMQP) is an open source published standard for asynchronous messaging by wire ..

PreviousHiveMQNextRabbitMQ

Last updated 1 month ago

Advanced Message Queuing Protocol

AMQP (Advanced Message Queuing Protocol) brokers facilitate high-performance, reliable, and scalable messaging services between applications. This makes them ideal for scenarios needing:

  • Asynchronous Communication: Decoupling producers and consumers, allowing them to communicate without waiting for responses, enhancing application responsiveness.

  • System Decoupling: Applications communicate through messages rather than direct method calls, improving modularity and scalability.

  • Resilience and Fault Tolerance: Messages can be queued, ensuring no loss of information on temporary failures, contributing to system robustness.

  • Load Balancing: Efficient distribution of message processing across multiple consumer instances, optimizing resource utilization and throughput.

  • Distributed Systems Communication: Simplifying communication patterns in microservices or distributed architectures by providing a unified messaging platform.

AMQP brokers, such as RabbitMQ, offer a standardized, open solution to address these requirements, making them a cornerstone in modern distributed application architectures.

Exchanges

Messages are not published directly to a queue; instead, the producer sends messages to an exchange. An exchange is responsible for routing the messages to different queues with the help of bindings and routing keys.

A binding is a link between a queue and an exchange.

The routing key is a message attribute the exchange looks at when deciding how to route the message to queues (depending on exchange type).

Exchanges, connections, and queues can be configured with parameters such as durable, temporary, and auto delete upon creation. Durable exchanges survive server restarts and last until they are explicitly deleted. Temporary exchanges exist until RabbitMQ is shut down. Auto-deleted exchanges are removed once the last bound object is unbound from the exchange.

In RabbitMQ, there are four different types of exchanges that route the message differently using different parameters and bindings setups. Clients can create their own exchanges or use the predefined default exchanges which are created when the server starts for the first time.

This is the default exchange type and it just broadcasts all the messages it receives to all the queues it knows. Above example code shows how to define this exchange type and if exchange type is not defined then by default it will be fanout. In fanout exchange type headers, binding or routing keys will be ignored even if it is provided and messages will be published to all the available queues.

Fanout exchanges can be useful when the same message needs to be sent to one or more queues with consumers one at a time based on default Round Robin method. Hence there is a possibility that the message 11 will be received by Consumer 2 and message 22 will be received by consumer 2.

Queues

In RabbitMQ, queues are central structures where messages are stored for a consuming application or worker to process them. Messages flow into the queue from producers and wait there until a consumer retrieves and processes them, allowing for asynchronous communication and workload decoupling between producers and consumers.

Queues have several important characteristics:

  • Durability: Queues can be durable or transient. Durable queues are saved on disk and survive broker restarts, whereas transient queues do not.

  • Exclusivity: An exclusive queue is used by only one connection and is deleted when that connection closes.

  • Auto-delete: An auto-delete queue will delete itself when the number of consumers drops to zero.

RabbitMQ allows for the dynamic creation of queues via client libraries, and these queues can be configured with various parameters (e.g., durability, exclusivity, auto-deletion) depending upon the application's requirements. Messages in a queue are typically consumed in a FIFO (first in, first out) manner, although RabbitMQ supports various other message dispatching strategies.

Workshops

RabbitMQ is a widely adopted open-source AMQP broker, renowned for its performance, reliability, and scalability. It enables asynchronous communication, system decoupling, resilience, load balancing, and seamless distributed system integration, making it an essential component in modern application architectures.

RabbitMQ provides a flexible routing model, multiple messaging protocols, and a broad range of client libraries, facilitating the development of complex messaging solutions for various computing environments.

RabbitMQ
AMQP Model
Exchanges
Fanout
Queues