Categories: BigData, Java, Programming

Exchanges and Queues

I have worked with various message queue systems in the past, but RabbitMQ’s “exchanges” confused me at first. They are actually quite simple:

• RabbitMQ message queues work like in-memory queue types - subscribers race to take messages of the queue, and when one has been removed then it is not seen by any other subscriber to the same queue.
• Message publishers do not write directly to queues; instead they always write to an exchange and the exchange decides which queue(s) to place the message on. Like queues, exchanges also have names.

Each queue is bound to one or more exchanges; an exchange only ever writes messages to queues which are bound to it. When a queue is bound to an exchange, “binding properties” are also specified: a binding routing key and a set of headers. The exact effect of the key and headers depends on the exchange type.

When a publisher writes a message, it actually specifies (exchange-name, message-routing-key, headers, message-body).

An exchange must be one of the following types:

• fanout
• direct
• topic
• headers

Exchanges of type fanout are the simplest: a message written to such an exchange is placed on every queue bound to the exchange. The “binding properties” associated with each queue are completely ignored.

Exchanges of type direct place incoming messages on every queue where the message routing key associated with the published message matches the binding routing key associated with the queue. This is a simple string-equality comparison. The “headers” binding parameter for each queue is ignored.

Exchanges of type topic are a more flexible version of direct. The routing-key string of the published message is split on char “.” to produce a list of tokens. This is then matched against the list of tokens specified in the binding routing key for each queue. The message is written to each queue which “matches”. The binding parameter may include wildcards, eg “sport.*.cycling.#” will match a message routing key of “sport.team.cycling.english.amateur”. This isn’t regular-expression type matching; it reminds me more of “pattern matching” from functional languages - possibly not a coincidence. See the RabbitMQ manuals for a full definition of the matching process.

Exchanges of type headers ignore the routing-key of the incoming message, and instead check for a match between the headers of the published message and the headers binding-parameter for each queue. See the RabbitMQ manuals for the full details.

There is a single inbuilt exchange called the default exchange, whose name is the empty string. This exchange is of type direct and every queue defined in RabbitMQ is automatically bound to the default exchange with binding-parameter routing-key set to the name of the queue. This means that any RabbitMQ client application can place a message on any queue by publishing to the default exchange with routing-key set to queue-name.

Client applications (or system administrators) can define additional exchanges for any desired purpose, and bind queues to them in whatever way is appropriate.

Broadcast-type message patterns, where a single message is received by multiple consumers, is configured in RabbitMQ by defining a separate queue for each consumer and then attaching those queues to the same exchange with either type=fanout or with identical binding-parameters. As each message is written to that exchange, it is then placed on multiple queues simultaneously. Each consumer application then reads from its own queue, at its own pace.

Queues and Clusters

As noted on the RabbitMQ clustering page page:

All data/state required for the operation of a RabbitMQ broker is replicated across all nodes. An exception to this are message queues, which by default reside on one node, though they are visible and reachable from all nodes. To replicate queues across nodes in a cluster, see the documentation on high availability (note that you will need a working cluster first).

As noted on the RabbitMQ High Availability page, mirroring (replication) of queues should be configured via policies.