Message Queue

A producer sends a message to a broker, the broker stores it in a named queue, and consumers take messages off the queue and process them. The producer only waits for the broker to accept the message, not for the work to finish, which is what lets the two sides run at different speeds and fail independently.

Reliability comes from acknowledgements. In RabbitMQ a consumer acks a message once it has finished with it; if the consumer dies (channel or connection closed, TCP connection lost) without acking, RabbitMQ re-queues the message for another worker. Amazon SQS uses a visibility timeout instead: a received message stays in the queue but is hidden from other consumers, 30 seconds by default, and becomes visible again if it is not deleted before the timeout expires.

A crash can happen after the work is done but before the ack or delete, so queues like these deliver at least once, and SQS documents that a message can be delivered more than once. Consumers should therefore be idempotent, for example by recording processed message IDs. Messages that keep failing are usually moved to a dead-letter queue for inspection rather than retried forever.

With several consumers on one queue the work is shared. RabbitMQ dispatches round-robin by default; setting a prefetch count of 1 tells it not to give a worker a new message until that worker has acked the previous one, so slow jobs do not pile up on one worker.

A RabbitMQ worker in Python (pika), from the official "Work Queues" tutorial. It acks only after the work is done.

import pika, time

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)

def callback(ch, method, properties, body):
    print(f" [x] Received {body.decode()}")
    time.sleep(body.count(b'.'))          # simulate work: one second per dot
    print(" [x] Done")
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_qos(prefetch_count=1)        # fair dispatch: one unacked job per worker
channel.basic_consume(queue='task_queue', on_message_callback=callback)
channel.start_consuming()

Start two workers and publish a few messages. Kill one worker while it is sleeping on a long message: because it never sent basic_ack, the broker hands that message to the surviving worker instead of losing it.

What is the difference between a message queue and an event streaming platform like Kafka?

In a classic queue a message is removed once a consumer acknowledges or deletes it, and each message is normally handled by one consumer. Kafka stores events in topics and does not delete them after consumption; retention is set per topic and old events are discarded only after that period. Many independent consumers can therefore read, and re-read, the same stream, and Kafka preserves order within each partition.

Is a message queue the same as publish/subscribe?

Not quite. In a queue, consumers compete: each message goes to one of them. In publish/subscribe, every subscriber receives its own copy of each message. Many brokers support both patterns, so the difference is in how the queue or topic is configured rather than in the product.