Asynchronous messaging separates the time a producer creates work from the time a consumer processes it. That can absorb bursts, isolate temporary outages, and let several consumers react independently. It also introduces delivery states that a synchronous call can hide.

The central distinction is simple: a broker accepting a message is not the same as the business operation completing.

Draw the delivery boundaries

A producer creates a message and sends it to a broker. Depending on the product and configuration, the producer may receive a confirmation that the broker accepted the message under a defined durability or replication contract. That confirmation does not say that a consumer has seen the message, that a database transaction has committed, or that a user-visible result exists.

A consumer later receives the message, performs work, and acknowledges progress. In a queue, acknowledgement commonly allows the broker to remove or advance past the delivery. In a retained event stream, the record remains according to retention policy while a consumer group records or commits its position.

Between those boundaries, several failures are possible:

  • the producer times out even though the broker accepted the message;
  • the broker accepts but loses data outside the configured durability guarantee;
  • the consumer receives a message and fails before completing work;
  • the consumer completes work but fails before acknowledgement;
  • acknowledgement succeeds but a downstream side effect is still pending.

Each transition needs an owner and a recovery rule. Describing the system as “asynchronous” does not define any of them.

Producer confirmation needs an application outcome

When a producer cannot tell whether a send succeeded, it faces an ambiguous result. Retrying may create a duplicate; not retrying may lose the intended event.

Assign a stable message identifier or operation identifier before the first send. The producer can retry with the same identity, and consumers can recognize repeated delivery. If the message represents a database change, avoid a fragile sequence where the application commits the database transaction and then separately publishes. A crash between those actions leaves committed state without its event.

The transactional outbox pattern addresses that gap by writing the business change and an outgoing message record in one local database transaction. A relay publishes outbox records and marks progress. The relay can still publish more than once, so consumers remain idempotent, but the system has a durable recovery source for committed events.

The reverse order—publish first, then commit local state—has a different gap: consumers can observe an event for a transaction that ultimately fails. Choose a protocol that represents the business truth rather than hoping two independent operations behave atomically.

Acknowledgement marks a chosen point

Consumer acknowledgement should occur only after the work covered by the acknowledgement contract is complete. Acknowledging on receipt reduces redelivery but can lose work if the consumer fails after the acknowledgement. Acknowledging after work protects against that loss but permits duplicate processing when work completes and the acknowledgement is lost.

That second case is normal in at-least-once delivery. The broker cannot inspect a consumer’s database to know that the business transaction committed. It sees only whether acknowledgement arrived.

Keep the acknowledgement timeout or visibility timeout longer than expected processing, or renew it through a supported mechanism. An interval that is too short can cause concurrent redelivery while the first consumer still works. An interval that is excessively long can delay recovery from a dead consumer. Long-running jobs may be better represented as explicit job state with leases and checkpoints.

Prefetch or batch size also affects fairness and recovery. A consumer that reserves a large batch can hold work away from others, and its failure may release many messages at once. Tune with realistic processing duration and memory cost.

Retries require classification and limits

Not every error should be retried. A temporary dependency timeout may recover. A message that violates a schema or references permanently invalid data will probably fail again unchanged.

Classify failures into retryable, non-retryable, and ambiguous. Apply bounded attempts, delay, and backoff for retryable failures. Add jitter when many consumers might retry together. Keep retries inside the business deadline; succeeding hours later may be wrong even if technically possible.

Immediate requeue can create a tight poison-message loop that consumes capacity while making no progress. Delayed retry queues or scheduled redelivery provide space for dependencies to recover. After the attempt or age limit, move the message to a dead-letter path or terminal failure state with enough context for diagnosis.

A dead-letter queue is not a solution by itself. Define who monitors it, how messages are inspected safely, whether replay is permitted, and how a repaired message avoids duplicating side effects already completed.

Idempotency is business-specific

An idempotent consumer can process the same logical message more than once without changing the intended final outcome after the first successful processing. This is not achieved merely by using an idempotent HTTP verb or checking a broker delivery tag.

Use the stable operation identifier in the consumer’s transactional boundary. A database-backed consumer might insert that identifier into a table with a uniqueness constraint in the same transaction as the business update. If the identifier already exists, the consumer knows the committed operation was previously applied.

The stored result may matter too. If a duplicate request needs the original response, retain enough outcome data to return or reconstruct it. Define retention for deduplication records based on the maximum replay horizon; deleting them while old messages remain replayable reopens duplication risk.

External side effects complicate idempotency. Sending email, charging a provider, or calling another system may not share the local transaction. Pass an idempotency key where supported, persist intent and outcome, and reconcile ambiguous responses. “Exactly once” at one broker boundary does not automatically make the end-to-end business process exactly once.

Ordering is scoped, not universal

Queues and streams often preserve order only within a queue, partition, key, or consumer arrangement. Parallelism changes observation order. A failed message sent to delayed retry can allow later messages to complete first.

If order matters per account, order, or aggregate, partition by that key and ensure one effective writer for the ordered sequence. This can create hot partitions when one key dominates. Global ordering usually limits scalability and availability, so require it only when the domain truly needs it.

Consumers should also handle stale or out-of-order events. Include an entity version or sequence when appropriate, reject impossible transitions, and make event semantics clear. A timestamp from different machines is not always a safe total order.

Queues and retained streams serve different consumption models

A traditional work queue commonly distributes messages so one consumer instance handles each delivery, with routing and acknowledgement controlling progress. It fits commands and jobs where work should be claimed and completed.

A retained event stream keeps records for a configured time, size, or compaction policy. Independent consumer groups track their own positions and can replay retained history. This fits event distribution, derived views, and independent consumers, but replay requires compatible schemas and idempotent side effects.

The categories overlap across products. Choose from routing, replay, ordering, retention, throughput, latency, operating skill, and recovery needs—not from a generic queue-versus-stream slogan.

Operate the backlog as a bounded resource

Monitor message age as well as count. A small backlog of very old messages can be more important than a large backlog that drains quickly. Track publish rate, completion rate, retry rate, dead letters, consumer lag, broker storage, acknowledgement time, and end-to-end business completion.

Set storage and retention limits intentionally. If producers can outpace consumers indefinitely, the broker eventually becomes the incident. Decide whether to reject new work, shed lower-priority traffic, increase consumers within downstream capacity, or degrade the feature.

An architecture review can map these delivery boundaries to database transactions, external side effects, and recovery ownership. Reliable messaging does not come from assuming duplicates will never happen. It comes from making acceptance, completion, acknowledgement, retry, and idempotency explicit all the way through the business operation.