Queue-Based Asynchronous Architecture: Backpressure and Delivery Guarantees
Asynchronous architecture under high traffic: queue vs event log, backpressure design, the reality of "exactly-once" and idempotent consumers, ordering, DLQs and poison message handling.
What does asynchronous architecture solve under high traffic?
It decouples time: producer and consumer no longer must be up, fast and healthy at the same moment. Peak traffic is absorbed in the queue, heavy work is digested in the background, transient failures close through retries. But the power is not free: delivery guarantees, ordering and failure handling become contracts your architecture must design explicitly. This article builds those contracts.
Queue or event log?
Dimension | Message queue (RabbitMQ-style) | Event log (Kafka-style) |
|---|---|---|
Model | Work distribution: message processed, then gone | Record stream: events retained, re-readable |
Strengths | Task queues, RPC-like flows, flexible routing | High volume, many consumers, replay |
Ordering | Per queue; weakens with competing consumers | Strong guarantee within a partition |
Typical use | Email, PDF generation, integration tasks | Event distribution, CDC, stream processing, audit trail |
The decision follows the business model: "do this work once and be done" is queue territory; "make this event the truth for multiple systems" is log territory. Architectures where both coexist are normal; forcing one into the other's job is what goes wrong.
Backpressure: the system's braking system
The most-skipped part of async design is backpressure: what a growing queue does to the rest of the system when consumers slow down must be designed. The toolset has four parts: bounded queue depth (an unbounded queue is a postponed crash — invisible until memory/disk fills); producer-side throttling (slowing production or rejecting low-priority work past a threshold); a consumer prefetch limit (never pulling more messages than the consumer can digest); and a queue-age alert (watch the oldest message's wait time, not just depth). In a campaign scenario the queue is the buffer — but if the buffer's overflow behavior was never written down, the crash is merely postponed.
Delivery guarantees: 'exactly-once' marketing vs reality
The practical default in distributed systems is at-least-once: the message arrives at least once — meaning sometimes twice. The 'exactly-once' label describes special mechanisms valid within specific boundaries (inside one streaming platform, for instance); in an end-to-end flow that writes to your database, your mail server, your ERP, there is no magic. Hence the invariant rule: consumers are written idempotent — a business key (order number, message ID) answers "have I processed this before?", and a second arrival returns the existing result without new effects. The producer side has its twin: the outbox pattern, binding publication atomically to the database transaction. Idempotent consumer + outbox producer are the load-bearing columns of async architecture.
Ordering: ask for it per key, not globally
Global ordering contradicts scale — it means a single-queue, single-consumer bottleneck. The correct target is per-key ordering: one order's events processed in sequence, different orders flowing in parallel. On the log side this comes naturally via the partition key (orderId); on the queue side it is built with per-key exclusive consumers or session mechanisms. The design question is: "within which key is order mandatory?" — the business answers it, infrastructure implements it.
Failure handling: retries, DLQs and poison messages
Transient failures (network, locks, a briefly busy target) are handled with exponential backoff + jitter retries; permanent failures (corrupt data, business rule rejection) do not heal with retries — after the attempt limit, the message lands in the dead-letter queue (DLQ). For poison messages (records that crash the consumer on every attempt) that limit is vital; without it, one message locks the whole line. A DLQ is not a trash bin but an operations dashboard: its size is wired to alerts, contents are inspectable, and messages replay safely after fixes — risk-free thanks to idempotent consumers. Correlation IDs are logged end to end; "where is this message?" is answered in minutes.
Business impact: the queue is the customer experience's shock absorber
The commercial value reads in two places. First, peak endurance: the order-taking line decouples from integration speed — even if the ERP slows during a campaign, sales continue and the queue digests. Second, operational calm: transient failures become a morning report instead of a night page, because retry and DLQ mechanics already managed the incident. Both values have crisp measures: queue age on target, DLQ empty in normal operation.
Frequently asked questions
Should everything be async?
No: queries where the user awaits the answer, and critical writes needing strong consistency, stay synchronous. Async is for work that is deferrable, retryable and peak-sensitive.
Is Kafka always better?
No — the power is in the fit. For task distribution and flexible routing a classic queue is simple and sufficient; for high-volume event distribution and replay needs, log architecture leads.
How do message schemas evolve?
Contracts are versioned: adding fields is free, removing/retyping requires a new version; consumers ignore unknown fields. A schema registry automates the discipline.
How should queue latency show to users?
With honest state design: visible "received → processing → done" steps, completion notifications, and estimated times where needed. Experience is managing the delay, not hiding it.
Async architecture checklist
Sync/async split written in business semantics
Queue vs event log chosen by usage pattern
Queue depth bounded; producer throttling threshold defined
Consumers idempotent; producers atomic via outbox
Ordering needs defined and implemented per key
Backoff+jitter retries, attempt limits and DLQ + alerts in place
Queue age and DLQ size dashboarded; replay procedure written
SSH Yazılım designs queue-based architectures and drives async transformation of existing systems end to end. Let us build your peak endurance together.