Run a mock round

At-least-once delivery means duplicates will happen, so a consumer must produce the same result whether it sees a message once or several times.

Techniques:

• Deduplication table: store processed message ids with a unique constraint; skip ids already seen. Use the insert result to detect duplicates atomically.
• Natural idempotency: use deterministic upserts keyed by business identity, such as INSERT ... ON CONFLICT DO UPDATE.
• State-machine guards: only apply a transition if the current state allows it, for example "mark shipped" only from "paid".
• Idempotent side effects: send a notification keyed by an event id, or use conditional writes.

INSERT INTO processed_messages (id) VALUES (:id)
ON CONFLICT (id) DO NOTHING;

Tie deduplication to the same transaction as the state change so the marker and the effect commit together. Keep the deduplication window at least as long as the retry horizon, and design compensations for effects that cannot be undone, such as emails already delivered.

Consumer lag is the gap between the latest produced offset and the committed offset. Rising lag means producers outpace consumers, and left unchecked it grows unbounded and delays downstream data.

Diagnose first: is it a traffic spike, a slow downstream, a hot partition or expensive per-message work? Then:

• Scale consumers up to the partition count; beyond that, add partitions.
• Batch and parallelise processing where ordering allows, and increase fetch sizes.
• Optimise the slow dependency, add caching or bulk writes.
• Apply backpressure to producers: rate limit, buffer with bounded queues, or shed non-critical load.
• Move poison messages to a dead-letter queue so one record cannot stall a partition.
• Alert on lag as an SLI and cap retention so unbounded growth is visible.

lag = latest offset - committed offset

Prefer design that keeps consumers fast and stateless so scale-out actually helps, and load-test the pipeline before peak events.

A dead-letter queue (DLQ) is a separate queue that receives messages a consumer cannot process after exhausting retries. It prevents one poison message from blocking a partition or queue forever and preserves the payload for diagnosis and replay.

Typical setup: a consumer retries a few times with backoff, then routes the message to the DLQ along with metadata such as the original topic, error, attempt count and timestamp.

retry:
  max-attempts: 5
  backoff: exponential
dead-letter:
  queue: orders.dlq

Operate DLQs actively: alert on non-zero depth, inspect and classify failures, fix the bug, then redrive messages after the fix. Without a redrive process a DLQ becomes a silent graveyard. Distinguish transient failures (network, downstream outage) that deserve retries from permanent ones (malformed schema, unknown type) that should skip retries. Also cap message retention and age.

Retrying immediately in a tight loop amplifies outages. Instead:

• Classify errors: retry transient ones (timeouts, 503, deadlocks) and stop retrying permanent ones (schema violations, 400-class errors).
• Retry with exponential backoff and jitter, and cap attempts.
• Use delayed retry queues or scheduled redelivery so the broker, not the consumer, holds the delay.
• After the cap, route to a dead-letter queue with the error and attempt count, and alert.

max-attempts: 5
backoff: "1s, 5s, 30s, 5m"

Ensure the consumer is idempotent, since retries can process the same message twice. Watch out for the head-of-line problem: a message that always fails can block a partition if retried in place, so move it aside quickly. Also add circuit breakers for downstream calls so retries do not pile onto a service that is already failing, and load-test the retry path.

• At-most-once: the broker may deliver a message once or not at all. It never duplicates, so it can lose data. Achieved by acknowledging before processing. Suitable for metrics or non-critical telemetry.
• At-least-once: the broker retries until acknowledged, so nothing is lost, but duplicates are possible. This is the practical default; consumers must be idempotent or deduplicate.
• Exactly-once: each message affects state once. True end-to-end exactly-once across independent systems is extremely hard; Kafka achieves it within its own ecosystem using an idempotent producer and transactions, but once data leaves Kafka for an external system, exactly-once becomes at-least-once plus idempotent writes.

ack before work -> at-most-once
ack after work  -> at-least-once (retry on crash)

Most teams should design for at-least-once and make consumers idempotent, rather than paying the complexity cost of chasing true exactly-once semantics.

Ordering is usually per partition or per queue, not global.

• Kafka orders records within a partition only. Records with the same key go to the same partition, preserving per-key order. Across partitions there is no order guarantee.
• RabbitMQ keeps FIFO order per queue when a single consumer processes it; multiple competing consumers can interleave messages.
• SQS standard queues offer best-effort ordering; FIFO queues provide ordering within a message group id.

key = customerId -> same partition -> ordered per customer

If you need strict order, ensure single-consumer processing per partition or queue, and avoid parallel retries that can reorder. A common failure is processing per-key events across multiple partitions and assuming they arrive in order. When global order is required, it is usually a sign to serialise on one key or to use timestamps and conflict resolution rather than depending on transport order.

A consumer group is a set of consumers that cooperatively consume a topic. Kafka assigns each partition to exactly one consumer in the group, so the group as a whole processes every record once, while multiple consumers share the load.

group "billing":  consumer A -> partition 0
                  consumer B -> partition 1
                  consumer C -> partition 2

Scaling works up to the number of partitions: a fourth consumer would sit idle. If consumers leave, crash or join, Kafka triggers a rebalance and reassigns partitions. Cooperative rebalancing reduces stop-the-world pauses compared with the older eager protocol.

Different groups are independent; each maintains its own offsets, so the same topic can feed billing, analytics and search independently. Key pitfalls: keep processing bounded so a member is not evicted for missing heartbeats, avoid blocking the poll loop for long periods, and commit offsets after successful processing to preserve at-least-once semantics.

Producers publish to an exchange, which routes to queues via bindings. The exchange type decides the routing:

• Direct: routes by exact routing key match. Common for task queues.
• Fanout: ignores the key and copies to every bound queue, which is pub/sub.
• Topic: matches routing keys against wildcard patterns where * is one word and # is zero or more words, for example orders.# or *.error.
• Headers: routes on message header attributes rather than the routing key.

exchange (topic) --binding "order.created"--> queue A
                 --binding "order.*"       --> queue B

A queue can be bound to multiple exchanges, and unroutable messages are dropped unless a mandatory flag or an alternate exchange is configured. Default exchange routes directly to a queue named by the routing key. Choose topic exchanges for flexible event routing, fanout for broadcasting and direct for simple work distribution.

Kafka combines two features:

• Idempotent producer: each producer gets a producer id and sequence numbers. The broker deduplicates retries within a session, so a network retry does not append the same record twice.
• Transactions: the producer atomically writes to multiple partitions and commits consumer offsets in one transaction. Consumers with isolation.level=read_committed only see committed data and do not read aborted or uncommitted records.

enable.idempotence=true
transactional.id=order-processor-1
isolation.level=read_committed

This gives exactly-once processing inside a read-process-write Kafka Streams pipeline. It does not extend to external systems: writing to a database or calling an API can still duplicate or partially commit. For those, use the outbox pattern with an idempotent sink or a two-phase commit-free design.

The cost is higher latency and complexity, so use transactions only where duplicates are genuinely unacceptable and simpler idempotent consumers cannot solve the problem.