The Fatal Flaw of Fire-and-Forget

When building asynchronous web applications, developers frequently look for quick shortcuts to offload long-running operations—such as sending transactional receipts, encoding user media, or syncing customer data to third-party CRMs:

# Danger: Naive fire-and-forget in-memory coroutine
asyncio.create_task(process_billing(user_id))

This naive pattern fails disastrously in production:

• If the application container restarts, redeploys, or gets killed by an Out-Of-Memory (OOM) signal, the pending coroutine is instantly annihilated without leaving an audit trail.
• If a downstream external API suffers a temporary 503 outage, the job fails permanently because there is no durable retry schedule.
• Engineering teams have zero visibility into backlog depth, latency percentiles, or failure rates.

---

1. At-Least-Once Delivery: The Foundation of Reliability

In distributed systems subject to network partitions and hardware failures, true Exactly-Once Delivery is physically impossible without two-phase commit overhead. Resilient backends embrace At-Least-Once Delivery:

1. Atomic In-Flight Leasing: When a worker consumes a task from the broker, the task is not deleted; it is atomically shifted into a leased in-flight queue.
2. Crash-Recovery Reapers: If a worker process abruptly dies midway through execution, a watchdog monitor detects that the worker’s heartbeat expired. The task is reclaimed and re-queued for another healthy worker to pick up.
3. Safe Deletion: The task is only purged from the persistence broker once the worker explicitly commits a successful execution acknowledgement (ACK).

---

2. Idempotency: The Antidote to Duplication

Because network retries and worker crashes mean that tasks can and will be delivered more than once, every worker handler must be strictly idempotent: executing the same job five times must yield the exact same end state as executing it once.

Practical Implementation: Idempotency Keys

Require every task payload to carry an immutable idempotency_key:

async def process_payment(order_id: int, amount_cents: int, idempotency_key: str):
    async with db.transaction():
        # 1. Atomic lookup: Has this operation already succeeded?
        existing_tx = await db.fetch_payment(idempotency_key)
        if existing_tx:
            logger.info("Payment already finalized, skipping duplicate execution", 
                        extra={"idempotency_key": idempotency_key})
            return existing_tx

        # 2. Reserve the transaction in pending state
        await db.create_pending_payment(idempotency_key, order_id)

    # 3. Call external payment provider using the same key
    charge_result = await stripe.charge(amount_cents, idempotency_key=idempotency_key)
    
    # 4. Finalize state
    await db.mark_payment_completed(idempotency_key, charge_result.id)

If a network timeout occurs during response delivery and the worker retries the job, the second execution safely recognizes the existing transaction, preventing duplicate charges to the customer.

---

3. Exponential Backoff with Jitter and Dead-Letter Queues (DLQ)

When an external provider goes down, naive linear retries (retrying every 1 second) trigger a self-inflicted Distributed Denial of Service (DDoS) against the recovering service.

The Production Retry Formula:

• Exponential Backoff: Multiply retry delays exponentially ($1s \rightarrow 2s \rightarrow 4s \rightarrow 8s$).
• Full Jitter: Add pseudorandom noise to the delay to break up synchrony across workers and prevent thundering-herd spikes.
• Dead-Letter Queue (DLQ): Once a task exhausts its configured retry budget (e.g. 5 attempts), shunt it to an isolated DLQ. This keeps the primary processing pipeline fast, prevents queue starvation, and gives on-call engineers a safe, durable place to inspect payloads and replay them once underlying bugs are resolved.

---

Architectural Rules

Concern	Production Strategy
Delivery Model	At-Least-Once via atomic broker leasing (BLMOVE / Streams)
Duplicate Safety	Strict idempotency keys backed by relational unique constraints
Failure Recovery	Exponential backoff with full jitter
Poison Pill Messages	Shunt to Dead-Letter Queue (DLQ) with automated alerting