Retry and Timeout

Context

A remote call is not like a local function call. It travels over a network that drops packets, through load balancers that occasionally reset connections, to a service that might be in the middle of a rolling deploy or briefly garbage-collecting. Many of these problems are transient — they last milliseconds and the very next attempt would succeed. A connection reset, a momentary 503 while a pod restarts, a brief network hiccup: none of these means the dependency is truly broken.

Problem

Two failure modes hide inside every remote call. The first is the unbounded wait: a call with no time limit can hang for the entire duration of a TCP timeout — minutes, on some defaults — while a thread sits idle holding a connection. The second is the transient blip: a single attempt fails for a reason that has already passed, and giving up entirely turns a recoverable hiccup into a user-visible error.

But naive fixes make things worse. Retrying immediately, without delay, hammers a recovering dependency. Retrying in lockstep across thousands of clients creates a thundering herd — a synchronized wave of retries that all arrive at the same instant and knock the dependency back over. And retrying a non-idempotent operation — “charge this card”, “ship this order” — can execute it twice.

So the forces are: you want to recover from blips automatically, you must never wait forever, you must not amplify load on a struggling dependency, and you must not duplicate side effects.

Solution

Combine two disciplines on every remote call.

Timeout. Put a hard upper bound on how long a single attempt may take. When the budget elapses, the attempt is cancelled and counted as a failure. This caps the latency a caller can ever experience and frees the resource immediately.

Retry with exponential backoff and jitter. On a transient failure, try again — but wait longer between each attempt (the delay grows exponentially: roughly base, base × 2, base × 4, …) and add a random jitter to each delay so concurrent clients spread out instead of retrying in unison. Cap the number of attempts so failure is eventually surfaced, and only retry operations that are idempotent — running them twice has the same effect as running them once.

sequenceDiagram
  participant C as Caller
  participant D as Dependency
  C->>D: attempt 1 (timeout 1s)
  Note over C,D: no response within 1s
  C--xC: timeout fires, attempt cancelled
  Note over C: wait ~0.5s (base + jitter)
  C->>D: attempt 2 (timeout 1s)
  D--xC: 503 transient
  Note over C: wait ~1.0s (base*2 + jitter)
  C->>D: attempt 3 (timeout 1s)
  D-->>C: 200 OK

Timeout bounds each attempt; retries wait progressively longer with jitter until one succeeds or the budget runs out

Example

Here is a helper that runs an operation with a per-attempt timeout and retries transient failures using exponential backoff plus jitter. It assumes the operation is idempotent. Each example is self-contained and idiomatic to its language.

TypeScript

interface RetryOptions {
  attempts: number;
  timeoutMs: number;
  baseDelayMs: number;
  isTransient: (err: unknown) => boolean;
}

function withTimeout<T>(fn: (signal: AbortSignal) => Promise<T>, ms: number): Promise<T> {
  const ctrl = new AbortController();
  const timer = setTimeout(() => ctrl.abort(), ms);
  return fn(ctrl.signal).finally(() => clearTimeout(timer));
}

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

async function retry<T>(
  op: (signal: AbortSignal) => Promise<T>,
  opts: RetryOptions,
): Promise<T> {
  let lastErr: unknown;
  for (let i = 0; i < opts.attempts; i++) {
    try {
      return await withTimeout(op, opts.timeoutMs);
    } catch (err) {
      lastErr = err;
      if (!opts.isTransient(err) || i === opts.attempts - 1) throw err;
      // Exponential backoff with full jitter.
      const ceiling = opts.baseDelayMs * 2 ** i;
      await sleep(Math.random() * ceiling);
    }
  }
  throw lastErr;
}

Python

import asyncio
import random
from typing import Awaitable, Callable, TypeVar

T = TypeVar("T")

async def retry(
    op: Callable[[], Awaitable[T]],
    *,
    attempts: int,
    timeout_s: float,
    base_delay_s: float,
    is_transient: Callable[[Exception], bool],
) -> T:
    last_err: Exception | None = None
    for i in range(attempts):
        try:
            return await asyncio.wait_for(op(), timeout=timeout_s)
        except (asyncio.TimeoutError, Exception) as err:
            last_err = err
            transient = isinstance(err, asyncio.TimeoutError) or is_transient(err)
            if not transient or i == attempts - 1:
                raise
            # Exponential backoff with full jitter.
            ceiling = base_delay_s * (2 ** i)
            await asyncio.sleep(random.uniform(0, ceiling))
    assert last_err is not None
    raise last_err

Go

type RetryOptions struct {
    Attempts    int
    Timeout     time.Duration
    BaseDelay   time.Duration
    IsTransient func(error) bool
}

func Retry[T any](
    ctx context.Context,
    op func(context.Context) (T, error),
    opts RetryOptions,
) (T, error) {
    var zero T
    var lastErr error
    for i := 0; i < opts.Attempts; i++ {
        attemptCtx, cancel := context.WithTimeout(ctx, opts.Timeout)
        v, err := op(attemptCtx)
        cancel()
        if err == nil {
            return v, nil
        }
        lastErr = err
        if !opts.IsTransient(err) || i == opts.Attempts-1 {
            return zero, err
        }
        // Exponential backoff with full jitter.
        ceiling := opts.BaseDelay * (1 << i)
        jitter := time.Duration(rand.Int63n(int64(ceiling)))
        select {
        case <-time.After(jitter):
        case <-ctx.Done():
            return zero, ctx.Err()
        }
    }
    return zero, lastErr
}

Rust

use std::future::Future;
use std::time::Duration;
use rand::Rng;
use tokio::time::{sleep, timeout};

pub struct RetryOptions {
    pub attempts: u32,
    pub timeout: Duration,
    pub base_delay: Duration,
}

pub async fn retry<T, E, F, Fut>(
    opts: &RetryOptions,
    mut op: F,
    is_transient: impl Fn(&E) -> bool,
) -> Result<T, E>
where
    F: FnMut() -> Fut,
    Fut: Future<Output = Result<T, E>>,
    E: From<&'static str>,
{
    let mut last_err: Option<E> = None;
    for i in 0..opts.attempts {
        match timeout(opts.timeout, op()).await {
            Ok(Ok(v)) => return Ok(v),
            Ok(Err(e)) => {
                let transient = is_transient(&e);
                last_err = Some(e);
                if !transient || i == opts.attempts - 1 {
                    return Err(last_err.unwrap());
                }
            }
            Err(_) => last_err = Some(E::from("attempt timed out")),
        }
        // Exponential backoff with full jitter.
        let ceiling = opts.base_delay * 2u32.pow(i);
        let jitter = rand::thread_rng().gen_range(0..=ceiling.as_millis() as u64);
        sleep(Duration::from_millis(jitter)).await;
    }
    Err(last_err.unwrap())
}

Resulting context

What you gain:

• Bounded latency. Because every attempt has a timeout, no call hangs forever, and a caller knows the worst-case time it can spend (roughly attempts times the timeout plus the backoff delays).
• Automatic recovery from blips. Transient resets, brief 503s, and momentary network hiccups heal themselves without surfacing an error to the user.
• Spread-out load. Jitter breaks the synchronization that creates thundering herds, so a recovering dependency is not knocked over by a wall of simultaneous retries.

What it costs you:

• Risk of retry storms. Retries multiply load. If a dependency is genuinely overloaded — not just blipping — naive retries add fuel to the fire. This is exactly why retry must be paired with a circuit breaker: once the breaker trips, retries stop entirely until the dependency recovers.
• Idempotency is mandatory. Retrying a non-idempotent operation can execute its side effect twice. Either restrict retries to safe operations (reads, idempotent writes) or make writes idempotent with an idempotency key so a duplicate is detected and ignored.
• Tuning the budget. Too many attempts or too long a timeout makes a failing call expensive for the caller; too few makes the retry pointless. The total time budget must fit inside the caller’s own deadline.

Related patterns

• Circuit Breaker — the essential partner that stops retries from becoming a storm.
• Bulkhead — bounds the resources retries can consume while they wait.
• Saga — its compensating transactions must be idempotent precisely because they may be retried.

Benefit	Trade-off
Handles transient failures automatically — a network blip doesn’t reach the user	Retries can add load to a dependency that is already overloaded
Timeouts prevent thread leaks from a slow dependency	Too short a timeout fails an operation that would have succeeded
Exponential backoff reduces thundering herd	Retry storms — many clients retrying at once can look like a DDoS on the dependency
Jitter spreads retries out instead of bunching them	Non-idempotent operations need extra care against duplicate side effects

Common mistakes

Retrying a non-idempotent operation — retrying POST /payment without checking idempotency. Symptoms:

• The user gets charged twice by the retry.
• Any operation with a side effect needs an idempotency key.

Retrying without backoff — retrying immediately, over and over. Symptoms:

• 100 requests land on an already-overloaded service in one second.
• This slows the service’s recovery or pushes it further into failure.
• Always use exponential backoff plus jitter.

💡 Real-world examples

AWS SDK: The standard retry mode makes at most 3 total attempts (the initial call plus 2 retries), using truncated exponential backoff with randomized jitter and a MAX_BACKOFF of roughly 20 seconds for most SDKs — there is no fixed millisecond sequence, because the formula itself is randomized (AWS SDKs and Tools: Retry behavior).

Google Cloud: its docs recommend truncated exponential backoff with jitter for all APIs, with a typical maximum backoff of 32–64 seconds to prevent retry storms (Google Cloud: Retry strategy).

🧩 Only retry what is safe to repeat

A retry can cause the same request to be processed more than once — the first attempt may have actually succeeded on the server even though the response never reached you. For reads this is harmless. For writes it is not: a retried “charge card” can double-bill. Retry only idempotent operations, or give each write an idempotency key the server uses to recognize and discard duplicates. Without idempotency, a retry policy is a correctness bug waiting to happen.

Why does every remote call need a hard timeout?

To make the call return faster on success So a single attempt can never hang indefinitely, capping worst-case latency and freeing the resource Because the network protocol requires one To force the dependency to respond more quickly

What problem does adding jitter to retry delays solve?

It makes retries faster It spreads concurrent clients out in time, preventing a synchronized thundering herd of retries It encrypts the retry payload It guarantees the retry will succeed

Which operations are safe to retry without extra care?

Any operation at all Idempotent operations, where running them twice has the same effect as running them once Only operations that write to a database Only operations that take less than one second

Why should retries be paired with a circuit breaker?

The breaker makes retries faster Without a breaker, retries against a genuinely overloaded dependency multiply load into a retry storm A breaker is required to measure timeouts Retries cannot run without a breaker present

Submit

Mark as complete