jdfive

Everyone Waiting — Deadlocks

One morning, QuickCart froze. No errors in the logs. No crashes. Orders just stopped moving. Priya opened a thread dump and saw two threads stuck forever — each waiting for a lock the other one held. Nobody could move. Everyone was waiting. That is a deadlock.

A QuickCart story with two locks

Priya has two shared resources:

  • Lock A — the inventory database row for a product.
  • Lock B — the payment gateway connection.

Two threads run at the same time:

Thread 1 — update inventory, then charge payment:

synchronized (inventoryLock) {
    synchronized (paymentLock) {
        updateStock();
        chargeCard();
    }
}

Thread 2 — refund payment, then restore inventory:

synchronized (paymentLock) {
    synchronized (inventoryLock) {
        refundCard();
        restoreStock();
    }
}

What can go wrong:

  1. Thread 1 grabs inventoryLock.
  2. Thread 2 grabs paymentLock.
  3. Thread 1 waits for paymentLock (held by Thread 2).
  4. Thread 2 waits for inventoryLock (held by Thread 1).

Neither thread can finish. Neither releases its lock. They wait forever.

This is the classic deadlock pattern — two threads, two locks, opposite order.

Four conditions — all must be true

Deadlock needs all four of these at the same time:

  1. Mutual exclusion — only one thread can hold a lock at a time. Locks work this way by design.

  2. Hold and wait — a thread keeps one lock while waiting for another. Thread 1 holds A and waits for B.

  3. No preemption — you cannot force a thread to give up its lock. It must release it itself.

  4. Circular wait — threads form a circle of waiting. Thread 1 waits for Thread 2’s lock. Thread 2 waits for Thread 1’s lock.

To prevent deadlock, break at least one condition. In practice, developers usually break circular wait (lock ordering) or hold and wait (tryLock with timeout).

Fix 1: Lock ordering (break circular wait)

Always grab locks in the same order. Every thread. No exceptions.

QuickCart transfers value between two customer wallet accounts. Without ordering, two transfers can deadlock. With ordering, they cannot:

public class WalletTransferService {
    public void transfer(Wallet from, Wallet to, int amount) {
        Wallet first = from.getId() < to.getId() ? from : to;
        Wallet second = from.getId() < to.getId() ? to : from;

        synchronized (first) {
            synchronized (second) {
                from.withdraw(amount);
                to.deposit(amount);
            }
        }
    }
}

Both threads always lock the lower wallet ID first, then the higher. The circle never forms.

For Priya’s inventory and payment locks, pick a rule: always lock inventory before payment, in every code path. Document the rule so the whole team follows it.

Fix 2: tryLock with timeout (break hold and wait)

synchronized blocks forever. ReentrantLock lets you try for a limited time and back off if you fail.

import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;

public class SafePaymentAndStock {
    private final ReentrantLock inventoryLock = new ReentrantLock();
    private final ReentrantLock paymentLock = new ReentrantLock();

    public boolean updateAndCharge() {
        try {
            if (inventoryLock.tryLock(5, TimeUnit.SECONDS)) {
                try {
                    if (paymentLock.tryLock(5, TimeUnit.SECONDS)) {
                        try {
                            updateStock();
                            chargeCard();
                            return true;
                        } finally {
                            paymentLock.unlock();
                        }
                    }
                } finally {
                    inventoryLock.unlock();
                }
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        return false; // could not get both locks — retry or fail safely
    }
}

If Thread 1 cannot get paymentLock within 5 seconds, it releases inventoryLock and returns. Thread 2 gets a chance. No infinite wait.

Always unlock in a finally block so a crash inside the critical section does not leave the lock held.

Fix 3: Single lock (break hold and wait)

If two resources always change together, use one lock for both:

// Risky — two locks, two orders possible
synchronized (inventoryLock) {
    synchronized (paymentLock) {
        updateStock();
        chargeCard();
    }
}

// Safer — one lock, no ordering problem
private final Object orderLock = new Object();

synchronized (orderLock) {
    updateStock();
    chargeCard();
}

Simple and safe. The trade-off: only one thread can do order work at a time. For QuickCart’s hot paths, Priya uses single lock only when traffic is low.

Fix 4: Lock-free tools (break mutual exclusion)

Sometimes you can avoid locks entirely:

import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.ConcurrentHashMap;

// No locks needed for simple counters
private AtomicInteger ordersProcessed = new AtomicInteger(0);
ordersProcessed.incrementAndGet();

// No locks needed for concurrent maps
private ConcurrentHashMap<String, Integer> stockLevels = new ConcurrentHashMap<>();
stockLevels.merge("SKU-101", 1, Integer::sum);

AtomicInteger and ConcurrentHashMap handle thread safety internally. No nested locks, no deadlock risk for these operations.

Fix 5: Avoid deep nested locks

Each extra lock level adds deadlock risk:

// Risky — three locks deep
synchronized (lockA) {
    synchronized (lockB) {
        synchronized (lockC) {
            doWork();
        }
    }
}

// Better — flatten to one lock when possible
synchronized (orderLock) {
    doWork();
}

If you need multiple locks, keep nesting shallow and always use consistent ordering.

Retry with backoff

When tryLock fails, do not give up immediately. Retry a few times with a short pause:

public void transferWithRetry(Wallet from, Wallet to, int amount) {
    int maxRetries = 3;
    int retryCount = 0;

    while (retryCount < maxRetries) {
        if (tryTransfer(from, to, amount)) {
            return; // success
        }
        retryCount++;
        try {
            Thread.sleep(100 * (1 << retryCount)); // 200ms, 400ms, 800ms
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            return;
        }
    }
    throw new RuntimeException("Transfer failed after " + maxRetries + " retries");
}

private boolean tryTransfer(Wallet from, Wallet to, int amount) {
    Wallet first = from.getId() < to.getId() ? from : to;
    Wallet second = from.getId() < to.getId() ? to : from;

    try {
        if (first.tryLock(2, TimeUnit.SECONDS)) {
            try {
                if (second.tryLock(2, TimeUnit.SECONDS)) {
                    try {
                        from.withdraw(amount);
                        to.deposit(amount);
                        return true;
                    } finally {
                        second.unlock();
                    }
                }
            } finally {
                first.unlock();
            }
        }
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
    }
    return false;
}

Exponential backoff gives other threads time to finish before you try again.

Detecting deadlock

If QuickCart hangs again, Priya can capture a thread dump:

jstack <pid> > thread-dump.txt

Look for threads in BLOCKED state and lines like “waiting to lock”. You will often see two threads each waiting for what the other holds.

Java can also find deadlocks in running code:

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadMXBean;
import java.lang.management.ThreadInfo;

public class QuickCartDeadlockChecker {
    public void check() {
        ThreadMXBean bean = ManagementFactory.getThreadMXBean();
        long[] deadlocked = bean.findDeadlockedThreads();

        if (deadlocked != null) {
            ThreadInfo[] infos = bean.getThreadInfo(deadlocked);
            System.err.println("Deadlock detected!");
            for (ThreadInfo info : infos) {
                System.err.println(info.getThreadName() + " waiting on "
                    + info.getLockInfo());
            }
        }
    }
}

Run this on a schedule in production:

// Check every 60 seconds
@Scheduled(fixedRate = 60000)
public void scheduledDeadlockCheck() {
    ThreadMXBean bean = ManagementFactory.getThreadMXBean();
    long[] deadlocked = bean.findDeadlockedThreads();

    if (deadlocked != null) {
        log.error("Deadlock detected in QuickCart! Thread count: {}",
            deadlocked.length);
        // alert ops team, capture thread dump, maybe restart
    }
}

Deadlock vs livelock vs starvation

These three problems look similar but are different.

Deadlock — threads are blocked, waiting forever. Nothing moves. QuickCart’s inventory/payment freeze is a deadlock.

Livelock — threads are active but make no progress. Two people in a hallway both step aside the same way, then the other way, forever. In code: two threads both detect a lock conflict and both back off the same way, then retry at the same instant, forever.

Starvation — one thread never gets the resource it needs. A low-priority thread keeps getting skipped while high-priority threads grab the lock every time. Fix: use fair locks (new ReentrantLock(true)) or ensure every thread eventually gets a turn.

Problem Threads active? Progress?
Deadlock blocked none
Livelock running none
Starvation some running one thread stuck

Rules for QuickCart

  1. Always acquire locks in consistent order — wallet ID ordering, or inventory before payment.
  2. Use tryLock with timeout when blocking forever is not acceptable.
  3. Release locks in finally — every time.
  4. Avoid nested locks — flatten or use lock-free tools where you can.
  5. Never hold locks during I/O — do not call HTTP or database while holding a lock. Other threads wait for nothing useful.
  6. Monitor for deadlocks — scheduled ThreadMXBean checks or alerts on thread dump analysis.
// BAD — holds lock during slow HTTP call
synchronized (inventoryLock) {
    paymentClient.charge(orderId); // blocks other threads for 500ms
}

// GOOD — lock only around the memory update
String paymentResult = paymentClient.charge(orderId); // no lock held
synchronized (inventoryLock) {
    updateStock(orderId, paymentResult);
}

What to remember

  • Deadlock = two or more threads waiting for each other forever.
  • All four conditions must be true for deadlock to happen.
  • Lock ordering is the most common fix — always grab locks in the same order.
  • tryLock with timeout lets a thread give up and retry instead of waiting forever.
  • Single lock or lock-free tools remove the problem entirely for simple cases.
  • Use jstack or ThreadMXBean.findDeadlockedThreads() to detect deadlocks in a running app.
  • Livelock = active but no progress. Starvation = one thread never gets a turn.

What Priya does next: orders run in thread pools, but log lines lose the order ID. She learns how to carry request context across threads with MDC.