Idempotency Keys & Distributed Locks: Preventing Double Charges in APIs

The Double-Click Nightmare: How Network Retries Duplicate Transactions

Every payment engineering team eventually encounters the frantic customer support ticket: “I clicked ‘Pay Now’ once, but my bank statement shows two identical charges of €450!”

This bug almost never stems from malicious fraud. Instead, it is the direct consequence of naive API design colliding with the chaotic reality of modern mobile networks.

Consider this sequence of events:

1. A customer taps “Pay Now” on their smartphone while riding a bus.
2. The browser sends POST /api/v1/charges with an order payload.
3. The server charges the customer’s credit card successfully via Stripe.
4. Just as the server attempts to transmit the HTTP 200 OK response, the customer’s mobile connection switches cellular towers, dropping the TCP connection.
5. The customer’s mobile browser notices the dropped connection, displays a brief spinner, and automatically retries the HTTP POST request. Alternatively, the impatient customer vigorously taps “Pay Now” a second time.
6. The server receives the second request, treats it as a brand-new transaction, and charges the customer’s credit card a second time.

THE DOUBLE-CHARGE RACE CONDITION: NAIVE API PIPELINE

• Mobile Client (Weak Signal)
• API Server
• POST /api/charges (Tap 1) → (Charges CC #1) (Card Billed €450!)
• TCP RST / Connection Dropped by Cell Tower: X — (Client never knows!)
• (Browser or user retries request)
• POST /api/charges (Tap 2 - Duplicate) → (Charges CC #2!)
• 200 OK (Payment Confirmed) — (Card Billed €450 AGAIN!)
• Customer sees 1 confirmation on screen, but gets 2 bank notifications!

To build bulletproof financial workflows for web apps, software architects must design endpoints to be strictly idempotent: an operation that, when executed multiple times with identical inputs, leaves the system in the exact same state as if it had been called once.

Anatomy of an Idempotency Layer: Keys, States, and Atomic Storage

A resilient idempotency system relies on three synchronized architectural components:

1. Client-Generated Idempotency Keys: Before firing a mutating request, the client generates a unique UUIDv4 token and includes it in an HTTP header: Idempotency-Key: 7b9e2890-4e31-4122-83bc-f331f513904a.
2. Atomic Distributed Locking: Before executing payment logic, the backend attempts to acquire an exclusive lock on that key in an in-memory datastore like Redis. If another worker thread is already executing that exact key, the duplicate request is either held in a polling loop or rejected with 409 Conflict.
3. Persistent Response Storage: Once the transaction completes, the exact HTTP status code, response headers, and response body are persisted in PostgreSQL alongside the key for a retention window (e.g., 24 to 72 hours).

IDEMPOTENT REQUEST LIFECYCLE STATE MACHINE

• Incoming Request with Header: Idempotency-Key:
• Check PostgreSQL / Redis for Existing Record of : Exists & Status == “COMPLETED”?
• RETURN CACHED RESPONSE (Zero payment logic re-run!)
• Exists & Status == “PROCESSING”?
• RETURN HTTP 409 CONFLICT (“Transaction in progress”)
• Does Not Exist?
• 1. ACQUIRE ATOMIC LOCK (Redis SETNX)
• 2. INSERT Status = “PROCESSING”
• 3. Execute Card Payment via Payment Gateway
• 4. UPDATE Status = “COMPLETED” + Cache HTTP Response
• 5. RELEASE ATOMIC LOCK
• 6. Return Fresh HTTP 200 OK

When designed cleanly, this architecture provides ironclad financial safety while preserving the principles of anti-over-engineering.

Implementing Atomic Redis Locks and PostgreSQL Caching in Node.js

Let us inspect a concrete implementation using Node.js, Redis, and PostgreSQL. We leverage Redis SETNX for the atomic concurrency lock and PostgreSQL for durable long-term response caching.

First, the atomic lock helper:

// src/services/distributedLock.ts
import Redis from 'ioredis';
import crypto from 'node:crypto';

const redis = new Redis(process.env.REDIS_URL || 'redis://127.0.0.1:6379');

export async function acquireLock(
  resourceKey: string,
  ttlMs = 10000
): Promise<{ acquired: boolean; release: () => Promise<void> }> {
  const lockToken = crypto.randomUUID();
  const lockKey = `lock:idempotency:${resourceKey}`;

  // Atomic operation: sets key ONLY IF NOT EXISTS, with TTL in ms
  const result = await redis.set(lockKey, lockToken, 'PX', ttlMs, 'NX');

  if (result !== 'OK') {
    return { acquired: false, release: async () => {} };
  }

  // Safe release using Lua script to guarantee only lock owner can release
  const release = async () => {
    const luaScript = `
      if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("del", KEYS[1])
      else
        return 0
      end
    `;
    await redis.eval(luaScript, 1, lockKey, lockToken);
  };

  return { acquired: true, release };
}

Next, the middleware that intercepts requests and enforces idempotency:

// src/middleware/idempotencyMiddleware.ts
import { Request, Response, NextFunction } from 'express';
import { pool } from '../db/postgresPool.js';
import { acquireLock } from '../services/distributedLock.js';

export async function idempotencyMiddleware(req: Request, res: Response, next: NextFunction) {
  const key = req.headers['idempotency-key'] as string;
  if (!key) return next();

  // Step 1: Check durable storage in PostgreSQL
  const existing = await pool.query(
    'SELECT status, status_code, response_body FROM idempotency_keys WHERE key = $1',
    [key]
  );

  if (existing.rows.length > 0) {
    const record = existing.rows[0];
    if (record.status === 'COMPLETED') {
      return res.status(record.status_code).json(record.response_body);
    }
    if (record.status === 'PROCESSING') {
      return res.status(409).json({ error: 'Transaction is currently processing. Retry shortly.' });
    }
  }

  // Step 2: Acquire atomic distributed lock
  const { acquired, release } = await acquireLock(key);
  if (!acquired) {
    return res.status(409).json({ error: 'Concurrent duplicate request detected.' });
  }

  // Record initial processing state
  await pool.query(
    'INSERT INTO idempotency_keys (key, status, created_at) VALUES ($1, $2, NOW()) ON CONFLICT (key) DO NOTHING',
    [key, 'PROCESSING']
  );

  // Hook response to capture final status and output
  const originalJson = res.json.bind(res);
  res.json = (body: any) => {
    pool.query(
      'UPDATE idempotency_keys SET status = $1, status_code = $2, response_body = $3, completed_at = NOW() WHERE key = $4',
      ['COMPLETED', res.statusCode, body, key]
    ).finally(() => release());

    return originalJson(body);
  };

  next();
}

Running these services containerized via Docker provides complete parity between local development and cloud production.

Database Schema & Concurrency Race Conditions

In PostgreSQL, we model the idempotency table with strict constraints:

CREATE TABLE idempotency_keys (
    key VARCHAR(255) PRIMARY KEY,
    user_id UUID NOT NULL,
    request_path VARCHAR(255) NOT NULL,
    request_params_hash CHAR(64) NOT NULL, -- SHA-256 of request payload
    status VARCHAR(32) NOT NULL,           -- 'PROCESSING', 'COMPLETED', 'FAILED'
    status_code INTEGER,
    response_body JSONB,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    completed_at TIMESTAMPTZ
);

CREATE INDEX idx_idempotency_created_at ON idempotency_keys (created_at);

Notice the request_params_hash column. If a malicious user or broken client attempts to reuse the same Idempotency-Key but changes the payment amount from €5 to €500, the server must compute the SHA-256 hash of the incoming request body and verify that it matches the original hash. If the payload differs, the server rejects the request with 422 Unprocessable Entity to prevent payload tampering attacks. Open-source implementations can be audited on GitHub.

Production Best Practices: TTL Expiration and Client Retries

When operating financial transaction engines built to the enterprise standards of WebXtek, follow these mission-critical operational rules:

1. Define Strict TTL Policies: Idempotency records should expire after 24 to 72 hours. Maintain a daily PostgreSQL cleanup cron or partition tables by day to prune stale records automatically.
2. Client-Side Exponential Jitter: Client frontends and SDKs must never spam retries in an unconstrained loop. If an API returns 409 Conflict, the client must pause using exponential backoff with randomized jitter (e.g., 200ms, 600ms, 1400ms).
3. Distinguish Business Logic Errors from System Crashes: If a customer’s credit card is declined due to insufficient funds, that is a legitimate final outcome. Cache the declined 402 Payment Required response under the idempotency key so repeated button clicks do not continue hitting payment gateways and triggering card issuer fraud bans.

By pairing atomic Redis distributed locks with durable PostgreSQL idempotency logging, your engineering team can guarantee exactly-once payment semantics regardless of flaky cellular connections or impatient customer double-clicks.

In practice, we apply these exact architectural standards across specialized sectors including b2b saas platforms and devops & cloud architecture specialists. Whether you are modernizing core workflows through web apps and high-performance websites, connecting sector-specific requirements to disciplined execution is what turns technical theory into compounding commercial advantage.