Webhook Retries Without Duplicate Leads: An Idempotency Guide

A webhook can succeed and still look like a failure to the sender. Your receiver might create the lead, then lose the connection before its response arrives. The sender retries, and the same form submission creates another lead.

The fix is to give each event a stable identity, record that identity with an atomic write, and make the final side effect an upsert rather than a blind create. This guide builds that path with Node.js, DynamoDB, and a CRM-style destination.

The guarantee you actually want

HTTP defines an operation as idempotent when repeating the same request has the same intended effect as making it once. RFC 9110 applies that property to methods such as PUT and DELETE, but webhook deliveries normally use POST. Your receiver has to add the safety itself.

It helps to be picky with the wording:

• A conditional database write can record one receipt when two copies arrive together.
• DynamoDB Streams can deliver a receipt to a worker more than once.
• A CRM upsert keyed by the submission ID can turn those worker runs into one logical lead.

That is effectively-once business behavior. The code may run twice. The user-facing result should still exist once.

The finished path is:

Static Forms -> authenticated receiver -> DynamoDB receipt -> DynamoDB Stream -> CRM upsert

Use the submission, not the timestamp, as identity

Current Static Forms deliveries contain data.submissionId. They also send an Idempotency-Key in this form:

staticforms:form.submitted:<submissionId>

Every retry for the same delivery carries the same value. The timestamp is not a safe substitute because two events can share a timestamp, and a retried payload may reach you much later.

Treat the header and body as a pair. If the header says one submission ID and the body says another, reject the request before writing anything. This catches broken proxies, mapping mistakes, and hand-built test requests that do not match the production contract.

An idempotency key is not authentication. Validate the sender first, then run the receipt logic. The webhook authentication guide covers Bearer, Basic, and custom headers without mixing credentials into this example.

Store the receipt with a conditional write

Create a DynamoDB table whose partition key is pk and enable DynamoDB Streams with new images. The receipt item stores the key, payload, status, and cleanup timestamp.

Use a retention window that covers your real replay and support needs. The sample keeps receipts for 30 days. DynamoDB TTL removes expired items asynchronously, so expiry is storage cleanup rather than a precise correctness boundary. If someone can replay events after 30 days, extend the window or archive processed identities elsewhere.

The receiver below expects a parsed JSON body and an authenticated request. It uses @aws-sdk/lib-dynamodb and @aws-sdk/util-dynamodb.

import { PutCommand, UpdateCommand } from "@aws-sdk/lib-dynamodb";
import { unmarshall } from "@aws-sdk/util-dynamodb";

function readHeader(headers, name) {
  if (typeof headers?.get === "function") return headers.get(name);
  const match = Object.entries(headers ?? {}).find(
    ([key]) => key.toLowerCase() === name.toLowerCase(),
  );
  return match?.[1] ?? null;
}

export async function acceptWebhook(request, { doc, tableName, now = Date.now }) {
  const event = request.body?.event;
  const submissionId = request.body?.data?.submissionId;

  if (event !== "form.submitted" || typeof submissionId !== "string" || !submissionId) {
    return { status: 400, body: { error: "Invalid webhook envelope" } };
  }

  const expectedKey = `staticforms:${event}:${submissionId}`;
  if (readHeader(request.headers, "idempotency-key") !== expectedKey) {
    return { status: 400, body: { error: "Invalid idempotency key" } };
  }

  const receivedAt = new Date(now()).toISOString();
  const expiresAt = Math.floor(now() / 1000) + 30 * 24 * 60 * 60;

  try {
    await doc.send(new PutCommand({
      TableName: tableName,
      Item: {
        pk: expectedKey,
        event,
        submissionId,
        payload: request.body,
        status: "RECEIVED",
        receivedAt,
        expiresAt,
      },
      ConditionExpression: "attribute_not_exists(pk)",
    }));
  } catch (error) {
    if (error?.name === "ConditionalCheckFailedException") {
      return { status: 200, body: { accepted: true, duplicate: true } };
    }
    throw error;
  }

  return { status: 202, body: { accepted: true, duplicate: false } };
}

export async function processStreamRecord(record, { crm, doc, tableName, now = Date.now }) {
  if (record.eventName !== "INSERT" || !record.dynamodb?.NewImage) {
    return { skipped: true };
  }

  const receipt = unmarshall(record.dynamodb.NewImage);
  if (receipt.status !== "RECEIVED") return { skipped: true };

  const form = receipt.payload?.data?.formData ?? {};
  await crm.upsertLead({
    externalId: receipt.submissionId,
    name: String(form.name ?? ""),
    email: String(form.email ?? ""),
    message: String(form.message ?? ""),
  });

  await doc.send(new UpdateCommand({
    TableName: tableName,
    Key: { pk: receipt.pk },
    UpdateExpression: "SET #status = :completed, completedAt = :completedAt",
    ExpressionAttributeNames: { "#status": "status" },
    ExpressionAttributeValues: {
      ":completed": "COMPLETED",
      ":completedAt": new Date(now()).toISOString(),
    },
  }));

  return { skipped: false };
}

The important line is ConditionExpression: "attribute_not_exists(pk)". AWS documents this pattern for preventing a put from overwriting an item with the same key in its DynamoDB condition expression guide. Two receivers can race, but only one conditional put succeeds.

A duplicate is not an error at this point. The original event is already durable, so the receiver returns 200. A new receipt returns 202 because the downstream work has been accepted, not completed.

If DynamoDB is unavailable, let the request fail with a server error. Returning 200 before the receipt exists would tell the sender to stop while leaving you with no durable work to process.

Keep the side effect safe too

The Stream worker uses submissionId as the CRM's external ID. That detail carries the architecture.

AWS says Lambda event source mappings for DynamoDB Streams process records at least once, so a function can receive the same record again. Read the warning in Using Lambda with Amazon DynamoDB. The conditional receipt stops duplicate INSERT records caused by duplicate webhook requests, but it does not make the worker exactly once.

Your destination needs one of these behaviors:

• A native upsert with a unique external ID.
• A create request with the same idempotency key on every attempt.
• A local transaction that reserves the destination identity and writes the business record atomically.

Do not replace crm.upsertLead() with an unconditional crm.createLead() and call the design finished. If the CRM call succeeds but the worker crashes before updating the receipt to COMPLETED, DynamoDB Streams can run the worker again. An upsert returns or updates the same lead. A blind create makes another one.

For destinations without an upsert or idempotency API, place the side effect behind a database you control. Give the business record a unique constraint on the submission ID. A separate SELECT followed by INSERT is still racy unless the unique constraint or transaction enforces the decision.

Test the race and the worker retry

A happy-path test does not prove idempotency. Send the same identity twice and simulate the Stream record twice.

import assert from "node:assert/strict";
import test from "node:test";
import { marshall } from "@aws-sdk/util-dynamodb";

import { acceptWebhook, processStreamRecord } from "./receiver.mjs";

const body = {
  event: "form.submitted",
  data: {
    submissionId: "sub_123",
    formData: { name: "Ada", email: "ada@example.com", message: "Hello" },
  },
};

function request(key = "staticforms:form.submitted:sub_123") {
  return { headers: { "Idempotency-Key": key }, body };
}

test("stores one receipt and acknowledges a retry", async () => {
  const keys = new Set();
  const doc = {
    async send(command) {
      if (keys.has(command.input.Item.pk)) {
        const error = new Error("duplicate");
        error.name = "ConditionalCheckFailedException";
        throw error;
      }
      keys.add(command.input.Item.pk);
    },
  };

  const first = await acceptWebhook(request(), { doc, tableName: "receipts", now: () => 0 });
  const retry = await acceptWebhook(request(), { doc, tableName: "receipts", now: () => 0 });

  assert.equal(first.status, 202);
  assert.deepEqual(retry, { status: 200, body: { accepted: true, duplicate: true } });
  assert.equal(keys.size, 1);
});

test("rejects a key that does not match the payload", async () => {
  const doc = { send: () => assert.fail("DynamoDB must not be called") };
  const result = await acceptWebhook(request("wrong"), { doc, tableName: "receipts" });
  assert.equal(result.status, 400);
});

test("repeated stream delivery still produces one logical lead", async () => {
  const leads = new Map();
  let updates = 0;
  const crm = {
    async upsertLead(lead) {
      leads.set(lead.externalId, lead);
    },
  };
  const doc = { async send() { updates += 1; } };
  const receipt = {
    pk: "staticforms:form.submitted:sub_123",
    submissionId: "sub_123",
    status: "RECEIVED",
    payload: body,
  };
  const record = { eventName: "INSERT", dynamodb: { NewImage: marshall(receipt) } };

  await processStreamRecord(record, { crm, doc, tableName: "receipts", now: () => 0 });
  await processStreamRecord(record, { crm, doc, tableName: "receipts", now: () => 0 });

  assert.equal(leads.size, 1);
  assert.equal(leads.get("sub_123").email, "ada@example.com");
  assert.equal(updates, 2);
});

Run the test with node --test receiver.test.mjs. It checks three things: only one receipt survives a duplicate delivery, a mismatched key never reaches DynamoDB, and two worker runs still leave one logical lead.

The in-memory CRM is only a test double. In production, the uniqueness must come from the destination or a durable store, not a JavaScript Map.

Match the response to the state you reached

Status codes should describe what your receiver knows, not what it hopes will happen.

Receiver state	Response	Reason
Authentication failed	401	The sender did not prove its identity
Envelope or key is invalid	400	Retrying the same bad request will not repair it
Receipt was stored	202	Durable asynchronous work now exists
Receipt already exists	200	The duplicate is safely acknowledged
Receipt store is unavailable	500 or 503	The sender should retry because no durable acceptance is proven

Current Static Forms delivery code makes up to three attempts for network failures and HTTP 408, 409, 425, 429, and 5xx responses. It reuses the same idempotency key for those attempts. That is current product behavior, not a promise about every webhook provider, so check the retry rules of each sender you integrate.

You can inspect Static Forms attempts in Delivery logs. The Delivery documentation shows where to configure the endpoint, authentication, test request, and logs.

Failure cases worth testing before production

Test with a non-production destination first. The webhook payload preview is useful for seeing the envelope, but the final check should use your deployed receiver.

1. Send the same payload and idempotency key concurrently. Expect one 202, one 200, and one receipt.
2. Send the same body with a different key. Expect 400 and no new receipt.
3. Make the CRM upsert succeed, then throw before the receipt update. Re-run the worker and confirm that the same lead is updated rather than duplicated.
4. Make DynamoDB fail before the conditional put. Expect a server error and no success response.
5. Re-deliver an event after the receipt's retention window. Decide whether your archive or downstream unique ID still blocks a duplicate.

Also verify logs by content, not volume. Record the request ID, submission ID, outcome, and status. Do not log authentication values or the full form payload, which may contain names, email addresses, messages, and attachments.

Know where this pattern stops

This design is a good fit when duplicate leads, tickets, or notifications have a real operational cost and you already run AWS infrastructure. It adds a table, a Stream, a worker, alarms, and a reconciliation job. That is more machinery than every webhook needs.

For a small internal notification, a destination's native idempotency option may be enough. For a multi-step workflow, give each irreversible step its own stable identity instead of assuming one receipt makes the whole chain atomic.

Static Forms users connecting to n8n can start with the n8n webhook guide. When the workflow begins creating records that must not duplicate, move the deduplication check ahead of that side effect. The broader webhooks integration page covers setup and delivery visibility.

Before shipping, prove these four facts: the sender identity is checked, duplicate requests create one receipt, repeated worker runs upsert one business record, and a storage failure never receives a false success response.