How to test webhook delivery from BaaS and payment providers

Where webhook receivers lose money

A card clearing, an incoming ACH credit, or a returned transfer reaches your platform as a webhook, and your handler turns it into a ledger posting. Four defects in that handler reach production because a sandbox run only exercises the happy path:

• Double posting on retry. The provider times out waiting for your response, retries, and your handler posts the same transaction again. The end user's balance moves twice while the partner bank core shows one movement.
• Missed posting. Every retry fails during an outage, the provider stops, and the event never reaches the ledger. The account shows a stale balance until someone reconciles.
• Stale state wins. A pending status arrives after posted and your handler overwrites the newer state.
• Forged or replayed event. A receiver that skips signature or timestamp checks accepts a captured request sent again later.

For a consumer account, a computational or bookkeeping error made by the financial institution relating to an electronic fund transfer is an error under Regulation E, 12 CFR 1005.11(a)(1)(iv). Section 1005.11(c)(1) then requires the institution to determine whether an error occurred within 10 business days of the notice, subject to the exceptions in that paragraph. The Regulation E page covers the rest of the rule; the BaaS segment page covers where these postings sit in a programme ledger.

Procedure

The examples use two providers whose documentation states their webhook behaviour: Increase and Lithic (both read on 2026-09-22). Both use the header names defined in the Standard Webhooks specification: webhook-id, webhook-timestamp, and webhook-signature. Sender-side checks of the same scheme belong to API and webhook testing; this guide tests the receiver.

1. Capture a real delivery shape

Increase sends a thin event: identifiers only, with the full object fetched through the API. A captured sandbox delivery looks like this (test values):

POST /webhooks/increase HTTP/1.1
Host: receiver.example.com
Content-Type: application/json
webhook-id: event_test_001
webhook-timestamp: 1790000000
webhook-signature: v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDN/H/pI4=

{"id":"event_test_001","created_at":"2026-09-21T14:13:20Z","category":"transaction.created","associated_object_type":"transaction","associated_object_id":"transaction_test_abc","type":"event"}

The signature value above is a placeholder. Increase counts a delivery as successful only when your endpoint returns a 2xx within 10 seconds.

2. Verify the signature on the raw body

The signed content is webhook-id, a period, webhook-timestamp, a period, and the raw body, signed with HMAC-SHA256 and Base64-encoded with a v1, prefix. Lithic documents a secret with a whsec_ prefix whose Base64 remainder is the key. The header can carry several space-separated signatures during secret rotation.

import base64
import hashlib
import hmac
import time

TOLERANCE_SECONDS = 300  # receiver policy; both vendors' examples use 5 minutes


def verify(secret: str, headers: dict, raw_body: bytes, now: int | None = None) -> bool:
    msg_id = headers["webhook-id"]
    timestamp = headers["webhook-timestamp"]
    now = int(time.time()) if now is None else now
    if abs(now - int(timestamp)) > TOLERANCE_SECONDS:
        return False
    key = base64.b64decode(secret.removeprefix("whsec_"))
    signed = f"{msg_id}.{timestamp}.".encode() + raw_body
    expected = base64.b64encode(hmac.new(key, signed, hashlib.sha256).digest()).decode()
    for candidate in headers["webhook-signature"].split(" "):
        version, _, value = candidate.partition(",")
        if version == "v1" and hmac.compare_digest(value, expected):
            return True
    return False

The tests that matter feed the handler bytes, not parsed JSON:

import base64
import hashlib
import hmac
import json

from receiver import verify

SECRET = "whsec_" + base64.b64encode(b"test-only-secret").decode()
BODY = b'{"id":"event_test_001","category":"transaction.created"}'
TS = 1790000000


def sign(body: bytes, ts: int = TS, msg_id: str = "event_test_001") -> dict:
    key = base64.b64decode(SECRET.removeprefix("whsec_"))
    mac = hmac.new(key, f"{msg_id}.{ts}.".encode() + body, hashlib.sha256).digest()
    return {"webhook-id": msg_id, "webhook-timestamp": str(ts),
            "webhook-signature": "v1," + base64.b64encode(mac).decode()}


def test_reserialised_body_fails():
    reserialised = json.dumps(json.loads(BODY)).encode()  # adds spaces after separators
    assert not verify(SECRET, sign(BODY), reserialised, now=TS)


def test_stale_timestamp_fails():
    assert not verify(SECRET, sign(BODY), BODY, now=TS + 301)


def test_rotation_second_signature_passes():
    headers = sign(BODY)
    headers["webhook-signature"] = "v1,AAAA " + headers["webhook-signature"]
    assert verify(SECRET, headers, BODY, now=TS)

A handler that receives a parsed object from its framework fails the first test. The Standard Webhooks specification warns that even a stray space invalidates the signature and tells implementers to use a constant-time comparison for symmetric signatures.

3. Force retries and duplicates

Make the receiver fail on purpose: return 500, return 2xx after 11 seconds, and drop the connection after the ledger commit. Then assert one posting per webhook-id. Increase states that webhook-id is the Event ID and asks receivers to use it as the idempotency key; Lithic states that it stays unchanged when a message is resent.

CREATE TABLE webhook_inbox (
  webhook_id   text PRIMARY KEY,
  received_at  timestamptz NOT NULL DEFAULT now()
);

-- Same transaction as the ledger posting. No row returned means a duplicate: skip and return 200.
INSERT INTO webhook_inbox (webhook_id)
VALUES ($1)
ON CONFLICT (webhook_id) DO NOTHING
RETURNING webhook_id;

The inbox insert and the posting must commit together. A handler that writes the inbox row after posting double-posts when it crashes between the two writes. Request-side deduplication for your own outgoing calls is a separate problem; the payment idempotency testing guide covers it.

4. Deliver events out of order

Send transaction.created after a later update for the same object. With thin events, the handler fetches the current object, so the test checks that the handler re-fetches instead of reading a cached copy. With full payloads, the handler must compare a version or timestamp on the object and discard the older one.

5. Reconcile against polling

Webhooks are a notification path; the event list is the record. Increase returns events from its List Events API for 30 days after creation, oldest first when you ask for it, with a cursor:

GET /events?order_by.field=created_at&order_by.direction=ascending&cursor=CURSOR_FROM_PREVIOUS_PAGE HTTP/1.1
Host: api.increase.com
Authorization: Bearer TEST_KEY_NOT_REAL

Load the pages into provider_events and find what the receiver never processed:

SELECT p.event_id, p.category, p.created_at
FROM provider_events AS p
LEFT JOIN webhook_inbox AS w ON w.webhook_id = p.event_id
WHERE w.webhook_id IS NULL
  AND p.created_at < now() - interval '15 minutes'
ORDER BY p.created_at;

Run the job daily over the full 30-day window, so an outage longer than the retry schedule is recovered before the events expire from the list. The ledger reconciliation guide continues from the event level to balances.

6. Test locally through a tunnel

Expose the local receiver, register the tunnel URL as a sandbox endpoint, and replay captured deliveries through the ngrok agent API:

ngrok http 8080
curl -s "http://127.0.0.1:4040/api/requests/http?limit=20"
curl -s -X POST http://127.0.0.1:4040/api/requests/http \
  -H "Content-Type: application/json" \
  -d '{"id":"CAPTURED_REQUEST_ID"}'

Increase does not retry failed webhooks in the sandbox, so sandbox runs never exercise your retry path; replays and a local signer built from the sign() fixture do. Point tunnels at sandbox only: a tunnel that forwards production events can carry account data to a developer laptop.

The retention window vendor docs leave to you

Neither vendor page cited here states how long your deduplication record must live. The answer is the longest path by which the same webhook-id can arrive again:

A deduplication key in a cache with a 24-hour expiry passes every sandbox test and still double-posts the Increase attempt at about 72 hours and the last automatic Lithic attempt. Keep inbox rows at least as long as the longest path, 90 days in this table, and add a unique constraint on the provider object ID in the ledger as a second guard.

The replay window has the opposite pitfall. The Standard Webhooks specification says the attempt timestamp is updated on every retry, so a genuine retry always passes a 5-minute tolerance. An ngrok replay reuses the old timestamp and must fail. Teams that widen the tolerance to make local replays pass ship a receiver that accepts a captured request days later. Keep the tolerance and replay fresh signatures from the fixture instead.

The 10-second success limit at Increase is shorter than the 15 to 30 second timeout range the Standard Webhooks specification suggests to senders. A handler that posts synchronously and waits on a slow partner bank call can pass 10 seconds under load, and each retry it triggers repeats work already committed.

Test matrix

Partner banks reviewing a programme ask for this evidence alongside reconciliation reports; BaaS and partner bank testing packages it for that review.