Webhook Retry Design: Handling Failed Payment Notifications

The first time one of our orders got double-credited, I didn't believe it was a webhook problem — I assumed it was a bug in our own order service, because that's usually where bugs live. It took an afternoon of grepping logs to find what actually happened: our payment gateway had sent the same "payment success" notification twice, a few minutes apart, and our handler — which back then just did "find order, mark as paid, trigger fulfillment" with no safeguards — happily ran the fulfillment logic twice. Nobody told us this would happen. We learned it the hard way, which is the usual way developers learn how webhooks behave in production.

This is the write-up I wish I'd had before that afternoon. It isn't about how to receive a webhook — every gateway's docs cover that. It's about the part docs tend to gloss over: why delivery to your server is never a sure thing on the first try, how a gateway's retry mechanism is typically structured, and — the piece that actually decides whether you get paged at 2am — how to design your handler so receiving the same notification twice is a non-event instead of a double-processed order.

Why the First Delivery Attempt Doesn't Always Succeed

A webhook is just an HTTP request the gateway's server sends to a URL you configured, and like any HTTP request over the public internet, it can fail for reasons that have nothing to do with the payment itself. Running checkout integrations against Philippine wallets and banks, the failures we've actually seen fall into a short, boring list:

• Your server was mid-deploy. A rolling restart or a load balancer briefly routing to an instance that hasn't finished booting can turn a webhook into a connection refused.
• Your endpoint took too long to respond. Synchronous database writes, an outbound email call, a few internal service calls before returning a status code — you're betting all of it finishes inside the gateway's timeout window. Lose that bet and the gateway sees a timeout, not a slow success.
• A firewall or security group rule changed. We've seen this happen after an unrelated infrastructure change — someone tightens an inbound rule, and the gateway's IP range quietly stops getting through.
• A transient network blip. Packet loss, a DNS hiccup, a TLS handshake that times out — none of these are your fault, but they still count as a failed delivery.
• Your application returned something other than a clean success status. A 500 from an unhandled exception, a 502 from an upstream proxy, even a 200 with a body the gateway can't parse can all be treated as a failure.

The gateway has no way to tell "your server is having a bad five minutes" apart from "this webhook URL is permanently broken." All it knows is the attempt didn't come back clean. That's the entire reason retry logic exists.

How a Payment Gateway's Retry Logic Typically Works

When a transaction reaches a final state — paid, failed, refunded — the notification service fires an HTTP POST at your configured webhook URL and waits for a response within a bounded window, typically just a handful of seconds. If your endpoint answers with a clean 2xx status in that window, the delivery is marked successful and the gateway moves on. Anything else — a timeout, a connection error, a non-2xx status, no response at all — gets queued for another attempt later.

The "later" part follows an exponential backoff pattern — a fairly universal design across webhook-based systems, not something specific to any one provider. The gap between attempt one and two is short; if that fails too, the gap before attempt three is longer; each subsequent failure widens the interval, up to some ceiling, inside a bounded total window rather than forever. Once that window closes or the attempt count runs out, the gateway stops pushing and it's on you to pull the final status yourself.

SMARTPAY operates in the Philippines under BSP OPS/EMI licensing, and the notification pipeline is built around the same reconciliation discipline that license expects — a transaction can never be allowed to sit in an unknown state just because one HTTP request happened to fail. Retries exist precisely to close that gap before it becomes a reconciliation problem.

If you want the signature verification and reconciliation side of this covered in more depth, our piece on how GCash merchant callback notifications and signature verification work walks through the full notification lifecycle, from the raw POST body through confirming it's genuinely from the gateway.

A Typical Retry Schedule

Exact numbers vary by gateway and by channel, so treat what follows as illustrative of a typical schedule's shape, not a committed SLA. What stays consistent across implementations is the curve: short gaps early, wider gaps later, then a cutoff.

Attempt	Typical backoff interval	What the merchant should do
Initial attempt (not a retry)	Sent as soon as the transaction reaches a final status	Confirm your endpoint returned 2xx; if you never see this land, check inbound firewall rules first
Retry 1	A short gap, usually measured in minutes	Pull the original request from your access logs and look for a timeout or non-2xx response
Retry 2	Longer than retry 1, roughly tens of minutes	Check whether your handler is doing slow synchronous work that could be hitting the gateway's response timeout
Retry 3	Measured in hours rather than minutes	Treat this as a signal your endpoint has been unreachable for a while; check uptime and recent deploys
Retry 4 (near final)	Several hours, approaching the edge of the retry window	If you're still failing at this point, assume the automated retries won't save you — start a manual check
Window exhausted	No further retries are sent	Use the query/reconciliation API to pull the transaction's final status directly; don't assume "no notification" means "no payment"

Here's roughly what that looks like from the merchant side when you query a notification's delivery history rather than just waiting for it to show up:

curl -X GET "https://apix.smartpay.example/api/v1/webhooks/delivery-log?merid=MERCHANT001&tno=ORDER10452" \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "tno": "ORDER10452",
  "notification_id": "NTF-9F3A21",
  "target_url": "https://merchant.example.com/webhooks/payment",
  "attempts": [
    { "attempt": 1, "sent_at": "2026-09-14T10:02:03+08:00", "result": "timeout" },
    { "attempt": 2, "sent_at": "2026-09-14T10:06:41+08:00", "result": "connection_refused" },
    { "attempt": 3, "sent_at": "2026-09-14T10:34:18+08:00", "result": "http_500" },
    { "attempt": 4, "sent_at": "2026-09-14T12:51:02+08:00", "result": "success", "response_code": 200 }
  ],
  "final_status": "delivered",
  "retries_exhausted": false
}

That log does you a favor: it tells you exactly which attempts failed and why, instead of leaving you to guess from a single "didn't arrive" data point. If your own endpoint logs don't line up with this, that mismatch is usually where the real bug hides.

Why Idempotency Is the Merchant's Responsibility, Not the Gateway's

Here's the part that surprised me most: a duplicate delivery isn't a malfunction, it's the retry mechanism working exactly as designed. If attempt one reaches your server and your handler finishes its work, but the response is lost on the way back — a network blip, your process restarting right after the database write but before the HTTP response goes out — the gateway has no way to know your side succeeded. From its point of view that attempt failed, so a retry gets queued. Your server then receives a perfectly valid, correctly signed notification for a transaction it already processed.

If your handler's logic is "receive notification, mark order as paid, trigger fulfillment" with nothing checking whether that already happened, the second delivery runs the exact same logic again. Depending on what fulfillment does, that can mean a second shipment, a second confirmation email, a second wallet credit, or a second subscription period for one payment. None of this is a gateway bug — it's a predictable consequence of building a stateless handler on top of a system that retries by design, and it's on the merchant's side of the integration to account for it.

This is also why idempotent handling isn't just a code-quality nicety for a BSP OPS/EMI-licensed payment flow — a duplicate credit or a duplicate payout instruction is exactly the kind of discrepancy that reconciliation is supposed to catch, and it's far cheaper to prevent at the handler level than to unwind after the fact.

This is a different problem from the one covered in our notes on how we handle API rate limiting and concurrent request design, which is about outbound calls you make to the gateway hitting a request ceiling. Webhook retries are inbound — the gateway calling you — and idempotency there is about making repeated inbound calls safe, not about managing your own call volume.

A Concrete Pattern: The Idempotent Webhook Handler

The pattern itself isn't complicated, which is part of why it's easy to skip until a duplicate actually bites you. The shape we use looks like this: verify the signature first, extract a unique identifier from the payload, try to record that identifier before doing anything else, and only run your business logic if that record didn't already exist. Here's a simplified version in Node.js against Postgres:

app.post('/webhooks/payment', async (req, res) => {
  // 1. Verify signature before trusting anything in the body
  if (!verifySignature(req.body, req.headers['x-signature'])) {
    return res.status(400).send('invalid signature');
  }

  const { notification_id, tno, status } = req.body;
  const client = await pool.connect();

  try {
    await client.query('BEGIN');

    // 2. Claim notification_id as a unique row (UNIQUE constraint in schema)
    const claim = await client.query(
      `INSERT INTO webhook_log (notification_id, tno, received_at)
       VALUES ($1, $2, now())
       ON CONFLICT (notification_id) DO NOTHING
       RETURNING notification_id`,
      [notification_id, tno]
    );

    if (claim.rowCount === 0) {
      // Already seen this exact delivery — ack, don't reprocess
      await client.query('COMMIT');
      return res.status(200).send('already processed');
    }

    // 3. Only touch the order if it isn't already in the target state
    const order = await client.query(
      'SELECT status FROM orders WHERE tno = $1 FOR UPDATE',
      [tno]
    );

    if (order.rows[0]?.status === 'paid' && status === 'success') {
      // A second, distinct success notification for an already-settled order
      await client.query('COMMIT');
      return res.status(200).send('order already settled');
    }

    if (status === 'success') {
      await client.query("UPDATE orders SET status = 'paid' WHERE tno = $1", [tno]);
      await enqueueFulfillment(tno); // hand off to a queue, don't run inline
    }

    await client.query('COMMIT');
    return res.status(200).send('ok');
  } catch (err) {
    await client.query('ROLLBACK');
    console.error('webhook processing failed', err);
    return res.status(500).send('internal error');
  } finally {
    client.release();
  }
});

Two details do most of the real work here. The unique constraint on notification_id makes the "have I seen this exact delivery before" check atomic — two near-simultaneous deliveries racing each other will have exactly one win the insert. Checking the order's current status before mutating it protects you even against a second, distinct notification for the same transaction, not just an exact duplicate. Both checks matter; relying on only one leaves a gap.

Common Mistakes That Break Idempotency

Most of the duplicate-processing incidents we've seen — ours and other teams' — trace back to one of these:

• Checking idempotency after running business logic, not before. Credit first, log it after, and a crash in between leaves you with no record and a half-finished operation.
• Keying only on the order number, not the notification ID. A single order can legitimately get more than one distinct notification over its life — a payment, then later a refund — so deduplicating on order number alone can silently drop one you needed to act on.
• Doing slow work inline before responding. An email or a third-party call made synchronously inside the handler is exactly what causes the timeout that triggers the next retry — you end up manufacturing the duplicates you're trying to avoid.
• No row-level locking around the status check. Without something like FOR UPDATE, two near-simultaneous deliveries can both read "not yet paid" before either writes, and both proceed to update.
• Verifying the signature after touching the database. Beyond the security hole, it means malformed or unauthenticated requests can corrupt state before you've confirmed they're legitimate.
• Not persisting the raw payload. When something goes wrong, the original notification body is the difference between a five-minute diagnosis and an afternoon of guessing.

Our broader notes on general API error-handling practices cover a lot of the same discipline from the outbound-call side, if you're building both directions of this integration at once.

Respond Fast, Process Async

The single biggest improvement we made to our own handler wasn't a clever idempotency trick — it was moving everything slow out of the request path. Verify the signature, claim the notification ID, update the order's status, return 2xx. That's it. Fulfillment, confirmation emails, inventory updates, anything calling out to another service — all of that goes to a queue and gets processed afterward, outside the gateway's timeout window entirely.

This matters because a slow handler doesn't just risk one timeout — it turns every delivery into a near-miss, with the gateway's timeout and your processing time racing each other on every notification. Decoupling "acknowledge receipt" from "finish the business logic" removes that race completely, and keeps the idempotency check itself fast and simple, no longer entangled with whatever the fulfillment pipeline is doing that day.

Frequently Asked Questions

Is webhook delivery on the first attempt guaranteed?

No. Delivery depends on the state of your server and the network path at that exact moment, both outside the gateway's control. That's why retries and idempotent handling exist rather than being optional extras.

How many times will a failed notification actually be retried?

A bounded number of times over a bounded window, with the gap between attempts growing after each failure. If your endpoint is down past that window, pull the status yourself rather than waiting for an attempt that isn't coming.

What status code should my endpoint return to stop retries?

A clean 2xx. A 4xx, a 5xx, a timeout, or no response at all is read as a failed delivery and queued for another attempt.

What happens if my server is down for the entire retry window?

The transaction itself isn't lost — it's still recorded on the gateway side. Once retries are exhausted, pull the final status through a query or reconciliation endpoint rather than assuming no webhook means no payment.

Can I request that a specific notification be resent manually?

Typically yes, by looking it up through the order or transaction reference and triggering a resend instead of waiting out the automated schedule. With solid idempotency handling, a manual resend is no riskier than any other retry.

Should I check for duplicates before or after verifying the signature?

Signature first, always. You don't want an unauthenticated request touching your idempotency records or database state before you've confirmed it came from the gateway.

If I already poll a status API, do I still need idempotent webhook handling?

Yes. Polling is a useful backstop for the rare case where a webhook never arrives, but it doesn't change the fact that webhooks you do receive can arrive more than once. The two mechanisms solve different problems.

If you're building out your own webhook handler against SMARTPAY's notification flow and want a second set of eyes on the idempotency design before it ships, reach out on Telegram at @LouisSMARTPAY or by email at [email protected].