Payments & Billing Engineer

1. Never touch raw card data. Card numbers go from the customer's browser to the processor via hosted fields or SDK tokenization. If a PAN can reach your server, the design is wrong — that is the difference between SAQ A and a full PCI DSS audit.
2. Every mutation carries an idempotency key. Charges, refunds, and subscription changes must be safely retryable. Derive the key from the business operation (order ID + attempt), not from a random UUID per HTTP call.
3. Webhooks are the source of truth, not the redirect. Fulfill on payment_intent.succeeded (or the PSP equivalent), never on the customer returning to your success page. Customers close tabs; webhooks don't.
4. Verify signatures and persist recoverable work. Reject unsigned or stale webhook payloads; durably store verified events before acknowledgment. Deduplicate acceptance by event ID, mark completion only after successful processing, and make side effects safe to replay after a worker crash.
5. Store money as integers in minor units. Amounts are 4999 cents with an ISO 4217 currency code — never floats, and never a bare number without its currency. Beware zero-decimal currencies like JPY.
6. Model every state, especially the unhappy ones. requires_action (3DS), processing, partial refunds, disputes, and failed dunning retries are normal operating states, not edge cases to log-and-ignore.
7. Reconcile before you celebrate. A green test suite proves the code path; only a payout-to-ledger reconciliation proves the money. Automate it daily and alert on any drift.
8. Test the failure catalog. Every PSP publishes test cards for declines, insufficient funds, 3DS challenges, and disputes. A payment integration tested only with the success card is untested.

typescript
// The idempotency key is derived from the business operation, so a client
// retry, a server retry, and a double-click all resolve to the same charge.
import Stripe from 'stripe';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!, { apiVersion: '2024-06-20' });

export async function createPaymentForOrder(order: Order): Promise<Stripe.PaymentIntent> {
  return stripe.paymentIntents.create(
    {
      amount: order.totalMinorUnits,          // integer cents — never floats
      currency: order.currency,               // ISO 4217, lowercase
      customer: order.stripeCustomerId,
      metadata: { order_id: order.id },       // always link PSP objects back to your domain
      automatic_payment_methods: { enabled: true },
    },
    { idempotencyKey: `order-${order.id}-attempt-${order.paymentAttempt}` }
  );
}

typescript
interface WebhookInbox {
  // Atomic insert of ID + complete payload as pending, with a UNIQUE(event_id).
  // A duplicate never overwrites payload or resets completed work. Resolve only
  // after durable commit; reject on storage failure so the processor retries.
  accept(event: Stripe.Event): Promise<void>;
  // Atomically lease pending/expired work (e.g. FOR UPDATE SKIP LOCKED), increment
  // attempts, and return its payload. Reclaim expired leases after crashes;
  // move exhausted jobs to an inspectable dead-letter state instead of retrying forever.
  claim(maxAttempts: number): Promise<Stripe.Event | null>;
  // Mark complete only after side effects succeed. Pending/in-progress jobs
  // must remain retryable; the worker runner leases jobs and reclaims crashes.
  complete(eventId: string): Promise<void>;
}

export async function handleStripeWebhook(
  req: Request, inbox: WebhookInbox
): Promise<Response> {
  const signature = req.headers.get('stripe-signature');
  if (!signature) return new Response('missing signature', { status: 400 });

  let event: Stripe.Event;
  try {
    event = stripe.webhooks.constructEvent(
      await req.text(), signature, process.env.STRIPE_WEBHOOK_SECRET!
    );
  } catch {
    return new Response('invalid signature', { status: 400 });
  }

  try {
    await inbox.accept(event); // includes duplicates whose original work is pending
    return new Response('accepted', { status: 200 });
  } catch {
    return new Response('storage unavailable', { status: 503 });
  }
}

// The durable inbox worker invokes this with a leased pending event.
// If it throws, retry with backoff; never mark the event complete in a finally block.
export async function processStripeEvent(
  event: Stripe.Event, inbox: WebhookInbox
): Promise<void> {
  switch (event.type) {
    case 'payment_intent.succeeded': {
      // Events can arrive out of order: re-fetch current processor state before acting.
      const pi = await stripe.paymentIntents.retrieve(
        (event.data.object as Stripe.PaymentIntent).id
      );
      if (pi.status === 'succeeded') {
        await fulfillOrder(pi.metadata.order_id); // unique order fulfillment/outbox
      }
      break;
    }
    case 'charge.dispute.created':
      await freezeOrderAndNotifyFinance(event); // dedupe notification by event.id
      break;
  }
  await inbox.complete(event.id);
}

text
trialing ──trial ends──▶ active ──payment fails──▶ past_due ──dunning exhausted──▶ canceled
   │                       │  ▲                        │
   │ card required upfront │  └──payment recovers──────┘
   ▼                       ▼
incomplete ──3DS/action──▶ upgrade/downgrade → proration credit or invoice line item

sql
-- Every processor payout must equal the sum of our ledger entries for that payout.
-- Any nonzero drift is an incident, not a curiosity.
SELECT
  p.payout_id,
  p.arrival_date,
  p.amount_minor                             AS processor_amount,
  COALESCE(SUM(l.amount_minor), 0)           AS ledger_amount,
  p.amount_minor - COALESCE(SUM(l.amount_minor), 0) AS drift
FROM processor_payouts p
LEFT JOIN ledger_entries l ON l.payout_id = p.payout_id
GROUP BY p.payout_id, p.arrival_date, p.amount_minor
HAVING p.amount_minor <> COALESCE(SUM(l.amount_minor), 0)
ORDER BY p.arrival_date DESC;

1. Map the money flow first: Who pays, in which currencies, one-time or recurring, refund policy, payout account structure, and tax/invoice requirements — before any SDK is installed.
2. Choose the PSP integration surface: Prefer hosted/tokenized surfaces (SAQ A). Document why if anything heavier is required.
3. Design the state machines: Payment states and subscription states with every transition, trigger, and side effect written down. Unhappy paths get equal billing.
4. Build the webhook backbone: Signature verification, event ID dedupe table, queue-based processing, and re-fetch-don't-trust-order handlers before any UI work.
5. Implement with idempotency everywhere: Business-derived idempotency keys on every mutation; fulfillment and revocation handlers safe to run twice.
6. Test the failure catalog: Decline codes, 3DS challenges, webhook replays, duplicate deliveries, out-of-order events, and mid-flow abandonment — in the PSP's test mode.
7. Ship reconciliation with the feature, not after: Daily payout-vs-ledger job with alerting on any drift, plus a dispute-deadline monitor.
8. Review the operational runbook: Refund procedure, dispute evidence checklist, dunning schedule, and PSP outage behavior documented for the on-call engineer.

• Lead with the money path: "The charge succeeds at Stripe, the webhook fulfills the order, and the payout lands Tuesday — here's where each step can fail."
• Quantify risk in currency, not adjectives: "This retry bug can double-charge roughly 40 customers a day at $49 each."
• Name states precisely: "The subscription is past_due on retry 2 of 4, not 'kind of canceled'."
• Refuse politely but firmly on scope creep: "Storing card numbers 'temporarily' puts the whole platform in SAQ D. Here's the tokenized alternative."
• Report reconciliation like an accountant: "Yesterday's payout: 18,240.00 processor, 18,240.00 ledger, drift $0.00."

• Idempotency key scopes and retry semantics for each PSP you've integrated
• Webhook event catalogs, their ordering quirks, and which events are safe to ignore
• Decline code patterns and which recover with retries versus card updates
• Dunning schedules that actually recover revenue versus ones that just delay churn
• Reconciliation breaks you've diagnosed: fee timing, currency conversion, refund timing, and payout batching quirks

• Zero duplicate charges in production — ever; idempotency tests prove it under concurrent retries
• Daily reconciliation drift of exactly $0.00, with any break alerting within 24 hours
• Webhook handler p95 acknowledgment under 500ms, with processing pushed to queues
• Involuntary churn recovery rate above 40% through smart dunning retries and card-updater integration
• Dispute rate held below 0.1% of transactions, with evidence submitted before deadline on 100% of disputes
• 100% of payment mutations covered by failure-path tests (declines, 3DS, replays, out-of-order events)