That choice has two traps. This post is about both, and about the queue that grew out of them.

Trap 1: Cloudflare Pages decides you are an SPA

A Pages project with no top-level 404.html is treated as a single-page application. Every unmatched path gets index.html with HTTP 200. Convenient for client-side routing. Fatal for observability, because a request for an asset chunk that no longer exists also gets index.html with HTTP 200.

For four months, /tools-assets/ chunks that had been renamed by a build were being served as the app shell. The deploy was green. No monitor complained. No status code was wrong.

Adding public/404.html turned the project into a normal static site: unmatched paths are real 404s. That fixed the assets and created the opposite problem. Any app route that is not prerendered to its own file, and has no explicit rewrite, is now a hard 404 for real users. Nothing in the build catches that by default.

So the build catches it on purpose:

// scripts/check-spa-routes.mjs: fail the build if an App.tsx route would 404
const declared    = routesIn('src/App.tsx');            // <Route path="...">
const prerendered = routesIn('scripts/prerender.mjs');  // ROUTES = [...]
const rewritten   = rules200In('public/_redirects');    // "/path /index.html 200"

// A rule covers a route if it matches exactly, or a splat rule is a prefix.
const covered = (route) =>
  rewritten.some((rule) =>
    rule.endsWith('/*') ? route.startsWith(rule.slice(0, -1)) : rule === route);

const missing = declared.filter((r) => !prerendered.includes(r) && !covered(r));
if (missing.length) {
  console.error('Routes with no prerender and no 200 rewrite:', missing);
  process.exit(1);
}

Add a route and forget both, and the deploy fails instead of the page.

Trap 2: IndexNow believes you

IndexNow lets you tell Bing and Yandex about a URL instead of waiting for a crawl. The naive integration submits every generated URL the moment it is written. Two things go wrong:

1. The page is not deployed yet. The commit is pushed, but Pages has not built it.

2. Because of trap 1, a status check proves nothing. Before 404.html existed, the site returned 200 for every path, including pages that did not exist.

So the queue does not check the status. It fetches the body and looks for the one thing only a real generated page contains: its own canonical tag.

async function checkLiveness(slug: string): Promise<boolean> {
  const res = await fetch(`${SITE_BASE}/${slug}`, {
    redirect: 'follow',
    signal: AbortSignal.timeout(10_000),
  });
  if (!res.ok) return false;
  const html = await res.text();
  return html.includes(`<link rel="canonical" href="${SITE_BASE}/${slug}/"`);
}

The queue

The queue is a JSON file on a persistent volume with two lists, pending and submitted, and a cron every three hours:

export async function processIndexNowQueue() {
  const queue = loadQueue();
  const live: string[] = [];
  const remaining: PendingEntry[] = [];

  // 10 at a time, 10s timeout each
  for (const chunk of chunks(queue.pending, LIVENESS_CONCURRENCY)) {
    const results = await Promise.allSettled(chunk.map(async (e) => ({ e, ok: await checkLiveness(e.slug) })));
    for (const r of results) {
      if (r.status !== 'fulfilled') continue;
      const { e, ok } = r.value;
      if (ok) { live.push(e.slug); continue; }
      e.checkCount++;
      if (e.checkCount >= MAX_CHECK_COUNT) {         // 56 checks, about 7 days
        logger.warn({ slug: e.slug }, 'never went live; dropped');
      } else {
        remaining.push(e);
      }
    }
  }

  if (live.length) {
    try {
      await submitToIndexNow(live.map((s) => `${SITE_BASE}/${s}`)); // batches of 10,000
      queue.submitted.push(...live.map((slug) => ({ slug, submittedAt: now() })));
    } catch (err) {
      // nothing is marked submitted on hope
      remaining.push(...queue.pending.filter((e) => live.includes(e.slug)));
    }
  }

  queue.pending = remaining;
  saveQueue(queue);
}

Four properties fall out of that:

• Deduplicated. enqueueSlugs skips anything already pending or submitted.

• Bounded. A slug that never goes live is dropped after 56 checks with a warning, not retried forever.

• Honest on failure. If the POST fails, the live slugs go back to pending. The submitted list only ever contains URLs the API accepted.

• Batched. Up to 10,000 URLs per request, with three retries five seconds apart.

The part with no API

There is no IndexNow removal call. When a page is unpublished, the worker does the only thing that works:

export function forceRecheck(slug: string): void {
  const queue = loadQueue();
  queue.submitted = queue.submitted.filter((e) => e.slug !== slug);  // forget it was submitted
  saveQueue(queue);
  enqueueSlugs([slug]);                                               // check it again
}

The next liveness pass fetches the tombstone, finds no canonical tag, counts a miss, and after 56 misses drops it. The search engines re-crawl a 404 on their own schedule. Slow, but it is the mechanism that exists.

Takeaways

• A static site with an SPA catch-all lies to every status check you write. Look for something only the real page has.

• Never tell a search engine about a URL you have not fetched yourself.

• Keep a submitted list that only the API's acceptance can write to.

• Put the route check in the build, because a green deploy is not evidence.

The other half of this engine, the AI quality gate that decides which pages exist at all and the deterministic per-slug layout variation that keeps them from looking like doorway pages, is in the full write-up: How I built a 1,898-page programmatic SEO engine with AI quality gates, static HTML and IndexNow.

I build programmatic SEO pipelines and audit production deployments. kiragu@alkenacode.dev, or a fixed-price engagement on Contra. Case studies and more of this work: https://kiragu.alkenacode.dev

An M-Pesa webhook arrives. The interceptor claims the key. The payment row inserts cleanly into Postgres. The transaction commits.

Then, before the subscription end date is extended, before the receipt SMS goes out, before the RADIUS packet reaches the router, the container gets recreated by a deploy.

The ledger says paid. The router says no. A subscriber in Kisumu is offline with a receipt on their phone, and the ISP owner is calling.

This is the failure mode I spent the most time on in FyberPay, a billing and network platform for ISPs that runs eight payment rails behind one interface. The fix is old and boring and it works: a transactional outbox, drained by workers using SELECT ... FOR UPDATE SKIP LOCKED.

The rule: nothing important happens inside the request

The request handler is allowed to do exactly one thing with side effects: write to the database. Everything that talks to the outside world (SMS gateway, email, the router's RADIUS port) is recorded as an intent in the same transaction, and executed later by something that can be retried.

// payments.service.ts (simplified)
await this.db.transaction(async (tx) => {
  const payment = await tx.payments.insert({ gateway, receiptNumber, amount, subscriberId });

  await tx.outboxEvents.insert({
    type: 'PAYMENT_CONFIRMED',
    aggregateId: payment.id,
    payload: { subscriberId, amount, tenantId },
    status: 'pending',
  });
});
// COMMIT. Both rows exist, or neither does.

That "or neither does" is the entire point. There is no state of the world where the payment exists and the intent to act on it does not.

The claim query

Workers poll the outbox. The naive version, SELECT * FROM outbox_events WHERE status = 'pending' LIMIT 1, falls apart the moment you run two workers: both read the same row, both send the SMS.

Postgres has a three-word fix:

UPDATE outbox_events
SET status = 'claimed', claimed_at = now(), attempts = attempts + 1
WHERE id = (
  SELECT id FROM outbox_events
  WHERE status = 'pending' AND next_attempt_at <= now()
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

FOR UPDATE locks the row the subquery picks. SKIP LOCKED tells every other worker to ignore rows someone else already holds and take the next one. Ten workers hit the table at the same instant, ten different rows come back, nobody blocks, nobody double-sends.

The UPDATE ... RETURNING wrapper matters too: claiming and reading are one statement, so there is no window where a worker has read a row but not yet marked it.

What runs off the event

In FyberPay a PAYMENT_CONFIRMED event fans out to three listeners:

• Subscription extension. Credit-aware, so a partial payment does the arithmetic instead of failing.

• Receipt notification. SMS or email, through whichever of nine SMS gateways the tenant configured.

• RADIUS Change-of-Authorization. A hand-rolled UDP packet with an MD5 authenticator, sent to the MikroTik NAS so the subscriber is online within seconds instead of at their next session.

Each listener is independent. If the SMS gateway is down, the router still gets its packet.

When a worker dies mid-way

This is the case the outbox exists for, so look at it closely.

worker A claims row 9931  (status = claimed, claimed_at = 10:41:03)
worker A sends the SMS       ok
worker A opens a UDP socket to the NAS
worker A is killed           (deploy recreated the container)

Row 9931 is now claimed with a claimed_at that is getting older. A reaper job runs every minute:

UPDATE outbox_events
SET status = 'pending', next_attempt_at = now() + (interval '30 seconds' * attempts)
WHERE status = 'claimed' AND claimed_at < now() - interval '2 minutes';

The row goes back in the queue with exponential backoff. Another worker picks it up. The SMS goes out a second time.

Yes, a second time. And that is fine, because of a rule that is easy to state and easy to forget:

The outbox gives you at-least-once. Every listener has to be safe to run twice.

A duplicate SMS is a minor annoyance. A duplicate CoA packet for the same session is a no-op on the router. A duplicate subscription extension would be a real bug, which is why the extension listener checks the payment id it has already applied before it moves the end date. The outbox does not make your side effects idempotent; it makes sure they eventually run, and it forces you to make them idempotent yourself.

What the ledger and the outbox are not

The outbox is layer three of three. It sits underneath two layers that keep the ledger exactly-once:

1. A Redis interceptor keyed by principal, tenant, route, body hash and the client's idempotency key, claimed with SET NX.

2. A UNIQUE (gateway, receipt_number) constraint on the payments table, so a replayed receipt rolls back before it credits anything.

The outbox never has to be exactly-once precisely because those two layers already are. Redis is speed. Postgres is truth. The outbox is durability.

Numbers, for scale

In production this runs as 75 outbox listeners across 49 queue processors in 17 Docker services, on a schema of 263 hand-written SQL migrations with about 8,200 automated tests behind it.

If you take one thing

Do not send the SMS in the request handler. Write a row. Let a worker send the SMS. Give the worker SKIP LOCKED and a reaper, and make every listener safe to run twice.

The interceptor and the unique-constraint layers, and the circuit breakers that keep a wrong PIN from looking like an outage, are in the full write-up: Building 3-layer idempotency and webhook resilience across 8 payment gateways.

I build and audit payment backends like this one. kiragu@alkenacode.dev, or a fixed-price engagement on Contra. Case studies and more of this work: https://kiragu.alkenacode.dev