The published site and every signed-in workspace returned a server error. No customer data was lost, altered, or exposed: the failure was in the server bundle that renders pages, not in the database or its access policies.
A markdown renderer and a browser-only charting library were reachable from a server-rendered route module. The production worker evaluates every route module at boot, so the bad import chain took down the whole bundle rather than one page. A stale cached service worker then kept serving the broken shell to browsers after the fix shipped, which is why recovery took a second publish and a cache purge.
Severity: Sev-1 — full public outage · Reported by: Owner (Joe Barnhill) after a failed page load on the published site
Every failure, diagnosis, fix, publish, and cache refresh, in the order it happened. Times are Central and elapsed markers count from the first failed request.
A publish finished successfully at the build step, but the production worker could not evaluate its server bundle. Every route, including the marketing home page, began returning a server error.
500 on GET / — worker boot error, no route handlers registered
A failed page load was reported by the owner and independently confirmed against the health endpoint from outside the network.
GET /healthz — 500 (expected 200 ok)
Local development still worked because the dev server evaluates modules lazily. Reproducing against a production build surfaced the real error at boot instead of at request time.
ReferenceError: window is not defined — during route module evaluation
The markdown renderer used by help and guide pages was imported at module scope, so the server evaluated it on boot. It now loads lazily in the browser only.
A dashboard chart module pulled a browser-only library into a shared utility that server-rendered routes imported. Shared values were split into a server-safe module so the heavy library stays on the client.
The two fixes were published together. The server came back, but browsers with a previously cached service worker still rendered a broken shell.
GET /healthz — 200 ok; hard-reloaded browsers fine, returning browsers still broken
The offline cache was serving the pre-outage application shell and its now-missing asset URLs, so a healthy server still produced a blank page for anyone who had visited before.
Workbox: precache entry not found — cached shell references removed asset hashes
The cache revision was bumped, oversized entries that had been silently failing to precache were dropped, and the worker now claims clients and deletes caches from older revisions on activation.
The new build was published and the edge cache was purged so that the HTML document and the service worker script were both refetched from origin rather than served from an intermediate cache.
A diagnostic endpoint was added that imports every server-rendered route module and reports any that fail to evaluate or that pull in a browser-only library — turning this class of outage into a pre-publish check instead of a post-publish surprise.
GET /routesz — all route modules evaluated, no browser-only leaks
Sign-in, workspace switching, the ledger, invoices, and reconciliation were exercised on the published site. Record counts matched pre-outage values and access policies behaved as expected.
Health checks, the route boot probe, and a post-publish smoke test all passed repeatedly with no recurrence, and the incident was closed.
Post-publish smoke test — all checks passed
The safeguards added so this class of failure is caught before a publish reaches you.
The /routesz probe runs as part of the post-publish smoke test, so a server bundle that cannot boot is caught immediately.
The offline cache revision changes with every build, so a new publish can never be shadowed by an old cached shell.
Server-side rendering is exercised against a production build, not only the development server, because the development server hides boot-time failures.
An outside monitor pinging the health endpoint on a short interval so detection does not depend on someone loading a page.
Questions about this incident, or want the same detail for a future one? See Security & Trust for how we handle notification and review, or reach us through the security contact form there.