The Next.js bug that hid my site from every crawler

I set out to write seven long guides for My Camino Guide. Planning pages, a packing list, a route chooser. Standard content marketing: about 16,000 words, some structured data, link it all up, then wait.

Before writing a word I wanted to see what those guides would be landing on top of. That's where the afternoon went sideways.

The 131-byte body

I'd just built the first two guides. They rendered fine in the browser. The build was green. I ran a script to confirm the primary keyword showed up in the h1, which is exactly the kind of box ticking that makes you feel productive, and the script threw an exception because it couldn't find an h1.

So I curled the page and stripped out the script tags to see what was actually in the body.

<body class="__className_f367f3">
  <!--$!--><template data-dgst="BAILOUT_TO_CLIENT_SIDE_RENDERING"></template><!--/$-->
</body>

That's the whole thing. 131 bytes. No heading, no paragraphs, no content of any kind. Every word on that page existed only inside the React Server Components payload, encoded as escaped JSON inside a script tag.

I assumed I'd broken it that morning. So I checked the home page. Same. About page, routes page, same. Then I checked the blog posts, thirty-eight of them, written over the previous six months. Same.

The entire site had been serving an empty body to anything that didn't execute JavaScript. For months.

Why useSearchParams took down every route

The root layout looked like this:

<Suspense fallback={null}>
  <PostHogProvider>{children}</PostHogProvider>
</Suspense>

And inside PostHogProvider:

<PHProvider client={posthog}>
  <PostHogPageView />
  {children}
</PHProvider>

PostHogPageView is an analytics component. It renders nothing. Its entire job is to call usePathname() and useSearchParams() and fire a pageview event.

useSearchParams() is the problem. In a statically rendered route, calling it forces React to bail the nearest Suspense boundary out to client-side rendering. Next.js documents this behavior in Entire page deopted into client-side rendering, and honestly it's reasonable, because the search params aren't knowable at build time.

What's easy to miss is the blast radius. My nearest boundary was the one in the root layout, and that boundary wrapped {children}. So a component that renders null took the entire application down with it. Every route, every page, every blog post.

The four-line fix

Give the component that suspends its own boundary, and the bailout stays where it belongs:

<Suspense fallback={null}>
  <PostHogPageView />
</Suspense>
{children}

Body markup on the guide page went from 131 bytes to 36,891.

This isn't a PostHog problem, by the way. Any client component reading useSearchParams() does the same thing. Analytics wrappers just happen to live in the root layout, which is where the damage is maximized. Next.js has a whole error page for the missing boundary case, and I'd read it at some point, filed it under "input validation thing," and moved on.

Why every check stayed green

This is the part I keep turning over.

The build passed. TypeScript passed. The pages worked perfectly in a browser, because a browser runs the JavaScript and hydrates the whole thing in milliseconds. Lighthouse would have scored it well. Every automated check I had was green, and had been green the entire time the site was invisible.

Google does execute JavaScript, so this wasn't total darkness. But client-only rendering puts you in a slower, lower-priority queue for indexing, and every crawler that doesn't run JS saw nothing at all. That includes a good number of the AI ones, which matters more every month. I spend most of my working life thinking about how machines read developer surfaces. It's the entire premise behind axrank.ai. I still missed it on my own site.

The only reason I found it is that I checked the rendered HTML instead of the rendered page. Those are different things, and I'd been treating them as the same thing.

The curl check I run now

Embarrassingly simple:

curl -s https://yoursite.com/some-page | grep -c '<h1'

If that returns zero, stop and find out why.

There's a second check worth having. My seven guides are registered in one file, and that registry feeds the hub page, the sitemap and the llms.txt. Which means a page that isn't in the registry is invisible to all three. It builds, it renders, you can visit it, and it never gets crawled. That's the same shape as the bug I'd just spent an afternoon on: silent, passive, and invisible to every check that was already running.

So there's a pre-commit hook now that walks the app directory, derives every route, and refuses the commit if one isn't accounted for. It only runs when app routes changed, so it stays out of the way.

It caught its first real omission about twenty minutes after I wrote it, when I added the hub page itself and forgot to register it.

Verify the artifact, not the experience

Three things I'd tell myself six months ago.

Verify the artifact, not the experience. Your page working in your browser tells you the JavaScript is fine. It tells you nothing about what a crawler receives, and those two things can differ completely.

Automate the check for the bug you just fixed. Not because it'll come back in the same form, but because the class of bug will. Silent failures need active detection, and the fact that nothing was complaining was the entire problem.

If this is your kind of problem, I write up more of them at quintonwall.com, and I post build videos on YouTube at @seeqcode.

And be suspicious of green. Every check I had was passing while the site served nothing. Green means the things you thought to check are fine. It says nothing at all about the things you didn't.