Robots.txt and Sitemap.xml in Next.js: One Declared Date, 420 Routes

In the Next.js App Router, robots.txt and sitemap.xml aren't static files — they're generated by app/robots.ts and app/sitemap.ts, typed against MetadataRoute.Robots and MetadataRoute.Sitemap, built at deploy time. Wiring that up is a few lines. The part that actually matters is what lastModified says on each URL, because a sitemap that claims every page changed on every deploy is a claim Google can check, and once it catches the lie it stops trusting the field — which is a real defect this storefront shipped and later fixed, audited below with the code and the test that now guards it.

Two files, generated not written

robots.ts is the smaller of the two — a rule set plus a pointer to the sitemap:

// src/app/robots.ts
export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: "/",
      disallow: [
        "/dashboard",
        "/login",
        "/signup",
        "/forgot-password",
        "/reset-password",
        "/auth",
      ],
    },
    sitemap: `${BASE_URL}/sitemap.xml`,
  };
}

Six routes disallowed, matching the account/auth surface, and nothing else — the whole marketing site (templates, blog, pricing, docs, every category hub) is left crawlable. There's no host: directive here on purpose: it's a Yandex-only extension Googlebot ignores outright, and Search Console flags it as "Rule ignored by Googlebot" if you add it expecting it to declare a canonical domain.

sitemap.ts does the real work. This storefront's current build emits one entry per URL across four tiers — static marketing pages, the seven category/framework hubs, 111 product detail pages, and every blog post — for 428 routes total, 420 of them statically prerendered and 8 rendered on demand (the account/auth surface robots.ts excludes).

disallow and noindex are not the same lever, and this codebase uses both

It's easy to assume blocking a route in robots.txt and marking it noindex do the same job. They don't, and mixing them up in the wrong direction breaks the one you actually needed:

• noindex requires the crawler to fetch the page to read the tag. Block that same page in robots.txt and the crawler never sees the noindex directive at all — if anything external links to the URL, Google can still list a bare URL with no title or snippet, which is usually the opposite of what you wanted.
• disallow stops the fetch entirely. It saves crawl budget but tells you nothing about whether the URL is indexed — Google can still know the URL exists from external links even though it was never crawled.

This codebase actually applies both, on different routes, for different reasons. /saved (the wishlist page) ships a noindex meta tag and nothing in robots.ts:

// src/app/saved/page.tsx
robots: { index: false, follow: true },

It's deliberately crawlable — the comment above it explains why: it's a client-rendered empty shell with no server data to show a crawler, so noindex is the right tool for "don't show this specific empty page in results," not "don't fetch it." /dashboard, /login, /signup and the rest go the other way — blocked in robots.ts, no noindex tag — because there's no content-discovery reason to spend crawl budget on an authenticated shell or a bare form, and nothing external links to them anyway. Neither pattern is universally correct; picking between them is about whether the page has content worth a noindex tag deciding not to show, or nothing worth fetching in the first place.

The defect: a sitemap that claimed everything changed, on a date nothing did

Here's sitemap.ts's actual lastModified logic for a product page:

// src/app/sitemap.ts
export const STOREFRONT_COPY_REVISED = new Date("2026-08-21T00:00:00Z");

function newer(a: Date, b: Date): Date {
  return a.getTime() >= b.getTime() ? a : b;
}

function productLastModified(product: TemplateProduct): Date {
  const newest = product.changelog[0]?.date;
  return newest ? new Date(`${newest}T00:00:00Z`) : new Date();
}

// ...
const templateRoutes: MetadataRoute.Sitemap = catalog.map((product) => ({
  url: `${BASE_URL}/templates/${product.slug}`,
  lastModified: newer(productLastModified(product), STOREFRONT_COPY_REVISED),
  changeFrequency: "monthly",
  priority: 0.8,
}));

STOREFRONT_COPY_REVISED is a declared constant, not new Date() at build time — and that distinction is the whole post. An earlier version of this sitemap used the build clock for every lastModified, which is the default mistake: it's the easiest thing to write, it's wrong the moment more than one page exists, and it's wrong in a way nobody notices locally because every date just looks "recent."

The actual incident: an August 2026 SEO pass rewrote the <title>, <h1> and meta description on all 111 product pages and the copy on four category hubs — without releasing a single new template version, so no changelog entry recorded it. A changelog date answers "when did this product last ship a feature," which is the right question for a buyer and the wrong one for a crawler deciding whether to re-fetch a page. The sitemap kept reporting July dates for pages whose actual rendered output had changed weeks later, on a date no field in the catalog captured at all.

STOREFRONT_COPY_REVISED is the fix: a single declared date, bumped only when a change alters what the rendered page actually says, newer()-combined against each product's own changelog date. It's deliberately not new Date() — a clock reading here is exactly the "100+ URLs changed on every deploy" claim the whole mechanism exists to stop making.

The test that keeps it from regressing back to the clock

The failure mode this guards against — swapping a declared date back for new Date() — wouldn't show up in a manual read of the diff; both look like valid Date objects. What catches it is a very specific assertion:

// src/data/__tests__/internalLinking.test.ts
it("every product and hub date is a declared day, not a clock reading", () => {
  // A declared date is midnight UTC; `new Date()` at build time never is.
  // This catches the mistake even on a day when the constant happens to
  // equal today.
  for (const entry of tiered) {
    const iso = new Date(entry.lastModified!).toISOString();
    expect(iso.endsWith("T00:00:00.000Z"), `${entry.url} → ${iso}`).toBe(
      true,
    );
  }
});

A declared "2026-08-21T00:00:00Z" always serializes to midnight UTC. new Date() at whatever second the build ran essentially never does — the test doesn't need to know today's date to catch the regression, it just checks the shape. There's a second test alongside it that checks the actual value: every product/hub lastModified must equal the later of its real changelog date and STOREFRONT_COPY_REVISED, computed independently in the test rather than imported from the sitemap code, so a bug in newer() itself would still get caught.

Troubleshooting

FAQ

Do I need both robots.txt and sitemap.xml? They do different jobs. robots.txt tells crawlers what not to fetch; the sitemap tells them what exists and when it last changed, to help them prioritize re-crawls. A site can technically run with neither, but skipping the sitemap means Google has to discover every page by links alone, and skipping robots.txt means no crawl-budget control over routes you never want indexed.

What should lastModified actually be set to? The real date the page's rendered content last changed — not the build timestamp. If nothing tracks that per-page, the honest fallback is a manually bumped constant for storefront-wide changes, combined with whatever real per-item date already exists (a changelog, a updatedAt column).

Why does Google ignore my host: directive in robots.txt? It was always a Yandex extension, never part of the actual robots exclusion standard, and Googlebot has never read it. Search Console's "Rule ignored by Googlebot" warning on that line is expected, not a bug to chase.

Should every page really be in the sitemap? No — only indexable ones. This storefront's sitemap excludes every route robots.ts disallows (the account/auth surface) for the same reason: no crawl value, no index value, no reason to list it.

Templates in this post

ASoc Compound is an investment-platform landing page, ASoc Cortex an AI agency template, and ASoc Cover an insurance-company page — three of the 111 product pages whose lastModified date is computed by the exact logic audited above.

Browse the full sets: Next.js landing page templates, Tailwind landing page templates. For the broader route census these two files sit inside, see the Next.js routing audit.