Creating dynamic XML | sitemaps with Netlify

We all agree, XML sitemap files are important things to have for SEO. They are the way we communicate site architecture to Googlebot and the thing we submit to Search Console for the same reason. They are what we often reference in our Robots.txt file, our instruction file for crawlers.

It's equally clear that dynamic sitemaps are better than static. When you add a new page or post to your site, you don't want to have to be manually updating an XML file, you want any site changes to be automatically recorded.

So, why am I writing about this?

Because there's been a big uptick in the creation of websites made on platforms like Lovable, Replit, Bolt, Webflow - to name only a few of a big number of AI powered tools that make websites and webapps in a very Javascript/React/Typescript heavy way and Netlify is a perfect and preferred cloud hosting platform for these kinds of projects.

But this is not Wordpress. It's not as no-code, user friendly as that and to SEO optimise Javascript heavy web apps on this kind of architecture takes a bit of understanding of the nuance of it.

And I'll talk about broader SEO optimisation in this kind of tech stack in later articles. Here, we're going to be focussed on XML sitemaps only, so let's get into it.

2 Possible Implementation Approaches

Right, so there are two main ways to tackle this and which one you choose depends on how often your content changes.

1/ Serverless Function Method

If your site's content is constantly updating - think e-commerce stores, news sites, or blogs that publish frequently - this is your best bet.

The beauty of serverless functions is they generate your sitemap on-the-fly every time it's requested. So whenever Googlebot comes knocking, it gets the absolute latest version of your site structure. No build required, no deploy needed.

Here's what that looks like in practice:

javascript

// netlify/functions/sitemap.js
exports.handler = async function(event, context) {
  const blogPosts = await getBlogPosts();
  const products = await getProducts();
  
  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yoursite.com</loc>
    <lastmod>${new Date().toISOString()}</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  
  ${blogPosts.map(post => `
  <url>
    <loc>https://yoursite.com/blog/${post.slug}</loc>
    <lastmod>${post.updated_at}</lastmod>
    <changefreq>weekly</changefreq>
  </url>
  `).join('')}
</urlset>`;

  return {
    statusCode: 200,
    headers: {
      'Content-Type': 'application/xml',
      'Cache-Control': 'public, max-age=3600'
    },
    body: sitemap
  };
};

You're basically creating a mini API endpoint that builds your sitemap XML dynamically. Fetch your posts, fetch your products, loop through them, and return properly formatted XML. Simple.

2/ Build-Time Generation Method

Now, if you're running a portfolio site, a documentation site, or anything where content only changes when you deploy, this approach makes more sense.

Frameworks like Next.js and Gatsby can generate your sitemap during the build process. Your content changes when you push new code anyway, so why not generate the sitemap at the same time? It's more efficient and means you're not running serverless functions unnecessarily.

Here's how that looks with next-sitemap:

javascript

// next.config.js with next-sitemap
module.exports = {
  siteUrl: 'https://yoursite.com',
  generateRobotsTxt: true,
  sitemapSize: 5000,
  changefreq: 'weekly',
  priority: 0.7,
  exclude: ['/admin/*', '/privacy'],
  transform: async (config, path) => {
    return {
      loc: path,
      changefreq: config.changefreq,
      priority: config.priority,
      lastmod: new Date().toISOString(),
    };
  },
};

This configuration runs every time you build your site, automatically excluding pages you don't want indexed (like admin areas), and generating a robots.txt file for you as a bonus.

The key difference? One builds on-demand, one builds on deployment. Choose based on your content update frequency.

Configuration Requirements

Alright, so you've got your sitemap generation sorted. But there's a couple of configuration bits you need to nail down or none of this will actually work.

Netlify Redirects

First up, you need to tell Netlify where to send requests for your sitemap. This happens in your netlify.toml file.

toml

[[redirects]]
  from = "/sitemap.xml"
  to = "/.netlify/functions/sitemap"
  status = 200
  force = true

[[redirects]]
  from = "/sitemap-index.xml"
  to = "/.netlify/functions/sitemap-index"
  status = 200

What's happening here? When someone (or more importantly, Googlebot) requests /sitemap.xml, Netlify intercepts that request and routes it to your serverless functon instead.

That force = true bit is important - it means this redirect takes priority over any other routing rules you might have set up. Without it, you might find your sitemap request getting caught by a catch-all redirect, which is a nightmare to debug.

Robots.txt Configuration

Next, you need to actually tell search engines where your sitemap lives. This is straightforward - just update your robots.txt file:

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

This is telling every crawler ("User-agent: *") that they're allowed to crawl everything ("Allow: /"), and here's where to find the sitemap.

Dead simple, but critical. Without this, search engines will eventually find your sitemap through other means, but why make them work harder than they need to?

Best Practices

Right, so you've got the basics down. But there are a few things worth keeping in mind to make sure this actually runs smoothly in production.

Cache your sitemap. You don't want to be regenerating your sitemap XML every single time it's requested - that's wasteful and expensive. Set a reasonable cache header, anywhere from one to twenty-four hours depending on how often your content changes. It's in the example code above (Cache-Control: public, max-age=3600), but worth calling out explicitly.

Handle errors gracefully. If your function tries to fetch blog posts from a database and that call fails, you don't want your entire sitemap to break. Wrap your data fetching in try-catch blocks and have sensible fallbacks. Return at least your core pages even if dynamic content fails to load.

Validate your output. Once you've got this running, submit your sitemap to Google Search Console. It'll tell you immediately if there are any formatting issues or protocol violations. Don't assume it's working - check.

Monitor your function costs. Serverless functions aren't free once you exceed the generous free tier limits. Keep an eye on execution times and how often your sitemap is being hit. If you're getting hammered, increase your cache duration.

Segment large sitemaps. If your site has more than 50,000 URLs, you'll need to split your sitemap into multiple files and use a sitemap index file. This is a protocol requirement, not a suggestion.

Expected Outcomes

Once you've got all this configured and deployed, you should never have to think about your sitemap again.

New blog post published? It's in the sitemap automatically. Product removed from your store? Gone from the sitemap. Page added to your React app? It's there on the next build or request, depending on which method you've chosen.

The whole point of this setup is that search engines always have the most current view of your site structure. Better indexing, potentially better SEO performance, and zero manual maintenance on your part.

That's the dream, really.

Case Study: TypeScript and React Implementation

Let me walk you through a real example. This is from a portfolio site I built using TypeScript, React, and Vite and it shows exactly the kind of problem you'll run into if you're not careful.

The Problem: Hardcoded Routes

So here's what I did initially, and it was stupid. I hardcoded all my routes directly into the sitemap function:

javascript

// The problematic approach - hardcoded and error-prone
const staticRoutes = [
  '', '/cv', '/projects', '/blog', 
  '/reddit-extractor',  // Page since deleted
  '/colour-palette-generator'  // Page since deleted
];

See those comments? Those are pages that don't exist anymore. I deleted them from my React app but forgot to update the sitemap function. So Google was trying to index pages that 404'd. Not ideal.

And every time I added a new page to my site, I had to remember to manually update this array. Which defeats the entire point of having a dynamic sitemap in the first place.

The Solution: Automated Route Extraction

The fix was to make the sitemap function actually read my React Router configuration instead of maintaining a separate hardcoded list.

Here's how it works: a Node.js script runs before every build, parses my App.tsx file, extracts all the routes, and saves them to a JSON file that the sitemap function can then read.

Route Extraction Script

javascript

// build-routes.js
import fs from 'fs';
import path from 'path';

const appContent = fs.readFileSync('src/App.tsx', 'utf-8');

// Extract all <Route path="..." /> declarations
const routeRegex = /<Route\s+path="([^"]+)"/g;
const routes = [];
let match;

while ((match = routeRegex.exec(appContent)) !== null) {
  const route = match[1];
  
  // Filter out dynamic routes (:slug), catch-all (*), and function routes
  const isDynamicRoute = route.includes(':');
  const isCatchAll = route === '*';
  
  if (!isDynamicRoute && !isCatchAll) {
    routes.push(route === '/' ? '' : route);
  }
}

// Write manifest for the sitemap function to consume
fs.writeFileSync('public/routes.json', JSON.stringify([...new Set(routes)], null, 2));

This script looks for anything matching <Route path="..." /> in my App.tsx file, filters out dynamic routes (like /blog/:slug) and catch-all routes, then writes the clean list to public/routes.json.

To make this run automatically on every build, I just added it to my build script in package.json:

json

{
  "scripts": {
    "build": "node build-routes.js && vite build"
  }
}

Now every time I build, routes get extracted first, then Vite does its thing.

The Sitemap Function

Here's the actual Netlify function that generates the sitemap. It reads that routes.json file we just created, plus a separate posts.json file for blog posts:

typescript

// netlify/functions/sitemap.ts
import { Handler } from '@netlify/functions';
import fs from 'fs';
import path from 'path';

export const handler: Handler = async (event, context) => {
  const baseUrl = 'https://mikehiggins.me.uk';
  
  // Load static routes from the generated manifest
  let staticRoutes: string[] = [];
  try {
    const routePathsToTry = [
      path.join(process.cwd(), 'public/routes.json'),
      path.join(__dirname, '../../public/routes.json'),
      path.resolve('/var/task/public/routes.json')
    ];
    
    for (const p of routePathsToTry) {
      if (fs.existsSync(p)) {
        staticRoutes = JSON.parse(fs.readFileSync(p, 'utf-8'));
        break;
      }
    }
    
    // Fallback to basic routes if manifest is missing
    if (staticRoutes.length === 0) {
      staticRoutes = ['', '/cv', '/projects', '/blog'];
    }
  } catch (error) {
    console.error('Error reading routes manifest:', error);
    staticRoutes = ['', '/cv', '/projects', '/blog'];
  }
  
  // Load blog posts dynamically
  let blogRoutes: string[] = [];
  try {
    const pathsToTry = [
      path.join(process.cwd(), 'public/blog/posts.json'),
      path.join(__dirname, '../../public/blog/posts.json'),
      path.resolve('/var/task/public/blog/posts.json')
    ];
    
    let postsData: any[] = [];
    for (const p of pathsToTry) {
      if (fs.existsSync(p)) {
        postsData = JSON.parse(fs.readFileSync(p, 'utf-8'));
        break;
      }
    }
    
    if (postsData.length > 0) {
      blogRoutes = postsData.map((post: any) => `/blog/${post.slug}`);
    }
  } catch (error) {
    console.error('Error reading blog posts:', error);
  }
  
  const allRoutes = [...staticRoutes, ...blogRoutes];
  
  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  ${allRoutes.map(route => `
  <url>
    <loc>${baseUrl}${route}</loc>
    <lastmod>${new Date().toISOString().split('T')[0]}</lastmod>
    <changefreq>${route === '' ? 'daily' : 'weekly'}</changefreq>
    <priority>${route === '' ? '1.0' : '0.8'}</priority>
  </url>`).join('')}
</urlset>`;

  return {
    statusCode: 200,
    headers: {
      'Content-Type': 'application/xml',
      'Cache-Control': 'public, max-age=3600'
    },
    body: sitemap,
  };
};

Notice all those different paths it tries? That's because Netlify's serverless environment can have different working directories depending on how it's deployed. The function tries multiple locations until it finds the files. And if it can't find them at all, it falls back to a basic set of routes rather than completely breaking.

Critical Configuration Gotchas

There are two configuration details that'll absolutely trip you up if you get them wrong.

1/ File Bundling

By default, Netlify functions can't access files in your public folder. They just... won't be there when the function runs. You need to explicitly tell Netlify to bundle those files with your function.

This goes in your netlify.toml:

toml

[functions]
  directory = "netlify/functions"
  node_bundler = "esbuild"
  included_files = ["public/blog/posts.json", "public/routes.json"]

2/ Redirect Priority

If you're building a React SPA (and you probably are), you'll have a catch-all redirect that sends everything to index.html so React Router can handle routing. Something like this:

/* /index.html 200

The problem? That'll catch your /sitemap.xml request too, and you'll get your React app instead of your actual sitemap.

The fix is to put your sitemap redirect first with a forced status. In your _redirects file:

/sitemap.xml /.netlify/functions/sitemap 200!
/* /index.html 200

What This Gets You

With this setup running, here's what changes:

Add a new page in React Router? It's in the sitemap automatically on the next build.
Delete a page? Gone from the sitemap immediately.
Publish a new blog post? Shows up in the sitemap without any manual intervention.
Your App.tsx is the single source of truth - there's no second list to maintain.
If your App.tsx is malformed, the build fails early rather than silently creating a broken sitemap.

Basically, you never have to think about your sitemap again. It just works.

Wrapping Up

Look, dynamic sitemaps aren't complicated once you understand the pieces. The serverless approach works brilliantly on Netlify because it scales automatically and only runs when it needs to.

Once you've got this deployed, don't forget to actually submit your sitemap to Google Search Console and Bing Webmaster Tools. No point having a perfect sitemap if you're not telling search engines where to find it.