Last Time

In the previous article, "Adding robots.txt and sitemap.xml to a blog created with Next.js and microCMS at lightning speed", I explained how to create a static sitemap.xml. Here's the sitemap.xml we created:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://zubora-code.net/sitemap-0.xml</loc></sitemap>
</sitemapindex>

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
<url><loc>https://zubora-code.net/ja</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/en</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/ja/aboutme</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/en/aboutme</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/ja/privacy_policy</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/en/privacy_policy</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
</urlset>

This time, I'll explain how to handle dynamically generated URLs (in the case of this blog, URLs under /articles and /tags). The basic approach is as follows, but we'll add the part where we make calls to the microCMS API. Please note that this article will focus on how to do this with the App Router, but the Pages Router is also covered below, and the process is quite similar.

https://www.npmjs.com/package/next-sitemap

Adding Dynamic URLs to the Sitemap

The general idea is to fetch dynamic URLs using the microCMS API and make them accessible via a URL called server-sitemap.xml. We will then reference these URLs in the sitemap.xml.

Fetching Dynamic URLs by Calling the microCMS API

First, create a directory called server-sitemap.xml under the app directory, and within it, create a file called route.ts.

.
├── app
│   └── server-sitemap.xml
│       └── route.ts

By the way, the directory structure above is displayed using the "tree" command installed with Homebrew.

$ brew install tree
$ tree src

Next, fetch the list of articles and tags by calling the microCMS API and pass them to the getServerSideSitemap function from next-sitemap.

import { getServerSideSitemap, ISitemapField } from 'next-sitemap'
import { Article, getList, getTagList, Tag } from '@/libs/microcms'
import { formatYYYYMMDD } from '@/libs/dateutil'
import { NUM_OF_ALL_PAGES_LIMIST, NUM_OF_PAGES_LIMIT } from '@/constants'

// cache for 24 hours
export const revalidate = 86400

export async function GET(request: Request) {
  const articlesData = await getList({
    limit: NUM_OF_ALL_PAGES_LIMIST,
    fields: 'id,updatedAt',
  })
  const articles = articlesData.contents

  const tagsData = await getTagList({
    limit: NUM_OF_PAGES_LIMIT,
  })
  const tags = tagsData.contents

  const fields: ISitemapField[] = []
  articles.forEach((article: Article) => {
    ;['ja', 'en'].forEach((locale) => {
      fields.push({
        loc: `${process.env.NEXT_PUBLIC_BASE_URL}/${locale}/articles/${article.id}`,
        lastmod: formatYYYYMMDD(article.updatedAt),
        priority: 1, // Priority of this URL compared to other URLs within the site.
        changefreq: 'weekly', // Page update frequency.
      })
    })
  })

  tags.forEach((tag: Tag) => {
    ;['ja', 'en'].forEach((locale) => {
      fields.push({
        loc: `${process.env.NEXT_PUBLIC_BASE_URL}/${locale}/tags/${tag.id}`,
        lastmod: formatYYYYMMDD(tag.updatedAt),
        priority: 1, // Priority of this URL compared to other URLs within the site.
        changefreq: 'weekly', // Page update frequency.
      })
    })
  })

  return getServerSideSitemap(fields)
}

Please note that in the case of the App Router, the fetch response is cached by default. However, by specifying revalidate = 86400, we ensure that the value is updated once every 24 hours. For more details on Next.js Cache behavior, please refer to the following link:

https://nextjs.org/docs/app/building-your-application/data-fetching/fetching-caching-and-revalidating#segment-cache-configuration

Add the following to next-sitemap.cnfig.js:

module.exports = {
  siteUrl: 'https://zubora-code.net',
  generateRobotsTxt: true,
  exclude: ['/server-sitemap.xml'], // Add this line
  robotsTxtOptions: {
    additionalSitemaps: ['https://zubora-code.net/server-sitemap.xml'], // Add this line
  },
  sitemapSize: 7000,
}

Additionally, for this blog, add the following to middleware.ts to exclude it from the middleware processing:

export const config = {
  matcher: [
    '/((?!robots.txt|sitemap.*.xml|server-sitemap.xml|api|_next/static|_next/image|favicon.ico|.*.png|.*.jpg).*)',
  ],
}

Check out the created sitemap

The sitemap will be created by the following commands:

$ yarn build
$ yarn start

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://zubora-code.net/sitemap-0.xml</loc></sitemap>
<sitemap><loc>https://zubora-code.net/server-sitemap.xml</loc></sitemap>
</sitemapindex>

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
<url><loc>https://zubora-code.net/ja</loc><lastmod>2023-09-09T23:56:05.065Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/en</loc><lastmod>2023-09-09T23:56:05.065Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/ja/privacy_policy</loc><lastmod>2023-09-09T23:56:05.065Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/en/privacy_policy</loc><lastmod>2023-09-09T23:56:05.065Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/ja/aboutme</loc><lastmod>2023-09-09T23:56:05.065Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/en/aboutme</loc><lastmod>2023-09-09T23:56:05.065Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
</urlset>

Go to http://localhost:3000/server-sitemap.xml by a web browser, I could see the following sitemap.

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
<url>
<loc>https://zubora-code.net/ja/articles/screen-recording-with-sound</loc>
<lastmod>2023-09-07</lastmod>
<changefreq>weekly</changefreq>
<priority>1</priority>
</url>
<url>
... 
<loc>https://zubora-code.net/ja/tags/sitemap</loc>
<lastmod>2023-09-10</lastmod>
<changefreq>weekly</changefreq>
<priority>1</priority>
</url>
<url>
...
</urlset>

When deploying this to the production environment, the following URLs are working successfully:

https://zubora-code.net/sitemap.xml

https://zubora-code.net/server-sitemap.xml

The changes for this update were addressed in the following three pull requests (PRs):

https://github.com/tkugimot/nextjs-microcms-blog-handson/pull/23?w=1

https://github.com/tkugimot/nextjs-microcms-blog-handson/pull/24/files?w=1

https://github.com/tkugimot/nextjs-microcms-blog-handson/pull/25/files?w=1

After this, Google should come to pick up the sitemap automatically, but just to be sure, let's go ahead and re-submit the sitemap through Google Search Console.

That's it! We hope this will help improve SEO to some extent!