Adding robots.txt and sitemap.xml to a blog created with Next.js and microCMS at lightning speed
Adding robots.txt and sitemap.xml to a blog created with Next.js and microCMS at lightning speed
September 10, 2023, Update
In this article, I summarized how to create a sitemap for static URLs. However, as a follow-up, I have written another article that explains how to create a sitemap for dynamic URLs (in the case of this blog, URLs under /tags and /articles). If you're interested, please check it out here:
https://zubora-code.net/en/nextjs-dynamic-sitemap
Overview
This guide explains how to add robots.txt and sitemap.xml to a blog created with Next.js and microCMS for faster performance. It includes steps for using the next-sitemap npm package, creating a next-sitemap.config.js file, and automatically generating robots.txt and sitemap.xml files during the build process.
Steps
Use the npm package named next-sitemap.
$ yarn add -D next-sitemapCreate a file named next-sitemap.config.js in the project's root directory:
/** @type {import('next-sitemap').IConfig} */
module.exports = {
siteUrl: 'https://zubora-code.net',
generateRobotsTxt: true,
sitemapSize: 7000,
}Add the following to the package.json file to automatically generate robots.txt and sitemap.xml after building:
"postbuild": "next-sitemap"Build the project:
yarn buildAfter building, the following files will be created in the public directory:
robots.txt:
# *
User-agent: *
Allow: /
# Host
Host: https://zubora-code.net
# Sitemaps
Sitemap: https://zubora-code.net/sitemap.xmlsitemap.xml:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://zubora-code.net/sitemap-0.xml</loc></sitemap>
</sitemapindex>
sitemap-0.xml:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
<url><loc>https://zubora-code.net/ja</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/en</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/ja/aboutme</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/en/aboutme</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/ja/privacy_policy</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
<url><loc>https://zubora-code.net/en/privacy_policy</loc><lastmod>2023-09-03T04:51:50.821Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
</urlset>In the case of this website, which supports multiple languages via middleware.ts, exclude robots.txt and sitemap.*.xml from the target files:
export const config = {
matcher: [
'/((?!robots.txt|sitemap.*.xml|api|_next/static|_next/image|favicon.ico|.*.png|.*.jpg).*)',
],
}Additional Notes
After deployment, register the sitemap.xml in Google Search Console. Even though the site was registered, not having the sitemap.xml registered prevented proper indexing in Google search results. Hope registering it could potentially increase site traffic.
