next.js: Next 13 - Sitemap can't fetch on Google Search Console

Verify canary release

I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: darwin
      Arch: x64
      Version: Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64
    Binaries:
      Node: 18.16.0
      npm: 9.5.1
      Yarn: 1.22.19
      pnpm: 7.29.3
    Relevant packages:
      next: 13.4.6
      eslint-config-next: 13.2.4
      react: 18.2.0
      react-dom: 18.2.0
      typescript: 4.9.5

Which area(s) of Next.js are affected? (leave empty if unsure)

App directory (appDir: true)

Link to the code that reproduces this issue or a replay of the bug

https://codesandbox.com

To Reproduce

export default async function sitemap() {
  const db = await connecToDatabase();
  const usersCollection = db.collection("Users");

  // articles
  const articles = await API.getPosts(
    "",
    undefined,
    undefined,
    "published"
  )
    .then((res) => res)
    .catch((error) => console.log("error fetching content"));
  const articleIds = articles?.map((article: Article) => {
    return { id: article?._id, lastModified: article?.createdAt };
  });
  const posts = articleIds.map(({ id, lastModified }) => ({
    url: `${URL}/${id}`,
    lastModified: lastModified,
  }));

  // users
  const profiles = await usersCollection.find({}).toArray();
  const users = profiles
    ?.filter((profile: User) => profile?.userAddress)
    ?.map((profile: User) => {
      return {
        url: `${URL}/profile/${profile.userAddress}`,
        lastModified: new Date().toISOString(),
      };
    });

  // tags
  const tagsFromDb = await articles
    ?.map((article: Article) => article?.categories)
    ?.flat();

  const uniqueTags = tagsFromDb.reduce((acc, tag) => {
    const existingTag = acc.find((item) => item.id === tag.id);

    if (!existingTag) {
      acc.push(tag);
    }

    return acc;
  }, []);

  const tags = uniqueTags
    ?.filter((tag) => tag?.id)
    ?.map((tag) => {
      return {
        url: `${URL}/tags/${tag.id}`,
        lastModified: new Date().toISOString(),
      };
    });

  const staticPages = [
    {
      url: `${URL}`,
      lastModified: new Date().toISOString(),
    },
    { url: `${URL}/about`, lastModified: new Date().toISOString() },
    { url: `${URL}/read`, lastModified: new Date().toISOString() },
  ];

  return [...posts, ...users, ...tags, ...staticPages];
}

Describe the Bug

Hello,

I’m using Next 13 with the /app directory and trying to configure the sitemap of my project on Google search console.

I have used the documentation as described there: Documentation

I have a sitemap.ts in the root of my /app directory, but it seems not recognized by GSC, and i know the sitemap is valid: URL and i’ve checked also using this tool

Expected Behavior

I want the /sitemap.xml to be recognized by Google search console.

Which browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

About this issue

Original URL
State: open
Created a year ago
Reactions: 9
Comments: 61 (2 by maintainers)

Most upvoted comments

I added a trailing slash to my sitemap and it started to work. Both links are loading fine on the browser.

/sitemap.xml/

And Google managed to pick it up.

https://ruchern.xyz/sitemap.xml/

+11

ruchernchong on Nov 12, 2023

Has someone solved this issue ? I’m still stuck on it without any pieces of possible solution…

i did as same document of next.js has wroten for robots.ts and sitemap.xml and has same problem

loverphp487 on Jul 9, 2023

I just found out that the sitemap.xml is returning HTTP 304 instead of HTTP 200.

robots.txt returns HTTP 200 and is working fine.

Really need @leerob @amyegan to chime in on this.

ruchernchong on Feb 6, 2024

Has someone solved this issue ? I’m still stuck on it without any pieces of possible solution…

anthonyjacquelin on Jun 29, 2023

It seems to be an issue of Google Search Console. My solution was, on Google Search Console, submitting the sitemap url with a suffix: yourwebsite.com/sitemap.xml?sitemap=1 instead of yourwebsite.com/sitemap.xml Reference

FYI, I used next-sitemap package to auto generate my sitemap.xml at every new build.

lydhr on Mar 27, 2024

I use React with Vite with the sitemap.xml in the public folder and have the same error: Couldn't fech in https://studio20.vercel.app/sitemap.xml So I tried: https://studio20.vercel.app/sitemap.xml/ and still not work. I change the sitemap file name to sitemap-1.xml and it doesn’t work too I no have idea of what to do.

Interesting. Although, this is issue is not related to Vercel, but interesting to see that React with Vite is not working as well.

ruchernchong on Mar 3, 2024

I just found out that the sitemap.xml is returning HTTP 304 instead of HTTP 200. robots.txt returns HTTP 200 and is working fine. Really need @leerob @amyegan to chime in on this.

I also had a 304 code

I tried to export const dynamic = "force-dynamic"

But i still experienced couldn’t fetch …

I used the unstable_noStore() and it did not work despite the sitemap.xml is now returning HTTP 200.

ruchernchong on Feb 15, 2024

After encountering the same issues as you had, in my case with "next": "13.4.19" App Router and their native solution for sitemap (https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap#generate-a-sitemap)

I found this article and applied the Next.js 13.2 and lower solution proposed by the article https://claritydev.net/blog/nextjs-dynamic-sitemap-pages-app-directory#nextjs-132-and-lower

What happened? The route app/sitemap.xml/route.ts didn’t work, and I suspected it might be due to caching by Google…

…so I tried app/sitemap2.xml/route.ts, and it worked (yep, same code…)

Now, sitemap2.xml is working properly in Google Search Console.

My sitemap.xml is still available with the same code, but Search Console is unable to fetch it. I removed it and added it again, and it’s still not working. So, my plan is to remove it for some days or weeks and then try adding it again. At least, I’m indexing with sitemap2.xml, which is dynamic.

iperdev on Sep 15, 2023

Hi guys!

I think it is not an error. Neither on Google nor Vercel. Better saying, I’m not sure it is not kinda an error on Google, because I really think it should have a better message to this situation. You can read further info about this in the link below: https://support.google.com/webmasters/thread/184533703/are-you-seeing-couldn-t-fetch-reported-for-your-sitemap?hl=en&sjid=15254935347152386554-SA

I spent 30 minutes searching on the web thinking it was a problem.

If it is not a bug and just due to time needed for google to process the sitemap, all our sitemaps would have been handled by google after a while. The fact is that even after 1 month i still see “can’t fetch”.

So there might be a bigger problem than just a messy error message + time needed for google to handle it.

anthonyjacquelin on Aug 12, 2023

Hi guys!

I think it is not an error. Neither on Google nor Vercel. Better saying, I’m not sure it is not kinda an error on Google, because I really think it should have a better message to this situation. You can read further info about this in the link below: https://support.google.com/webmasters/thread/184533703/are-you-seeing-couldn-t-fetch-reported-for-your-sitemap?hl=en&sjid=15254935347152386554-SA

I spent 30 minutes searching on the web thinking it was a problem.

lucas-soler on Aug 10, 2023

The most important thing is to have an up to date version of your sitemap if you have dynamic pages being created.

I agree.

Sorry if I didn’t communicate it well. I could see the build log at hand, which seems to be dynamically generated only at build time to begin with. I wish it would always be generated dynamically.

octane96 on Aug 9, 2023

Thanks for your feedback, so at the end of the day this is not dynamic anymore…

That’s right… I agree.

So maybe we could create a cron api route that will write this sitemap.xml file every day or week using fs

Thanks for the very good ideas! I’ve written a simple cron for now, so I’ll get by with that for a while!

octane96 on Aug 9, 2023

Do you have a file at app/robots.ts? See here for an example.

This file lets engines and crawlers know where to find your sitemap. You can read more about it here

same issue. I have enabled the sitemap and have added the following code to app/robots.ts but cannot register the sitemap.

import type { MetadataRoute } from "next"

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: "*",
      },
    ],
    sitemap: "https://my-url.xyz/sitemap.xml",
    host: "https://my-url.xyz",
  }
}

Maybe some more time needs to pass, so I’ll give it a little more time.

ryuji-orca on Jun 23, 2023