gatsby: Preventing 301 redirects on URLs with no trailing slashes (Netlify)

@wardpeet @KyleAMathews Would like to bring your attention to this issue again because it’s a major problem for SEO, as many people in the tread already mentioned. Asking for PRs is great, but this particular one looks to be related to some of the core functionalities of the Gatsby engine, so I myself wouldn’t take responsibility touching it, not knowing all ins and outs of Gatsby.

Could you or someone with deep knowledge take a look and offer some kind of solution?

Also, issue is not related to Netlify or Github pages. It can be reproduced locally, I have it on Firebase Hosting for production.

Additionally, there is already a PR from @cosmn. Maybe it already solves the issue or can be used as an idea on what can be done?

If you’re using Gatsby + Netlify, here’s how we solved it.

First, any pages in the /pages directory, which are created automatically by Gatsby, will have a trailing slash. You can see this behavior by entering a URL without a trailing slash directly into the search bar. The network tab will show a 301 as the page is redirected to the one with the trailing slash.

Try running this GraphQL query and you will see that the path for these pages has a trailing slash.

query MyQuery {
  allSitePage {
    edges {
      node {
        path
      }
    }
  }
}

To remove the trailing slash from the page path, you can add this code snippet from the Gatsby docs to your gatsby-node.js file:

// Replacing '/' would result in empty string which is invalid
const replacePath = path => (path === `/` ? path : path.replace(/\/$/, ``))
// Implement the Gatsby API “onCreatePage”. This is
// called after every page is created.
exports.onCreatePage = ({ page, actions }) => {
  const { createPage, deletePage } = actions

  const oldPage = Object.assign({}, page)
  // Remove trailing slash unless page is /
  page.path = replacePath(page.path)
  if (page.path !== oldPage.path) {
    // Replace new page with old page
    deletePage(oldPage)
    createPage(page)
  }
}

Next, you need to turn off the pretty URL setting in Netlify. If you have Asset optimization disabled, enable it, turn off pretty URLs, and then disable it again.

Push a new build and then observe the network requests for a page that had a trailing slash. For instance, after typing https://www.software.com/code-time then pressing Enter, the first request is a 200 (there isn’t a 301 like before):

If anyone has any better solutions, let me know! We haven’t solved the trailing slash on localhost.

This is definitely still an issue, and we’ve experienced SEO penalties as well. Additionally, turning off Netlify’s pretty URLs feature seems to result in errors stating “Missing resources for /“ or “Missing resources for /slash/“. We’ve tried solutions recommended here: https://github.com/gatsbyjs/gatsby/issues/11524 but haven’t had any luck.

Just wanted to reply that we are looking forward to this bug being resolved. We are using urls without trailing slashes btw, we have the issue that Twitter/Facebook will not show the thumbnail we set, because the url is redirected to the version with a trailing slash. (Twitter Card Validator will return: WARN: this card is redirected to https://foo.com/bar/ when submitting https://foo.com/bar). Twitter will only pick up the thumbnail after manually using Twitter Card Validator to enter the url, and it will forget it after a week (Twitter re-indexes card content every week or so, and it won’t index the urls that are hidden behind a redirect).

The performance cost is the synchronous delay for the first byte of useful data caused by the redirect.

Using the Hopper example above, visiting https://www.hopper.com/company takes 150-300ms for the redirect, before any page data is received. On cellular connections with high latency it can add up to 1s.

Hey 👋🏻

I know this issue lapsed and got closed but I really think it’s important to recap on a couple of things here. The impetus for this issue is that a) there’s a disconnect between Gatsby’s routing defaults and Netlify’s routing configurations, and b) there are serious SEO penalties in play if a Gatsby site doesn’t have the trailing-slash / no-trailing-slash issue solved, since a site serving the same content on both URLs (duplicate content) gets knocked on SEO pretty hard. Technically this isn’t Gatsby’s fault, but as it pertains to all Gatsby users hosting on Netlify, it does seem like a major issue… or a major risk at the very least.

Solving this problem by disabling “Pretty URLs” in the (yes, awfully borked / painful UX’d) Netlify Asset Optimization panel can open your site up to the duplicate content issue since content may be available at both the un-slashed and the slashed version of your URL path. It’s important too to note that if a Gatsby site is available on both /test and /test/ but ‘fixes itself’, you may just be seeing the Gatsby runtime adjust your address bar via the Browser History API - the super important part has nothing to do with what happens when Gatsby actually runs in the browser - it’s the part where Netlify is serving the same content on multiple URLs - the slash and the non-slash paths.

This is fixable and there is a way to get everything working smoothly and on a unified path / slash structure, but it’s not disabling ‘Pretty URLs’. The tl;dr: is that Netlify really works best / has biases toward using the trailing slash, and unified content pathing on Netlify requires the trailing slash. I elaborated on this in another Gatsby GH thread here:

https://github.com/gatsbyjs/gatsby/discussions/27889#discussioncomment-254854

But I would definitely urge folks to carefully check (from a CLI HTTP tool preferably) which paths (slash and/or no-slash) are resolving to their content on their sites. If both the slash and no-slash paths are resolving to your content, your SEO will hurt for it.

Hope that helps 😕

This is a bug in the Netlify UI.

Here’s a fix: https://community.netlify.com/t/remove-trailing-slash-redirect-for-gatsby-gatsby-cloud-netlify-website/20976/8

This is indeed the case.

Here is how your netlify config should look like:

Disabling optimization at the top level apparently turns on the pretty URLs, even though it visually looks like that isn’t the case:

So don’t check the checkbox next to “Disable asset optimization”

this also happens with nginx server.

EDIT: this issue has nothing with gatsby. it is web server misconfiguration. I have checked output files and it seems gatsby creates a directory for each page and and index.html in it. So I had to change my nginx url resolving as following:

location / {
  try_files $uri $uri/index.html $uri.html =404;
}

The $uri/index.html is resolving correct file without redirect. If that doesn’t exists or it is $uri/ ( most nginx conf examples uses that ) it will create a redirect with trailing slash. It is also stated in nginx documantation.

In response to a request with URI equal to this string, but without the trailing slash, a permanent redirect with the code 301 will be returned to the requested URI with the slash appended.

I have not used Netlify but I believe same thing would apply to it.

This is a problem in AWS as well. Gatsby is a scam. my solution is sadly, to append a forward slash everywhere. oneliner: if (to[to.length - 1] != "/") to += "/" if to is the link variable.

Just to clarify, the ‘Pretty URLs’ option in netlify WILL redirect you to the trailing slash variant

This is true but I haven’t noticed any difference in behaviour with or without “Pretty URLs”; /foo redirects to /foo/ in either case. The desired behaviour is to rewrite slashless urls to resolve without redirecting.

Making /foo/ canonical is a workaround for the SEO penalties but AFAIK a full solution would only be possible in Netlify’s routing layer… or by abandoning Netlify in favour of custom url rewriting …or by adopting slash/ urls. It’s an unfortunate situation.

This is a bug in the Netlify UI.

Here’s a fix: https://community.netlify.com/t/remove-trailing-slash-redirect-for-gatsby-gatsby-cloud-netlify-website/20976/8

Thanks for all the helpful comments which lead us in the right direction.

If it still doesn’t work after setting it to @mlenser’s comment (https://github.com/gatsbyjs/gatsby/issues/9207#issuecomment-695913229), make sure to check your netlify.toml for pretty_urls.

Settings in the toml take precedence, see the docs: https://docs.netlify.com/configure-builds/file-based-configuration/#deploy-contexts

UI settings are overridden if a netlify.toml file is present in the root folder of the repo and there exists a setting for the same property/redirect/header in the toml file.

Guys, it’s a setting within Netlify as described above. Perhaps a new issue needs to be created.

On Tue, 23 Jun 2020, 16:29 Lawrence Whiteside, notifications@github.com wrote:

This is a problem in AWS as well. Gatsby is a scam.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gatsbyjs/gatsby/issues/9207#issuecomment-648195328, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALSIKRE6IUIRXBXD4PVTCYLRYC35TANCNFSM4F5VAMWA .

Not at the moment. The closest would be name your pages like page.html.js. My advice to you would be to stick with trailing slashes. For non-trailing slashes to work you’ll need to do some server-side url rewriting.

@KyleAMathews

@himynameistimli I did find a solution, proposed a code change in another thread. But seems like it might not be accepted so have not created a PR.

The gist of it:

• Hosting on github pages. Github follows the directory structure when serving files. E.g. if you hit /somepage, it will redirect to /somepage/ because it’s a directory (actual file is /somepage/index.html.
• My proposed solution was that gatsby generate /somepage.html instead of /somepage/index.html

@KyleAMathews this is seo danger This subject isn’t explained at all in https://www.gatsbyjs.org/docs/gatsby-link/ If someones decides to use no-trailing-slash-urls, a couple of days later the google serp becomes full of 301 redirections for your site.

@0505gonzalez were you able to figure out a solution for this issue?

Just landed here as we’re having the same problems with the url parameters getting lost in the redirect from the version without the trailing slash to the version with the trailing slash.

Only difference is that we’re on S3 + Cloudfront.

We might look into using Lambda@Edge to handle the redirect unless we can figure out a way to get it to work with gatsby.

Update:

For our case, we implemented the fix from https://www.ximedes.com/2018-04-23/deploying-gatsby-on-s3-and-cloudfront/ with the following lambda js function:

const querystring = require('querystring');
exports.handler = (event, context, callback) => {
    const request = event.Records[0].cf.request;

    /* Parse request query string to get javascript object */
    const params = querystring.parse(request.querystring.toLowerCase());
    const sortedParams = {};
    const uri = request.uri;

    /* Sort param keys */
    Object.keys(params).sort().forEach(key => {
        sortedParams[key] = params[key];
    });

	/* Simple way return the index.html */
    if (uri.endsWith('/')) {
        request.uri += 'index.html';
    } else if (!uri.includes('.')) {
        request.uri += '/index.html';
    }

    /* Update request querystring with normalized  */
    request.querystring = querystring.stringify(sortedParams);

    callback(null, request);
};

I’m still trying to figure out what’s the best thing to do here, because I think as-is, there’s a negative impact to our SEO just because now we’re delivering the same page for the trailing and non-trailing slash version. Likely I will add a permanent redirect for the trailing slash version here too.

If you’re not using AWS/Cloudfront, I think you’d be able to accomplish this with Cloudflare Workers.

If you want your website to not have any trailing slash and also work with Netlify you can use the gatsby-plugin-netlify and in gatsby-node.js add

const replacePath = path => (path === `/` ? path : path.replace(/\/$/, ``))

exports.onCreatePage = ({ page, actions }) => {
  const { createRedirect } = actions
  if(!page.path.includes('.html') && page.path !== '/') {
    createRedirect({ fromPath: `${page.path}/`, toPath: page.path, isPermanent: true })
  }
}

this will redirect all trailing slash to non-slash paths with 301 status code. Note if you are also using createPages Gatsby node API you’ll need to add it there also

exports.createPages = async ({ actions, graphql }) => {
    const { createPage, createRedirect } = actions
       // ...
       pages.forEach(page => {
          // ...
          createRedirect({ fromPath: `${page.path}/`, toPath: page.path, isPermanent: true })
       })
    })
}

@alvinometric I’m not sure — I’ve sent this over to our UI team for review. it does look like if this isn’t a bug, it could do with some clarification

This is a bug in the Netlify UI. Here’s a fix: https://community.netlify.com/t/remove-trailing-slash-redirect-for-gatsby-gatsby-cloud-netlify-website/20976/8

This is indeed the case.

Here is how your netlify config should look like:

Disabling optimization at the top level apparently turns on the pretty URLs, even though it visually looks like that isn’t the case:

So don’t check the checkbox next to “Disable asset optimization”

I’ve just lost 3 hours because of this.

BOLEST.

this also happens with nginx server.

EDIT: this issue has nothing with gatsby. it is web server misconfiguration. I have checked output files and it seems gatsby creates a directory for each page and and index.html in it. So I had to change my nginx url resolving as following:

location / {
  try_files $uri $uri/index.html $uri.html =404;
}

The $uri/index.html is resolving correct file without redirect. If that doesn’t exists or it is $uri/ ( most nginx conf examples uses that ) it will create a redirect with trailing slash. It is also stated in nginx documantation.

In response to a request with URI equal to this string, but without the trailing slash, a permanent redirect with the code 301 will be returned to the requested URI with the slash appended.

I have not used Netlify but I believe same thing would apply to it.

This is not the case, even if you have a blank nginx config, any attempt to access a valid directory will result in a 301 with the trailing slash. It’s a common misconception that it is related to the try_files. The documentation is also poor surrounding this, the location documentation makes it seem like the 301 redirect is only for routes that are proxied.

I made a plugin from a code that I normally use in websites hosted at Netlify. It creates a .html file for each page, which disables the 301 redirect for paths without trailing slashes. It works very well with simple websites, I hope it helps someone. More info can be found on the plugin page.

Can someone summarise the current situation?

Solution: still WIP.

Hack:

1. Ensure trailing slashes are used in all routes
2. Add a trailing slash to the canonical link

Please feel free to correct me and I’ll amend/delete the above.

I don’t think that this issue is related to Netlify or Github pages. I can reproduce it locally running gatsby serve, which appears to be expected behaviour Gatsby’s redirect handling.

@lloydh / @luukdv - can you reproduce locally?

@himynameistimli I did find a solution, proposed a code change in another thread. But seems like it might not be accepted so have not created a PR.

The gist of it:

• Hosting on github pages. Github follows the directory structure when serving files. E.g. if you hit /somepage, it will redirect to /somepage/ because it’s a directory (actual file is /somepage/index.html.
• My proposed solution was that gatsby generate /somepage.html instead of /somepage/index.html

@jaapstronks Are you by any chance running with netlify? Managed to fix this issue here https://github.com/gatsbyjs/gatsby/issues/15317#issuecomment-530048373

I’ve just started evaluating Gatsby. Is there a way that, say for a given path /foo, have Gatsby generate public/foo.html instead of public/foo/index.html?

Just to clarify, the ‘Pretty URLs’ option in netlify WILL redirect you to the trailing slash variant:

In addition to forwarding paths like /about to /about/ (a common practice in static sites and single page apps), it will also rewrite paths like /about.html to /about/.