gatsby: [gatsby-source-drupal] "Failed to process remote content" error on Drupal images

Summary

Occasionally, during our Gatsby build, right after the line Starting to fetch data from Drupal, we will see about 20-25 images which say Failed to process remote content.

In the snippet below, I’ve change the exact filenames, but kept spaces, dashes, filetypes so you can get a sense of the kind of files that occasionally fail.

Note that some of the files that fail to process are large (4 mb) but others are tiny (80 kb). We’ve noticed pngs, gifs, and jpgs so far.

Is there a maximum file-size limit that Gatsby will be able to process for remote content? Or is there a file-naming issue with any of the files below (e.g. do we need to avoid dashes, underscores, or spaces)?

Any assistance would be appreciated!

Starting to fetch data from Drupal
Failed to process remote content http://site-url/sites/default/files/inline-images/Name_Of_File_S.jpg
Failed to process remote content http://site-url/sites/default/files/2019-03/name-file_0.png
Failed to process remote content http://site-url/sites/default/files/2019-03/Test%20File%20With-Dash_cropped.jpg
Failed to process remote content http://site-url/sites/default/files/nameoffile.gif
Failed to process remote content http://site-url/sites/default/files/Name%20of%20file_2018_1.jpg
Failed to process remote content http://site-url/sites/default/files/NameOfFile.gif
Failed to process remote content http://site-url/sites/default/files/TEST_file2_colour.jpg
Failed to process remote content http://site-url/sites/default/files/Nameoffile.gif
Failed to process remote content http://site-url/sites/default/files/Test_test-file_test-1440x960.jpg
Failed to process remote content http://site-url/sites/default/files/Test%name%20file_2018%20resized%2050.jpg
Failed to process remote content http://site-url/sites/default/files/Test.jpg
Failed to process remote content http://site-url/sites/default/files/Test.png
Failed to process remote content http://site-url/sites/default/files/Test_File_with_Underscores.jpg
Failed to process remote content http://site-url/sites/default/files/testfile_0.png
Failed to process remote content http://site-url/sites/default/files/Test%20File_RETOUCH_S.jpg
Failed to process remote content http://site-url/sites/default/files/testing.gif
Failed to process remote content http://site-url/sites/default/files/only-dashes-and-numbers-292636697.jpg
Failed to process remote content http://site-url/sites/default/files/TestFile.png
Failed to process remote content http://site-url/sites/default/files/Testchartupdated.jpg
Failed to process remote content http://site-url/sites/default/files/TEST%20File_RETOUCH_S.jpg
Failed to process remote content http://site-url/sites/default/files/test_file.png
Failed to process remote content http://site-url/sites/default/files/Test%20of%20Spaces%20File%20Test-176_.jpg
Failed to process remote content http://site-url/sites/default/files/testfile_0.png

success source and transform nodes — 11.138 s

Relevant information

Environment (if relevant)

  npmPackages:
    gatsby: ^2.0.76 => 2.0.117 
    gatsby-image: ^2.0.20 => 2.0.29 
    gatsby-plugin-favicon: ^3.1.5 => 3.1.5 
    gatsby-plugin-google-analytics: ^2.0.9 => 2.0.13 
    gatsby-plugin-manifest: ^2.0.9 => 2.0.17 
    gatsby-plugin-offline: ^2.0.24 => 2.0.24 
    gatsby-plugin-react-helmet: ^3.0.2 => 3.0.6 
    gatsby-plugin-sharp: ^2.0.14 => 2.0.20 
    gatsby-source-drupal: ^3.0.15 => 3.0.23 
    gatsby-source-filesystem: ^2.0.23 => 2.0.23 
    gatsby-transformer-sharp: ^2.1.8 => 2.1.13 
  npmGlobalPackages:
    gatsby-cli: 2.4.8

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 25 (10 by maintainers)

Most upvoted comments

We are seeing the same issue happening with gatsby-source-wordpress. About 50% of the time our builds fail with seemingly no rhyme or reason and no code changes. Any advice would be greatly appreciated.

Downloading remote files [=====================---------] 3066/4314 97.0 secs 71%Failed to process remote content https://blog.site.com/wp-content/uploads/2016/03/image.jpg
⠂ source and transform nodesFailed to process remote content https://blog.site.com/wp-content/uploads/2016/03/image.jpg
Downloading remote files [======================--------] 3137/4314 99.6 secs 73%Failed to process remote content https://blog.site.com/wp-content/uploads/2016/03/image.jpg
⠠ source and transform nodesFailed to process remote content https://blog.site.com/wp-content/uploads/2016/03/image.jpg
Downloading remote files [======================--------] 3153/4314 99.9 secs 73%Failed to process remote content https://blog.site.com/wp-content/uploads/2016/03/image.jpg

joxenford on Apr 1, 2019

Hi @bakeruk . I just realized that this has more to do with my server configuration than Gatsby. My IP address currently blocked because of too many requests. Thanks for the quick response

jamaljeantobias on May 21, 2019

Fantastic! This fixed my issue. Thank you @bakeruk for the fix and @wardpeet for publishing the new version. Very pleased with the turnaround time on this. 👍

mmafe on May 15, 2019

Duplicate of #12280 closing in favour of that issue.

wardpeet on Mar 26, 2019

Try swapping out your forEach loops for a forIn loop.

A forEach loop doesn’t honour a promise before it continues on to the next item in the loop.

For example,

for (const i in pages) {
  if (pages[i]) {
    const page = pages[i];

    ...
  }
}

Also try adding a trycatch into your code around the createRemoteFileNode function to see if you can see any uncaught errors.

try {
  const fileNode = await createRemoteFileNode({
    url: s3URL,
    store,
    cache,
    createNode,
    createNodeId,
  }); 
} catch (err) {
  console.log(err);
  
  // Optional depending how you want to proceed
  throw err;
}

bakeruk on May 10, 2019

Thanks for quick responses @coreyward and @wardpeet!

I agree #12280 is related, however I don’t think this is a duplicate issue. (i.e. From what I gather, issue #12280 appears to be questioning the logic behind showing the console warning instead of just rejecting it.)

Is it possible to re-open this issue? It’s still not clear to me why images that exist on our source Drupal site are randomly resulting in this error. Could it be the filenames? Filesize?

mmafe on Mar 26, 2019